The shift to the Intelligence Era will affect every industry and line of business. Everything — from databases to apps to visualization — is changing as we move from systems of record to systems of intelligence. The talent and technical demands of this new stack present challenges to CIOs and CTOs, and opportunities for emerging startups to target. We explore this new landscape below.
The Evolution of the Enterprise Stack
Here’s a simplified view of how the enterprise stack evolved.
Working from the bottom up:
- Databases were typically expensive, in proprietary formats, difficult to maintain and kept on premise. Today, databases are significantly cheaper, in open source formats, almost exclusively in the cloud, and maintained by open source communities or the vendors. Companies can enrich internal data stored in the cloud with external data, in-flight, on the way from their cloud data layer to third-party applications, allowing for richer data sets on which to experiment with machine learning.
- Companies previously needed a wide range of SDKs and middleware to hook their databases up to applications, either in-house or third-party. Today, they can do this and add functionality through well documented, public APIs.
- Different lines of business used different applications to execute manual workflows in software. Today, each line of business collects data and uses machine learning to make decisions.
- Software developers previously wrote internal applications ‘from scratch’ where third-parties could not provide appropriate solutions. Now, developers assemble functionality from APIs and their own code to make internal applications. Additional value is created by data scientists and machine learning engineers that build models to predict outcomes of commercial or industrial consequence, augmenting or automating decision-making across the organization.
- Software developers previously used rudimentary development tools for version control and general productivity. A complete set of advanced solutions are available to software developers today. Data scientists and machine learning engineers, however, are finding their own, new set of tools.
- Business owners requiring data analytics previously relied on internal, ‘business intelligence’ teams who would manually pull data, crunch it and put it in a report. Today, they can access their business data in real time, visualizing and analyzing it on-the-fly, through intuitive products, without requiring large teams with technical expertise and data crunching tools.
These new layers to the enterprise stack present today’s organizations with new opportunities to build a competitive advantage. Such opportunities include the following.
Business Optimization
Organizations can now distribute data for decision making to everyone — not just executives — through business optimization products. Domo is such a product that pulls data from internal and cloud-based systems (no coding required), cleans and refreshes data, visualizes or pipes data into other apps, then offers predictive analytics to help business users make decisions. This previously required teams of people wrangling data and making reports. Now, it’s all done through a product. Consequently, anyone in the organization — rather than just the CEO commanding a BI team — can access and utilize data, leading to more transparent, responsive and creative companies.
Intelligent Apps
The predominant function of legacy, cloud-based software was to get data from users’ heads into the cloud. The best of this software offered users workflow and productivity benefits, but were fundamentally an interface on top of a database.
Software in the Intelligence Era collects proprietary data across customers and public sources then applies machine learning to generate insights. This software thus helps users make decisions and become the Systems of Engagement and Intelligence. The software of the cloud era goes from being that with which we engaged (a ‘System of Engagement’) to being the System of Record.
Integrating External Data
Building an intelligent system starts with having the right data. This means effectively storing your organization’s proprietary data and complementing it with data from other sources. Modern cloud data systems and enrichment tools like Clearbit allow companies to enrich their own data with external data, in-flight, on the way from their cloud data layer to third-party applications, allowing for richer data sets on which to experiment with machine learning.
We see lots of novel data collection methods among corporates and startups today, for example, customer data networks (‘give to get’), crowdsourcing, teams of domain expert data labelers, Active Learning systems, special-purpose consumer apps, forging partnerships with legacy Systems of Record and forging partnerships with companies that have complementary data (e.g. Clearbit and Segment).
Building a Data Science and Machine Learning Team
Once the data has been collected, the organization needs people who understand how to work with that data. Above is a diagram showing what talent is needed (in the red boxes) for building each part of a data-driven organization.
Most companies do not have data scientists or machine learning engineers today and will need to hire them as we move into the Intelligence Era. Data Scientists tend to perform complex statistical analyses, build dashboards and visualize data. They typically have a quantitative background but come from a variety of fields, from physics to geology. Machine learning engineers are computer science and statistics experts that know ML algorithms, systems and feature engineering. They typically have Ph.D.s in computer science or applied mathematics.
Providing the Right Tools
Those data scientists and machine learning engineers require a new suite of tools that can, for example, version both code and data at the same time to allow for the production of reproducible, predictive models. For example, Domino can be seen as the ‘Atlassian’ of the Intelligence Era.
The Road Ahead
We hope that entrepreneurs and enterprise IT leaders can take this roadmap and run with it to find the right balance of buying and building their way into the Intelligence Era. Making such strategic investments to the enterprise stack will be key to developing an IT organization that delivers the next generation of intelligent applications to your users, augmenting and automating their daily decisions.