Data platforms are a case study in how progress through technology involves solving one problem, only to shift the constraints to another. As of today, effortlessly scaling intelligent decision-making has become the frontier problem—an outcome of solving a series of upstream challenges.
This image shows the evolutionary progress in the data stack, though not in any particular timeline order. The problems we’ve tackled include:
- Reducing data storage and compute costs
- Easing data ingestion from a variety of raw sources
- Enabling robust data pipeline building and maintenance
- Expanding the calculation and dissemination of metrics far beyond core financial data
- Shortening reporting cycles and reducing pain points for data producers
These advances have led to data abundance—a proliferation of data assets and reports. But this has also increased the amount of work consumers must invest to extract value. In addition, with the rise of the “self-service” model, the onus has shifted to consumers to extract the signal needed to operate, without the corresponding deep technical understanding of the data assets or training in data interpretation.
As a result, the primary constraint has shifted: how can consumers across the organization, armed merely with knowledge of the business, extract value from the data at hand?
This is the exciting new frontier to tackle. And all the buzzwords we hear—agents, AI, and automation—will play a crucial role in solving this challenge.
