Prior decade: data modeling, pipelines and DAGs, frameworks and tools for data producers
Next decade: metrics modeling, metric trees, frameworks and tools for operators
The past decade has been defined by the growth of a comprehensive ecosystem to enable data producers.
- Technologies like elastic cloud storage and compute via data lakes and warehouses have revolutionized how data is ingested, stored and processed.
- Tools like DBT for modeling have empowered data teams to model and manage complex data transformations efficiently.
- Abstractions like data pipelines, DAGs, orchestration are now table stakes.
- Frameworks for monitoring and ongoing quality checks ensure smooth operations.
This ecosystem has allowed data producers to collect, clean, transform, and manage data at scale, enabling organizations to establish a strong data foundation.
Looking ahead to the next decade, the focus is on super-charging data consumers and business operators. This will see the rise and adoption of:
- Standardized metrics for various business models and variants.
- Operational abstractions like metric trees.
- Advanced algorithms and deeper understanding of causality within the business
- Frameworks for automated monitoring and analytics
- The rise of AI agents to enable seamless data-driven workflows
Excited to work at this frontier!
