Customer and product data lived in 30+ disconnected systems holding 1B+ data elements — Salesforce CRM, EHQ (MySQL), Matomo real-time analytics, GovD (Oracle), scraped Open Cities content and more. Copy-based ETL multiplied storage and drift; teams reconciled sources before any analysis, and GenAI initiatives had no governed data foundation.
What was in the way.
- Customer and product data lived in 30+ disconnected systems holding 1B+ data elements.
- Copy-based ETL multiplied storage and drift; teams reconciled sources before any analysis.
- GenAI initiatives had no governed data foundation to build upon.
- Built a zero-copy data mesh on Dremio with Staging → Business → Application (semantic) layers using dbt on Apache Iceberg.
- Governed every Iceberg table via the Apache Polaris catalog on AWS S3.
- Auto-Reflections deliver sub-second queries; auto-scaling absorbs large workloads.
- Exposed the semantic layer to GenAI agents through Dremio MCP and Amazon Bedrock AgentCore.
The build, step by step.
- 01
Staging Layer
Virtualized and ingested raw sources from 30+ disconnected systems.
- 02
Business Layer
Conformed models built as code with dbt on Apache Iceberg.
- 03
Semantic Layer
Application views defined once, version-controlled and tested.
- 04
Governance
Apache Polaris catalog centralizes access control across all engines.
- 05
Performance
Auto-Reflections deliver sub-second governed queries at scale.
- 06
GenAI Enablement
Semantic layer exposed as Dremio MCP for agents and RAG on AWS.
- Query EngineDremioZero-copy lakehouse engine.
- Table FormatApache IcebergACID, schema evolution, time travel.
- Semantic ModelingdbtStaging, Business and Application layers as code.
- CatalogApache PolarisAWS S3Fine-grained governance.
- GenAIDremio MCPAmazon Bedrock
- BIHighchartsGoverned self-serve dashboards.
One governed semantic layer — humans, dashboards and AI agents query the same truth with zero copies. Eliminates copy-based ETL cost and drift across 30+ sources and 1B+ data elements, delivers sub-second governed queries via Auto-Reflections, ~70% faster customer setup and ~60% less research effort — and a production-ready foundation for GenAI agents on AWS.
