Ninety-plus retailers deliver POS data as encrypted, compressed archives in mismatched formats — XML, JSON, fixed-width and raw. The legacy per-client Python ETL (pyardm) was costly to extend, hard to test, and slow to onboard new clients or reconcile against the warehouse.
What was in the way.
- 90+ retailers deliver POS data as encrypted, compressed archives in mismatched formats.
- Legacy per-client Python ETL (pyardm) was costly to extend and hard to test.
- Slow to onboard new clients or reconcile data against the warehouse.
- Built a Snowflake-native platform: a .NET loader streams atomic rows into RAW_DATA.
- dbt transforms data through a typed layer flow into the governed ARDM star schema.
- A suite of Claude Code skills onboards clients, compares against the legacy load, and drives column-level parity autonomously.
The build, step by step.
- 01
Ingestion
Snowpipe Streaming loads encrypted POS files atomically into RAW_DATA.
- 02
Transformation
Flat mapping, enrichment, fastfact, validate and merge flows built with dbt.
- 03
Agentic Onboarding
/onboard-client authors models from legacy pyardm and live blob bytes in days.
- 04
Parity Refinement
/refine-ardmprod-parity diffs columns against legacy and rebuilds until parity.
- 05
Version Migration
Handles ARDM v1/v2/v3 with auto-detection of 12+3 or 8+4 table sets.
- 06
Observability
MONITORING.DBT_RUNS, App Insights and Streamlit dashboards.
- Data PlatformSnowflakeSnowpipe Streaming
- Loaders.NET Azure FunctionsStreaming atomic rows.
- TransformationdbtTyped layers flowing into the ARDM star schema.
- Agentic SkillsClaude CodeSkills for onboarding, parity and backfill.
- CI/CDAzure DevOpsAzure Pipelines
- SecurityPGP / GPGAzure Blob Storage
One governed retail star schema — 90+ clients across two regions kept at parity with the legacy load. Agentic onboarding compresses new-client delivery from weeks to days; an autonomous loop refines parity column-by-column without manual reconciliation; and a non-destructive incremental pipeline preserves full history.
