Building a Data Mesh Platform from Zero

The situation

At the start of 2025 I inherited a Data Mesh Environment (DME) that existed on paper but not in practice: a governed BigQuery environment with the right architecture, governance model, and tooling — and almost no data products inside it. The business unit’s actual data still lived in a legacy sandbox: unmonitored stored procedures, undocumented tables, and a handful of dashboards no one fully trusted, accumulated during years of rapid capex/opex growth.

My mandate was to build the foundation: take six business domains from that sandbox into a production-grade platform that teams could query without asking me for a JOIN.

The constraints

This wasn’t a greenfield where I could design the perfect system in isolation. Every migration had to:

• Run against live operational sources across six business domains without disrupting them.
• Replace systems people already depended on — which meant the hard part was rarely the SQL. It was building the case that the new system was better than the old one.
• Handle sensitive data safely from day one.
• Be maintainable by a small team, not just by me.

The approach: a medallion architecture, applied consistently

Rather than one-off pipelines, I standardized every domain on the same Bronze → Silver → Gold pattern, with each layer in its own dataset, enforcing audit fields, running on scheduled jobs with automated alerting.

flowchart LR
  S1[Operational sources<br/>SQL Server · MySQL · REST APIs]
  B[Bronze<br/>raw extraction<br/>no transforms]
  SI[Silver<br/>typed · validated<br/>business-enriched · PII-safe]
  G[Gold<br/>aggregated data marts]
  C[Consumers<br/>Looker Studio · self-service SQL]

  S1 --> B --> SI --> G --> C

The discipline mattered more than the diagram. Bronze is dumb — raw extraction, no transformations. Silver is typed and business-enriched. Gold is aggregated for consumption. Because each layer can be rebuilt independently, a source schema change only touches Bronze — not the marts leadership depends on.

What shipped

Six domains, each following the same pattern but with its own challenge:

• An asset-management domain — the canonical migration. A 192K+ device inventory came off an unversioned, unmonitored legacy stored procedure into 29 governed tables and a wide mart with 240+ columns any team can query. The legacy procedure was retired.
• A high-stakes findings domain — 28 tables, daily refresh. A Gold mart pre-calculates risk scores, aging buckets, and compliance rates, so analysts stop filing custom requests to the owning team.
• A freshness-critical domain — 22 tables refreshed five times a day for near-real-time operational data.
• Ticketing, people-data, and compliance domains — the long tail. The ticketing domain meant rewriting years of accumulated business logic; the people-data domain drove a sensitive-field masking framework that became the reference for every later pipeline; the compliance domain arrives via REST API, so Bronze needed a custom Python extractor.

Governance, the invisible half

Building pipelines is the visible part; trust is what makes them used. A mid-year audit flagged 145 production tables as low-quality — descriptions missing or too generic to be useful. I ran a systematic enrichment project to generate business-context descriptions across all seven domains and track the quality score monthly, measured on three axes: Discoverability, Efficiency, and Integrity.

Results

• 203 production tables, 162 published views, 93 active scheduled jobs by year’s end — none existed in January.
• Six domains moved from an ungoverned sandbox to a monitored, alerting, self-service platform.
• A brittle legacy stored procedure retired, with its 192K+ device inventory now governed and self-service.
• Self-service replaced request queues: consuming teams answer their own questions without routing through a source-system owner.

What I’d do differently

Communicate value earlier. I spent the year building and very little time explaining what the pipelines enabled — by the time I had 200+ tables in production, the value was obvious to me and invisible to everyone else. Architecture without narrative is infrastructure without users.

Ship smaller. The pull in a greenfield environment is to design the perfect system up front. The better move — which we learned partway through — is to get one domain to production, learn from it, then apply those lessons to the next.

Table and schema names are generalized; metrics reflect real outcomes from my team’s work.