Shipping the Operating Manual: What We Built to Make Our Data Mesh Usable
A Data Mesh with no operating manual is a platform only its engineers can use. Here's what we shipped to make ours serviceable: a formal RACI, a two-track access catalog, ten documented governance processes, and the pipeline service catalog that replaced the Slack messages.
This is part of a series about data culture on my team. An earlier post covered the gap between building data infrastructure and getting people to use it. This one is about the layer that has to exist underneath — the documentation that makes the platform actually serviceable.
The Platform Without an Operating Manual
There is a specific kind of problem you can only have if you’ve already solved a harder one.
By early 2026, our Data Mesh at MercadoLibre had reached genuine production scale: 288 tables, 237 published views, 150 scheduled jobs, running at under 1% error rate. Twelve months earlier, those numbers were 5, 5, and 5. We had built the infrastructure.
The problem we had arrived at: a platform that only its engineers could operate.
If you wanted to know which data was available, you asked someone on the team. If you wanted access to a table, you sent a Slack message and waited. If an alert fired at 3am, there was no documented path for who was responsible, what they were supposed to do, or when to escalate. The system ran — and remained, in a meaningful sense, a black box to everyone outside our own Jira board.
The operational documentation is the last thing you build, which means it’s always the last thing you think about. We built it. This is what it took.
The Architecture It Documents
Before describing the operating manual, it helps to describe what it’s documenting — because the architecture itself is more nuanced than the simple “consumers only touch Gold” framing you might expect.
Our Data Mesh follows the medallion model: Bronze ingestion → Silver conformation → Gold consumption-ready. But the consumption side is two-tier, not one.
Silver tables are accessible — but with column-level security (CLS) applied where the data is sensitive or PII-adjacent. Accessing a Silver table goes through a dedicated access request that requires co-approval from my team and takes up to 5 business days. The column-level security means a consumer gets access to the table but only to the columns they’re cleared for — so a team can query procurement patterns without seeing names, for example.
Gold tables surface as published views — exposed in the shared data warehouse, accessible through a standard access request, approved by my team alone, with a 3-business-day SLA. This is the path for most teams: clean, documented, sensitivity-managed at the view layer.
This distinction matters because “just read the Gold layer” isn’t always right. Some analytical questions require the fuller Silver schema, and the governance has to account for that rather than pretending it doesn’t exist. The access catalog is how that distinction becomes something a consumer can navigate without a data engineer on the phone.
What the Runbook Actually Contains
The governance runbook — published in Confluence and formalized in Sprint 1 of our documentation initiative — covers ten governance processes across the lifetime of a data asset on the platform.
The entry point is the RACI, which makes explicit something that previously lived in people’s heads: a small set of Data Owners, Data Stewards, and Contributors, with each of the ten processes mapped to specific responsible and accountable parties. It exists because the platform was built during a larger organizational change, and the people who built it won’t always be the people who run it. The runbook is the contract that survives that transition.
The ten processes are: pipeline maintenance, description enrichment, alert management, sandbox triage, data access, sensitivity classification, offboarding, monitoring, escalation, and onboarding. Each one has a documented owner, a documented trigger, and a documented output. Most of them existed informally before; writing them down forced precision about edge cases that had previously been resolved by judgment call.
A few highlights from what that precision revealed:
Severity is now defined, not felt. We had always known the difference between “this job failed and the table is stale” and “this job failed and it’s the only source for a critical executive dashboard.” We had never written it down. The runbook has a four-level severity table (P0 through P3) with explicit response-time targets: P0 incidents — complete data loss or a critical reporting gap — need a first response within one hour, not whenever someone happens to check chat.
Escalation is named, not implied. On-call engineer → data steward → data owners (myself included) → area leadership. Every person in that chain knows they’re in it. Before the runbook existed, the escalation chain was “whoever can be found,” which is a different thing.
Sandbox triage has a three-day rule. Any job in the sandbox that hasn’t been assigned a migration priority within three days of creation gets triaged out. This single rule eliminated about a third of the ambient debt that made the sandbox feel unmaintainable.
The Service Catalog: What’s Available, and How to Ask for It
The second major deliverable — distinct from the runbook but dependent on it — is the service catalog: the documented answer to “what data does my team publish, and how do I access it?”
Before the catalog existed, this question had one reliable answer: Slack me. After, it has two documented channels.
The standard channel is the path for non-sensitive data surfaced via published views. A request form, a 3-business-day SLA, my team approves. The catalog lists what’s accessible through this channel and what each dataset contains — operational data across our twelve domains.
The restricted channel is the path for tables with sensitive or PII-adjacent content that aren’t fully abstracted at the view layer — where a team legitimately needs Silver-layer access with column-level security applied. This channel adds a co-approval step and a 5-business-day window, because the sensitivity classification and the CLS configuration have to be verified before access is granted.
flowchart TD
R[Access request] --> Q{Sensitive or<br/>PII-adjacent?}
Q -->|No| STD[Standard channel<br/>published views · 3-day SLA<br/>team approves]
Q -->|Yes| RST[Restricted channel<br/>Silver + column-level security<br/>5-day SLA · co-approval]
STD --> G[Access granted]
RST --> G
The distinction isn’t bureaucracy for its own sake. It’s the difference between “we know what data is sensitive and why” and “we hope for the best.” Running our data-literacy program — built to grow data fluency across the organization — required us to be able to say with confidence that the data people were practicing with was handled appropriately. You can’t run a hands-on cohort session on tables whose access controls you haven’t formally documented.
The Pipeline Catalog: What the Sandbox Became
There is a third piece of operational documentation that sits below the runbook and service catalog: the pipeline catalog.
The sandbox was a collection of jobs, pipelines, and scripts that ranged from active production work to artifacts from three years ago. It contained roughly 85+ jobs when we began cataloguing it. It had no migration priority, no ownership mapping, and no documentation about which of its outputs were consumed downstream.
The catalog work changed that. Each job in the sandbox was assessed against a migration criterion: is it still active? Does it produce data anyone uses? Is it a candidate for promotion to the production Data Mesh? The three-day triage rule (above) applies to new arrivals; for the existing backlog, we worked through it domain by domain.
The output is a prioritized migration backlog that my colleagues can operate against without a weekly question to me about “what should we do next.” That handover — from implicit knowledge to documented process — is the whole point. A platform whose operations live in the heads of two people hasn’t finished being built yet.
What Writing It Forced Us to Learn
Documentation has a diagnostic effect. When you write down a process, you discover the places where the process doesn’t actually exist — where what you have is a pattern of behavior that three people have converged on without ever stating it.
The most useful thing about writing the runbook was not the document. It was what writing the document revealed.
We discovered that our low-quality description rate was at 81.9% when we started. This didn’t mean we had bad descriptions — it meant most tables had descriptions written for engineers who could infer context from column names, not for consumers who couldn’t. Writing the enrichment process down forced us to specify what “good” meant: a description that answers a consumer’s question without requiring them to read the schema. By the time the first phase of enrichment was complete, the rate had dropped from 81.9% to 15.8%. That’s a 66-point improvement that looks like metadata work. It’s actually governance work — the runbook made clear who was responsible for it and what the target was.
We also discovered that our Silver-to-Gold promotion wasn’t fully automated — there were tables in Gold whose Silver sources had schema changes pending. The view documentation made this visible because the service catalog entries for those Gold views had to accurately describe their columns, and we couldn’t write accurate descriptions until we’d verified the schema was stable. Documentation as test suite.
What “Shipped” Means for an Operating Manual
The runbook is Confluence Sprint 1 — the initial publication that establishes the RACI, the ten processes, the severity table, and the two access channels. It’s not done. An operating manual is the kind of thing that’s either continuously maintained or continuously outdated.
The model we’re running is a living document: every governance decision that changes a process updates the runbook, not just a team’s informal understanding. When we changed the sandbox triage window from five days to three, we updated the document and sent the diff to the team. When we promoted a new view into the service catalog, the access request template was updated before the announcement went out.
The test for whether an operating manual is working isn’t whether it was published. It’s whether a new Data Steward who’s never met the Data Owners can handle a P2 incident from start to resolution using only the documentation. We haven’t run that test in production yet. We’re close enough to it that when the next incident happens, we’ll know quickly whether the runbook is sufficient or whether it has a gap we need to fill.
Closing: Before the Data Is Ready
The temptation in data engineering is to treat the documentation as an afterthought — something you write once the “real” work is done. The operating manual, the service catalog, the access runbook: these feel like overhead on top of the pipeline work.
They’re not. They’re the last 20% of the pipeline work that determines whether anyone other than the builder can use what was built.
The Confluence runbook we shipped in Sprint 1 isn’t impressive-looking. It doesn’t have architecture diagrams or clever automations. It has a table of severity levels, two access-request URLs, and ten processes with named owners. But the teams consuming our data can now answer the question “how do I get access to this table?” without a Slack message. A teammate can triage a 3am alert without waking me up. Another knows exactly which domain’s descriptions they’re responsible for and what “done” means.
That’s what an operating manual does. Not exciting. Necessary.
Our Data Mesh serves an internal-systems organization. The governance runbook is maintained as a living document. The next post in this series covers our data-literacy program — built on top of this infrastructure to move the organization from data consumers to data practitioners.