MAESTRO
The properties of a good data pipeline.
No pipeline can be perfect in every dimension. The properties of a good one compete with each other, and good design is about naming the trade-offs explicitly rather than trying to maximise everything.
They also compete with the deadline. Every property costs time, so the question is rarely how good this could be. It is what you are willing to not do, and what good enough looks like.
The properties themselves are not a discovery. Anyone who has run a platform long enough arrives at some version of this list. What MAESTRO adds is the insistence that they compete, and that you say out loud which ones you are trading away.
How to read this
The seven properties are the same for every pipeline. The bar is not. Each check below is tagged with the models it applies to, because a staging passthrough has no business logic to unit test, while a model a customer sees needs reconciling against source.
Most tags name a position in the pipeline, running from source through staging and intermediate to marts and finally exposed. Exposed means any model something outside the warehouse consumes, whether that is a product, a dashboard or another team's system. It is usually a mart, but not every mart is exposed, and a model further up can be if a tool reads from it directly. The remaining tags scope by size or materialisation instead, like large models or incremental ones.
Maintainable
Can the next person change it safely? A model you cannot understand is one you cannot change without fear. Maintainability is what makes it legible to someone who did not write it.
The trade
It is the property most often traded away for the others. The incremental logic that makes a model scale is materially harder to read than the full rebuild it replaced.
How tangled the lineage is can be measured, and it takes two numbers rather than one. The first is the number of paths back to source, averaged across the sources the model draws on. The second is the minimum cut, the smallest set of models that intersects every path to source. Averaging matters, because it separates how tangled each route is from how many routes there are. When both are low the lineage is a clean chain. When the average is low and the cut is high, the model is genuinely combining from many places, and may be doing too much. When the average is high and the cut is low, there are pockets of rejoining logic sitting over a choke point, and that is where simplification pays. When both are high, the pipeline is spaghetti.
In practice
A model description in YAML
The purpose of the model is written down, not kept in memory.
Grain and caveats stated in the description
The grain (one row per what?) is the single most useful fact about a model, and the caveats stop it being misused.
Every output column documented
A consumer should not have to read SQL to know what a column means.
Layer and naming convention followed
Predictable names and locations make the model findable and its role obvious.
Lineage kept close to a single path to source
The more tangled the path to source, the harder impact is to trace and bugs are to find. A perfect single path is impossible with shared foundation tables, so the aim is to keep it close to one.
Accurate
Does it produce the right output? Correct within the tolerance agreed in Discovery, and staying correct as the code changes.
The trade
Source data is often bad, and human-entered data never fully cleans up. Every edge case you chase is another branch in the model, so the question is not how accurate you can get but where you stop.
The checks here are two different kinds. Most are enforced continuously, so once written they defend the model on every build. The last two are not, because proving a number against reality needs something outside the warehouse to compare against. That makes it a point-in-time exercise, and it decays whenever a business definition changes or a source system moves.
In practice
Not-null and uniqueness on every primary key
Proves the grain holds. Nulls or duplicates in the key are the most common data bug.
The key is the true grain, not a convenient surrogate
A generated surrogate id is unique by construction, so the test passes even when the real grain may be broken. The test only means something on the columns that genuinely define a row.
Tests on the critical logic fields
The inputs to important logic are guarded, so a silent upstream change is caught.
Not-null tests on the fields that must be populated
The primary key is not the only field that has to be there. A required attribute that arrives empty is a silent gap until a not-null test makes it loud.
Accepted-values and range tests on constrained fields
A status column should only hold the statuses that exist, and an amount should sit in a sane range. These catch values that are wrong even when they are present.
One-off validation done and recorded
Correctness was proven against reality once and linked, not assumed.
Source-to-output validation
Every source is accounted for at the output, with no unexplained drop or duplication along the pipeline.
Efficient
Does it run within acceptable time and cost? It does not waste compute or reprocess more than it needs to.
The trade
A view is cheap, simple and always fresh, about as efficient as a model gets today. But nothing is precomputed, so as the data grows and the queries pile up, it stops scaling.
In practice
A materialisation that fits the size and use of the model
The wrong choice wastes compute, or serves expensive or stale results.
Build time and compute cost within an agreed ceiling
One runaway model slows the whole run down and drives costs up.
Scalable
Will it keep working as it grows? It holds up as data volume, grain and complexity increase, not just at the size it is today.
The trade
You pay for that in efficiency and in legibility. It means more storage, more moving parts, more that can drift, and a model the next person has to work harder to follow.
In practice
Incremental builds with a unique key on large models
Big models cannot full-rebuild every run and stay affordable.
Incremental used only where warranted, with correct merge
Incremental adds complexity, so it should be the right call and deduplicate correctly.
Correct partitioning or distribution
Storage layout decides how much data the warehouse has to read to build or serve the model. The right strategy is specific to the engine, so it depends on knowing how yours physically stores and reads data.
Timely
Is the output fresh enough? Current enough for the decision it supports, with the freshness expectation made explicit.
The trade
Freshness competes with accuracy. Build early and the number is fresh, but the late-arriving data has not landed. Wait for it and the number is more complete but staler. How stale you are happy for the data to be is the real decision.
In practice
Freshness configured on every source
Staleness is only detectable if each source declares how fresh it should be.
Documented freshness or SLA for the output
Customers need to know how current the data is supposed to be.
A cron tier that matches the SLA
An hourly SLA on a weekly model run will always be stale.
Robust
Does it fail loudly, not silently? Late, missing, malformed or duplicate data is either handled or fails loudly, rather than silently corrupting outputs, and re-running is safe.
The trade
That choice costs you something either way. Absorb the problem and you stop noticing it happens. Stop for it and you serve nothing at all.
In practice
Idempotent re-run and backfill
Re-running, backfilling or replaying must not double-count or drop rows. Given the same inputs and the same logical run time, the output should be identical, which usually means parameterising the run timestamp rather than calling current_date inline.
A volume or row-count test that trips on empty output
The model cannot silently drop to zero rows.
Handles upstream schema change gracefully
When a source adds, drops or renames a column, it should be absorbed safely.
A published contract where data has consumers
A contract fixes the columns and types a consumer can build on, so they cannot change underneath them without the change being deliberate. Without one, an upstream edit breaks the consumer silently.
Observable
Do you know whether the other six hold, or do you only believe they do? Observable sits over the rest rather than beside them, because a pipeline that is accurate but unobserved is accurate by luck. The floor is that when something breaks, the team hears about it first and not a customer. Above that sits the harder question of whether you would notice a failure you never thought to check for.
The trade
Every check you add is more to maintain, and monitoring is the easiest thing to let slide. It produces nothing anyone consumes, so when it stops working, nothing complains.
The hard part of anomaly detection is telling an unexpected change apart from the normal variation you would expect anyway. Data moves for legitimate reasons all the time, so a detector that fires on every shift is noise. It also belongs close to source, because further downstream the transformation logic adds too much of its own movement for an anomaly to stand out.
In practice
Data-side failures and staleness alert the team, not an end user
When a source goes stale the build fails, or a volume test trips, the team is paged before a customer notices. The checks themselves live in Timely and Robust, which is the point. Observable is not a separate set of tests but the requirement that the existing ones are heard.
Test severity is calibrated, error vs warn per test
Errors halt what must halt, warnings flag the rest, so genuine failures alert or block while low-stakes checks do not drown that signal.
Anomaly and drift detection on what flows through
Declared tests only catch the failures you thought of. Watching what actually moves through the pipeline, and how it changes over time, catches the ones you did not, whether an acute break or a slow chronic drift.
How MAESTRO assesses a pipeline
A pipeline is a target model, the one a product, BI tool or other exposure consumes, together with all its upstream lineage back to source. This is scoped to the warehouse, the dependent models that transform data once it has landed. The pipelines that feed the warehouse in the first place need these same properties too, but reach them by different mechanisms, so they sit outside MAESTRO.
You assess a pipeline through the models it contains, and it is ready when each clears the bar appropriate to where it sits.
Governance and security sit outside it as well. Access controls and how PII is handled are set by platform-wide guidelines and carried in metadata rather than defined per pipeline. MAESTRO covers the properties of the data and its build, not the controls around it.
Agreement between pipelines sits outside MAESTRO too. Whether two pipelines should both exist, and whether they produce the same answer where they overlap, is a Discovery and Design question, and it is where the 5 Ds pick it up.
You will rarely achieve all seven at the highest level, and that is the point. Being explicit about which ones you traded, and why, is what separates a thoughtful design from one that just happened to work.
MAESTRO sits inside the Design phase of the 5 Ds of Data Development.
Need your pipelines brought up to production grade?Get in touch