The 5 Ds of Data Development

Discovery is the most important phase of the 5 Ds. Get this right and everything that follows is faster, cleaner, and more likely to land well. Rush it, or skip it entirely, and you risk spending weeks building the wrong thing. Or worse, building something that makes the problem harder to solve.

The central question is simple. What problem are we actually trying to solve? Stakeholders naturally think in outputs, like a dashboard or a data extract. The work of Discovery is to understand the outcome behind the request, and what will actually change once it exists.

Talking to stakeholders

The most valuable thing in Discovery is talking to the people closest to the problem, then listening for what they are not quite saying. A few techniques help:

• Ask them to show you what they do today. Rather than asking how they work, get them to walk you through it. You will often spot things they did not realise were relevant, and sometimes find the solution already exists.
• Ask what changes if they have this. The question “what will you do differently once this exists?” ties the request to a real decision or behaviour. If they cannot answer it, that is worth knowing.
• Use the 5 Whys. When you get an answer, ask why, and then ask again. The first answer is rarely the real one, and the problem someone presents is often a symptom of something further upstream.

Understanding the landscape

Discovery is also about understanding what already exists. Before designing anything, find out whether the problem has already been solved. What you find usually falls into one of three situations: something that exists and works, something that exists but is not trusted, or nothing at all. Each one changes the scope of the work, and it is far better to know which now than when you're halfway through Development.

The compromise

Discovery is the phase people most want to skip. The better response is to reframe it. Time spent here is risk reduction. An incorrect metric that gets used to make decisions is more dangerous than no metric at all. Discovery does not have to be endless, but it does need to be enough.

Discovery told you what problem you are solving. Design is where you work out how to solve it. It is the most technically demanding phase of the 5 Ds, but the most important thing you do here is not technical at all. Before models, layers or pipelines, you need to answer one question clearly. How does the output you are proposing lead to the outcome the stakeholder needs? If you cannot answer that, you are not ready to design yet. Go back to Discovery.

Start with the end

Begin by defining what the solution is, not how you will build it. It might be a data model with a defined grain and set of columns, a metric in the semantic layer, a report, or a connection between systems. Do not default to the familiar. The output should be whatever best solves the problem, not whatever you built last time. Then ask whether that output actually delivers the outcome. A well-specified output that solves the wrong problem is still the wrong solution.

Work backwards from where you want to be

Once you know where you are going, work out how to get there. What data already exists that you can build on? What needs building from scratch? What can be reused, and can what you build be reused by others? Where are the dependencies, and are any outside your control? This is rarely a straight line. You will start down one path, hit a blocker, try another, and hit another. Good Discovery is what reduces the dead ends. The more you understood the landscape upfront, the fewer you walk down here.

MAESTRO: the properties of a good pipeline

A pipeline cannot be perfect in every dimension. Its properties compete with each other, and good design is about naming the trade-offs explicitly rather than trying to maximise everything. MAESTRO is the model I use to have that conversation: Maintainable, Accurate, Efficient, Scalable, Timely, Robust, Observable.

Read the full MAESTRO breakdown

The wider questions

Design is where you ask the questions that touch everywhere. Be explicit about each:

• Governance and privacy. Can you use this data, does it need a privacy assessment, what access is required?
• Testing. How will you know it is good enough, and what will guard it going forward?
• Reusability. Are you building something bespoke, or something others could reach for?
• Deployment approach. A big bang rollout has different design implications than running old and new side by side.

Make it visible

Whatever you are designing, get it out of your head and into a form someone else can follow. A DAG diagram, a flowchart, a sketch. It does not need to be polished, it needs to be clear. It is both a communication tool and a thinking tool, and the person who comes back to it fresh in six months might well be you.

Development is the phase people think of as the job, the building, the code. It still is, but how we build has changed, and that brings a responsibility that is easy to overlook. The question to answer at the end of this phase is not “does it run?” It is “how do you know this is correct?”

How building has changed

Development used to be inherently iterative and human-led. Write a bit, run it, check it. A human eye was on the output at every step, so testing was woven in by default. AI tooling has changed that. Tools like Claude Code let you make larger changes in fewer steps, which is genuinely powerful, but the natural checkpoints that used to exist are no longer automatic. You can go from prompt to output without a single manual inspection in between, which makes it more important than ever to be deliberate about how you validate what you have built.

Validity and soundness

There are two questions to ask about anything you build, and most people only ask the first. Validity means the code runs. The syntax compiles, the references resolve, the model builds. That is the baseline, necessary but nowhere near sufficient. Soundness means the output is actually correct. Does it produce what you think it does? Stopping at validity is one of the most common failure modes in analytics engineering.

Proving soundness

Soundness is really two jobs, one for now and one for later.

• Validation confirms it is right today. Compare dev to prod, spot check records you know something about, check row counts and splits, and find an independent source that should show the same number.
• Automated tests confirm it stays right tomorrow. Constraint and data-quality tests guard structural integrity (uniqueness, not-null, accepted values, referential integrity, freshness), and logic tests cover your transformations.

Write the test plan before you build. Tests written afterwards make it too easy to confirm what you built rather than challenge it.

Ready for review

Review exists to catch what you missed, not to do your QA for you. If you submit a PR hoping the reviewer will find the problems, you have skipped a step, and in an AI-assisted world you have moved the problem rather than solved it. If you cannot demonstrate how you know it is correct, it is not ready for review.

Everyone complains about documentation. There is never enough of it, it is never clear enough, nobody can find what they need, and yet nobody wants to write it. It always gets paid for in the end, though. Either you invest in it as you go, or you pay later with interest, in onboarding time, in debugging sessions, and in you, six months from now, staring at a model and wondering why you made that decision. No one has ever looked back and wished they had documented something less well.

Two kinds of documentation

End-user documentation is for the people consuming the output. It is non-technical. What does this metric mean, what is this report showing me, and when something looks wrong, how do I debug it? Every question it answers is one that does not land in your inbox. Technical documentation is for the people building on top of your work. How does it work, what were the design decisions and why, what does someone need to know before they extend it? You need both. They serve different people, and neither substitutes for the other.

Do it along the way

If you leave all the documentation until the end, it shows. By the time you reach Deploy you are ready to ship, not write prose. The better approach is to let it accumulate across the other phases. The problem statement from Discovery, the design artefacts from Design, and the testing plan from Develop are all documentation. This phase is then about consolidating and completing, not starting from a blank page. Include documentation in your definition of done.

The AI dimension

Documentation matters more now, not less. When AI helps you build, it needs context. Without clear technical docs and consistent naming, you get generated code that reinvents what already exists or makes assumptions that do not hold. When an end user queries your data through an AI interface, that AI needs to understand what the data means, or you get confident answers that are subtly wrong. The audience for good documentation has expanded beyond humans.

Deploy is where your work meets the real world. The building is done, the tests pass, the documentation is in place. Now you need to get it into production without breaking what is already there. This is fundamentally a change management problem, and change management is one of the most consistently overlooked parts of analytics engineering.

Know what you might break

Before you deploy anything, understand what exists downstream. Which models depend on what you are changing, which reports are built on those models, and which teams rely on those reports to make decisions. This should not be a surprise at this stage. You should have identified downstream dependencies during Discovery and Design; Deploy is where that knowledge becomes operational.

How you deploy is a design decision

A big bang replaces the old with the new in a single deployment. Clean and simple, and sometimes the right call for smaller changes, but if something is wrong, everything is wrong at once. A parallel deployment runs old and new side by side, migrating downstream dependencies gradually. It is more work, but significantly lower risk, and it lets you compare old and new outputs directly, which loops straight back to the soundness question from Develop.

Deploy in phases, keep the picture visible

For larger work, it is rarely sensible to deploy everything in one enormous PR. Smaller, reviewable chunks reduce risk, but they create their own problem. The reviewer only sees a piece of the puzzle. The fix is simple. Keep an overall plan in a ticket that lists every PR in the change, which have merged and which are still to come, and link every PR back to it. A “you are here” for your deployment.

Tell people what is coming

Technical deployment is only half of Deploy. If your changes affect downstream users, and good Discovery and Design mean you know exactly who they are, they need to know what is coming, why it matters, and what they need to do. You are the one making the change, so it is your job to make it as easy as possible for the busy people you are affecting. Tell them the change, the reason, the timeline, and the exact steps.