Essay 01
What analytics engineering borrows from software engineering - and where it stops helping
In recent years the data world has taken a lot from software engineering - especially the software development lifecycle. For me it started with Looker: a semantic layer in code, even dashboards as code. Soon after I was building transformations in dbt. I didn't want to go back to drag-and-drop UIs.
Version control, pull requests, CI, tests, staging environments. Those things brought discipline. They made it easier to release with confidence, and much easier to collaborate on complex projects. The practices had been refined through huge investment in a more mature field. In hindsight, of course it made sense to borrow heavily.
I know people in both data and software who think the answer is to keep copying - that analytics skills are basically a subset of software skills. I don't think that's right. Somewhere along the way we forgot that data and software are not the same job. What we're trying to achieve is different enough that it needs a different mindset, process, and set of tools - even when we borrow the scaffolding.
So what's different?
The clearest example for me is testing.
In software engineering, unit tests are often highly effective because inputs and expected outputs can be tightly controlled. If a function passes a serious suite, there's a reasonable expectation it will keep behaving unless the code changes.
Data systems aren't like that. The code can stay the same while the underlying data evolves in ways that quietly invalidate the assumptions. A transformation can pass every unit test, every uniqueness check, and every referential integrity check, and still produce the wrong business outcome - because the shape, distribution, or meaning of the data moved underneath it.
I've seen teams treat a green CI run as permission to trust a number in a meeting. The pipeline was fine. Reality had drifted. That gap isn't fixed by adding one more schema test. It needs a different kind of attention:
- whether distributions shifted
- whether a metric suddenly moved
- whether an independent source still agrees
- whether the people who use the number still recognise it
Unit tests and data quality checks aren't pointless. They're essential foundations. The mistake is assuming that enough coverage creates the same confidence in a warehouse that it often does in application code.
Software engineers test whether code behaves as expected. Analytics engineers also have to keep asking whether reality still matches the assumptions encoded in that code.
What might we have missed?
Difference doesn’t mean borrow less of everything. It means be picky - and I think we were picky about the wrong half. We've borrowed a lot of the software lifecycle. I actually think we've under-borrowed the design discipline that sits underneath it - separation of concerns, clear responsibility boundaries, opinions about where different kinds of work belong.
dbt is a useful example. It brought enormous value by making SQL transformations version-controlled, testable, reviewable, and deployable. Its deliberately unopinionated nature was also part of the appeal: get out of the way and let teams build. The cost is that many teams recreate problems software engineering learned long ago to avoid.
A dbt model can become anything:
- a staging layer cleaning raw data
- a business entity
- a complex intermediate table
- an aggregate table
- a reporting-specific output
The software equivalent would be a class that handles database access, authentication, business rules, formatting, and sending email. You can do it. Mature teams developed patterns specifically so everything doesn't end up in one place with one name.
A developer using something like Django gets more opinions about where code belongs, how components interact, naming, and testing. The point isn't that frameworks make people less creative. It's that they stop every team reinventing a half-worked architecture from scratch.
Writing the "right" architecture in a Confluence page doesn't really solve it either. You can't reliably depend on every analyst and engineer reading it, remembering it under deadline pressure, and applying it the same way six months later.
Nor should every organisation have to invent, from scratch, all the elements of a sound analytics architecture. Two organisations could both decide on "dimensional modelling" but independently decide:
- folder structure for layered architecture
- how far filtering and deduping go in the first layer
- naming conventions for tables and columns
- if and how to represent model ownership and domains
- materialisation strategy
- (and much, much more!)
That cost gets paid again and again across the industry. I've seen enough different flavours to know that whatever was gained by making it "work for our unique business" wasn't worth the effort.
Imagine if something in the tooling said: this model appears to combine source cleaning, business meaning, and aggregation - those are different jobs. That's closer to the design discipline I mean than another layer of _int_ tables with the same confused responsibilities under new names - or another guide nobody opens.
Looking forward
Software engineering is mostly about building a system you control. Analytics engineering is mostly about representing a world you don't.
Keep the discipline we borrowed - versioning, review, tests, CI, ownership. Borrow more of the design habits too. But don't pretend a tidy repo settles meaning when the organisation keeps changing and the warehouse stays messy.
Analytics engineering doesn't need to become software engineering. It needs to grow up as its own craft: take what transfers, and be honest about the problems software practices were never meant to solve.