Data Engineering Best Practices
Five things I keep coming back to. None of them are clever, which is more or less the point.
1. Build for next year, not the next decade
Pick the simplest thing that survives the next twelve months. You'll be wrong about year three anyway, and the architecture you built for it will still be sitting there, still needing upgrades, still owned by someone who wasn't in the room when you chose it. Most of the complexity I've had to unwind was added for a scale that never arrived.
2. Instrument the boundaries, not everything
You don't need a metric per function. You need to know, at every hop where data crosses a system boundary, how many rows went in, how many came out, and when. That's three numbers and a timestamp per stage, and it answers most of the is-it-broken-or-just-slow questions before anyone opens a notebook.
3. Quality checks belong in the pipeline
A test suite on its own schedule tells you about yesterday's problem. Put the assertions in the same run as the transformation and let them fail it. That feels aggressive for about a week, and then you stop shipping bad data downstream, which was the trade you wanted.
4. Write down why, not what
The code says what. It never says why the currency conversion happens before the dedup, or why one source gets a 48-hour late-arrival window and the others don't. Those get decided once, in a thread, and then live in someone's head until they leave. Three lines of comment on the weird part beat a wiki nobody opens.
5. Make the loop short enough to use
If checking a change means waiting for the nightly run, people stop checking. Sampled data, a local runner, whatever gets it under ten minutes. The teams I have seen improve fastest weren't the ones with the best documentation. They were the ones who could try something before lunch.