Dagster Pipelines Need Clear Ownership
Nobody's real pipeline problem is the shape of the DAG. It's that a job failed at four in the morning and nobody knew whose it was.
Start from the asset, not the schedule
Dagster's asset model earns its keep when you define the thing you're producing and let the scheduling fall out of it. Source, output and the assumptions in between end up in one definition, instead of split across a DAG file and a cron entry nobody remembers writing.
An alert should say what to do
Run 4821 failed isn't an alert, it's a notification. The useful version names the asset, the upstream dependency most likely at fault, and the first thing to check. Most of that can live in the asset metadata once, instead of being typed into Slack by hand every time.
Every asset gets a name attached to a person
Owners in the definition, not in a spreadsheet somebody maintained until last March. It sounds bureaucratic right up until the first alert routes itself to whoever changed the thing. That does more for reliability than another dashboard full of red squares, mostly because dashboards don't wake anyone up.
import dagster as dg
@dg.asset(
deps=["orders_bronze"],
owners=["team:data-platform", "jeremias@example.com"],
description="One row per order, deduplicated, amounts in EUR.",
metadata={
"on_failure": "Check that orders_bronze landed today before rerunning.",
"consumers": "finance dashboard, churn model",
},
)
def orders_silver() -> None:
...Make local runs cheap
If a developer can materialise the slice they touched against sampled data and see a result in a couple of minutes, they'll do it before opening the PR. If it takes an hour and a cluster, they won't. Most of the incidents I've watched trace back to a change nobody could easily try.
Delete the schedules nobody reads
Every orchestrator I've inherited had jobs running that no one could explain. They cost money, they generate alerts people learn to ignore, and they make the graph unreadable. Once a quarter, find what nothing consumes and switch it off. Someone complains within two weeks, and that's how you find out it mattered.