Python Logging That Actually Helps
Most advice stops at 'use structured logs'. The harder part is deciding what's worth writing down before an incident makes that call for you.
I've read a lot of logs that told me a job failed and nothing else. The fix was almost never a new library. It was deciding, up front, who reads this line and what they're supposed to do about it.
Write for whoever gets paged
Every line costs something to read later, so before adding one, answer who it's for: you debugging locally, an auditor six months out, or whoever is on call tonight. Those three want different things, and a line that tries to serve all three usually serves none. Debug lines can be chatty. On-call lines should name what broke and what to check next.
Structured, but not JSON everywhere
The standard library already gives you extra, and a JSON formatter on top of it gets you queryable fields without touching a single call site: log.info('batch loaded', extra={'table': 'orders_silver', 'rows': 41233, 'run_id': run_id}). Keep plain text in local dev, though. Nobody wants to pipe their unit tests through jq.
Context beats events
A line that says Load failed is a dead end. The same line carrying a run id, the source partition and the row count that made it through tells you where to look without opening the pipeline code. LoggerAdapter or contextvars will push that context down the stack, so you're not threading a run_id argument through nine functions to get it there.
import contextvars
import logging
# Set once per run; every log line below picks it up without passing it around.
run_id = contextvars.ContextVar("run_id", default="-")
class RunContext(logging.Filter):
def filter(self, record):
record.run_id = run_id.get()
return True
handler = logging.StreamHandler()
handler.addFilter(RunContext())
handler.setFormatter(logging.Formatter("%(levelname)s run=%(run_id)s %(message)s"))
log = logging.getLogger("pipeline")
log.addHandler(handler)
log.setLevel(logging.INFO)
def load_partition(table, partition):
rows = 41233 # whatever the load actually returned
# A JSON formatter would turn these extra fields into queryable columns.
log.info("batch loaded", extra={"table": table, "partition": partition, "rows": rows})
run_id.set("2026-05-05T04:00")
load_partition("orders_silver", "2026-05-04")
# INFO run=2026-05-05T04:00 batch loadedLevels only work if you defend them
WARNING creeps. Something looks odd once, someone bumps it up to be safe, and a year later the warning stream is four hundred lines an hour that everyone filters out. Worth auditing every few months: anything nobody has ever acted on gets demoted or deleted. And retries belong at DEBUG until the last one fails.