The signal for people who keep the data flowing_
One curated weekly issue: orchestration war stories, dbt deep dives, and the DataOps tooling takes your Slack channel won't shut up about — with the vendor noise filtered out.
When Airflow Lies: Debugging Zombie DAGs in Production
The 2 AM page that wasn't Airflow's fault (it was)
A scheduler heartbeat timeout masked a silent executor memory leak for 6 hours.
dbt-unit-testing vs. dbt-expectations: A field guide
After running both in prod for 90 days, here's what the benchmarks don't tell you.
Monte Carlo vs. Great Expectations: Total cost of ownership
The license fee is the smallest line item. Engineering time is where it hurts.
Browse the archive
47 issues. Every tool, failure, and pattern that matters to people who build production data systems.
Airflow 2.7 KubernetesExecutor: What changed and why it broke your DAGs
Pod template changes in 2.7 silently invalidate resource configs you've had in prod for two years.
Prefect vs. Dagster in 2026: An honest comparison from someone who ran both
Not a feature matrix. A real account of migration cost, observability gaps, and on-call experience.
The hidden cost of dynamic task mapping in Airflow
XCom bloat, scheduler lag, and the memory patterns nobody puts in the docs.
The people who keep this going
Pipeline isn't one person. It's a community of engineers who annotate issues, flag vendor BS, and share what actually worked in their stack.
Stack of the Week
Each week we diagram a real production stack, annotate the failure modes, and explain the trade-offs nobody puts in the vendor docs.
Airflow XCom serialization at >50k rows causes scheduler memory pressure
Move large payloads to S3, pass only the key through XCom. Cuts memory 80%.
Issue #47 covers the full migration path with Helm config and DAG rewrites.
Join 12,000+ data engineers
Every Thursday. No vendor fluff. Just the signal your on-call rotation actually needs.
No spam. Unsubscribe with one click. Read by engineers at Stripe, Shopify, Airbnb, and 200+ more.
Engineers at these companies read Pipeline