From Scripts to DAGs: Architecting Predictable Multi-Step Pipelines

A pipeline written as a script is a promise that nothing will go wrong. Modeling it as a directed acyclic graph is what you do after something does.

Every multi-step pipeline starts as a function. Fetch the data, clean it, score it, summarize it, store the result, notify someone. Six steps, top to bottom, easy to read. It works on the first day and it works on the hundredth.

Then step four calls a model that times out. The retry re-runs all six steps, including the expensive ones that already succeeded. A deploy restarts the process mid-run and the pipeline simply vanishes — no partial result, no record it ever started. Someone adds a seventh step that quietly depends on a variable step two happened to leave lying around, and now the order of the file is load-bearing.

The fix isn't a better script. It's changing what the code is: from a sequence of instructions into a declared graph of steps with declared dependencies. Once the shape of the work is data rather than control flow, an engine can schedule it, parallelize it, retry it, cache it, and resume it — and none of that logic has to live in your business code.

The problem — A script hides its own structure

An imperative pipeline has a dependency graph. It just never writes it down. The graph lives implicitly in the order of the lines and in whatever variables are in scope, which produces four recurring failures:

• Retries are all-or-nothing. The unit of failure is the whole run, so the unit of retry is the whole run. A flaky call in the last step re-pays for every step before it.

• Nothing is resumable. Process state is the only record of progress. Lose the process — a deploy, a crash, a scale-in event — and you lose the run.

• Parallelism is manual. Steps three and four may be fully independent, but only a human reading the file knows that. So they run one after the other, forever.

• Coupling is invisible. Because dependencies aren't declared, nothing stops step six from reading something step two happened to produce. Reordering the file becomes a gamble.

"If the dependency graph isn't written down, it isn't a design — it's a habit."

The approach — Declare inputs, derive the graph

The core inversion is small and it changes everything: steps don't call each other. Each step declares the named inputs it reads and the named outputs it writes. The engine matches writers to readers, and that is the edge set. Nobody hand-draws the graph, and nobody hand-orders the steps.

A step declaration carries everything the engine needs to run it safely:

@node(
    name="score_signals",
    reads=("clean_signals", "baseline"),    # edges are derived from these
    writes=("signal_scores",),
    cache_key=("subject_id", "day"),        # identity of the work, not of the call
    retries=Retry(attempts=3, backoff="exponential", jitter=True),
    budget=Budget(max_model_calls=1, max_seconds=30),
)
def score_signals(ctx, clean_signals, baseline):
    ...
    return {"signal_scores": scores}

Four properties fall out of that declaration, and each one removes a class of bug:

• Acyclic is checked, not assumed. Because the graph is data, it's validated when the pipeline loads — a cycle is a startup error, not a hang in production at 3 a.m.

• Inputs are the only inputs. A node receives exactly what it declared. Reaching for undeclared state doesn't work, so the accidental coupling in Figure 1 becomes impossible rather than discouraged.

• Every node has a stable identity. The cache key names the work — this subject, this day — not the invocation. Identity is what makes memoization and idempotency possible at all.

• Policy lives beside the step, not inside it. Retries, timeouts, and budgets are declared metadata the engine enforces. The function body stays pure business logic with no error-handling scaffolding.

The process — What a run actually does

Once the graph exists, execution is mechanical. A run proceeds in layers: the engine takes every node whose inputs are satisfied, runs that whole set concurrently, records the outputs, and repeats. Depth of the graph sets wall-clock time; width sets throughput.

Three mechanisms turn that loop from "a parallel for-each" into something you can trust in production:

Checkpoints. Each completed node's output is written to durable storage keyed by run id and node identity. The run's progress lives outside the process, so a restart resumes from the last completed layer instead of starting over. Long pipelines stop being hostages to deploy windows.

Memoization. Before running a node, the engine checks whether that exact node identity — same code version, same declared inputs — already has a recorded result. If it does, the node is skipped. This is what makes a retry cheap and a re-run nearly free, and it's only sound because identity was declared up front.

Budgets. Any node that touches something expensive — a model call, a third-party API, a heavy query — declares a ceiling. The engine counts against it and fails the node loudly when it's exceeded, rather than discovering the overrun later on an invoice. Fan-out is where cost surprises live, so the guardrail belongs at the node, not at the pipeline.

Failure handling then has three distinct outcomes instead of one:

Transient failure (timeout, throttle) → the engine retries the node with backoff and jitter → blast radius: 1 node.

Permanent failure (bad input, logic bug) → the engine fails the node, skips its descendants, and keeps the rest of the run → blast radius: 1 branch.

Infrastructure failure (restart, scale-in) → the engine resumes from the last checkpoint; completed nodes read from cache → blast radius: 0 completed nodes.

Results — What changed

The wins are structural, which means they hold for every pipeline built this way rather than being tuned in one by one:

• Retry cost dropped from O(pipeline) to O(node). The most common production failure — a transient timeout on an external call — now re-does one step.

• Runs survive deploys. Because progress is checkpointed outside the process, a rolling restart no longer forces a choice between waiting for drain and abandoning in-flight work.

• Independent work runs in parallel for free. Nobody schedules the fan-out; the engine reads it off the declared edges. Adding a branch is a declaration, not a concurrency exercise.

• Re-runs are cheap enough to be routine. With memoization on stable node identities, re-running a pipeline to pick up one changed step doesn't re-pay for the rest.

• Pipelines became inspectable. The graph is data, so tooling can render it, diff it across versions, and answer "what does this step depend on?" without anyone reading the source.

Four numbers are worth instrumenting from day one, because they're the ones that tell you whether the design is paying off: wall-clock time per run (should track graph depth, not step count), redundant work ratio (cache hits ÷ node executions), resume rate (runs that survived a restart), and cost per run against the declared budget.

Where a DAG is the wrong tool

A graph is worth its overhead when steps are expensive, independent, or failure-prone. For three cheap sequential calls, a function is the better engineering choice — declaring a graph to run one path through it is ceremony. And a DAG models dependencies, not conversations: work that loops until a condition is met, or waits indefinitely on a human decision, wants a different primitive.

Where this fits at Hoomanely

Hoomanely builds technology that helps people care for the pets who depend on them — turning everyday signals into earlier, kinder, better-informed decisions about an animal's health. Turning raw signals into a useful insight is inherently a multi-step pipeline: gather, clean, compare against what's normal for that animal, interpret, and deliver something a person can act on. Modeling that as a declared graph is what lets those steps be added, reordered, and improved safely — and what makes sure a single slow external call never costs a family the day's insight.

Key takeaways

• Write the graph down. A script has a dependency graph either way; declaring it is what lets an engine schedule, retry, and parallelize on your behalf.

• Declare inputs, not order. Derive edges from declared reads and writes, and undeclared coupling stops being possible instead of merely frowned upon.

• Identity enables everything. A stable node identity is the prerequisite for memoization, idempotency, and cheap retries — declare it first.

• Checkpoint outside the process. Progress that lives only in memory is progress you lose on the next deploy.

• Bound the expensive nodes. Budgets declared per node catch cost overruns at the moment they happen, where fan-out makes them easy to miss.

Author's note — The hardest part of adopting a DAG isn't the engine; it's giving up the comfort of reading a pipeline top to bottom. What you get back is a pipeline that can be reasoned about by a machine — and one that keeps running when the machine underneath it goes away.