The Migration Contract: Proving Every Row, Recovering Every Failure

What a cross-account DynamoDB migration taught us about the gap between "validated" and "true."

A dry-run validates your read path. Only the cutover validates your write path. Everything expensive lives in the gap between the two.

We learned this while moving a few hundred DynamoDB tables and millions of items into a fresh AWS account with a redesigned schema. This wasn't a lift-and-shift: every row was transformed, tenant-scoped, and written through a new repository layer. And the data wasn't disposable — it carries the feeding history and health signals of our users' pets, which set the bar for how this migration had to behave.

The architecture wasn't the difficult part. The difficult part was proving that every row could be moved, every failure could be recovered, and a second run could not make things worse.

That became our migration contract.

1. The Problem: Migrations Fail in the Mundane

Nobody loses data to the failure they rehearsed. Migrations fail on a duplicated attribute name, a list where a string should be, a credential that expires while everyone sleeps. Individually trivial; at bulk-write scale, each one is a halted job at an unknown cursor position against a half-populated target.

So the real engineering problem is not "how do we move the data." It is how do we make every failure recoverable and every success provable. A migration you cannot resume, re-run, and independently verify is not a migration; it is a bet.

2. The Contract: Fixed Before Any Script Was Written

The contract wasn't written because these failure modes were theoretically interesting. It was written after we discovered how many ways a migration could be "working" while still being wrong. Every job, regardless of table or author, had to obey five clauses:

• Cross-account by construction. Read the source through an explicit AWS profile; write the target only through the application's repository layer. The repository is where the new schema's invariants live — key derivation, tenant scoping, stamped attributes, validation. Bypassing it for faster batch writes doesn't skip ceremony; it seeds the new environment with silent schema drift on day one, recreating the exact disease the migration was meant to cure.
• Resumable. Checkpoint the scan cursor and counters after every page. A killed job continues where it stopped.
• Idempotent. Deterministic target keys, put_if_absent or upsert. A re-run is a no-op.
• Dry-run first. Every script walks the source and writes nothing before it ever writes anything.
• Provable. Every job defines its independent completion proof before execution — counters, stable target counts, content hashes, second-run semantics. The golden rule: run every backfill twice — the second run must prove itself a no-op.

Once the contract was explicit, the migration stopped looking like a collection of scripts. Each job became a small state machine — scan, transform, write, checkpoint, verify — with failure states that had defined recovery paths. That reframing, more than any individual script, is what the rest of this post defends.

3. The Scenarios: Where "Validated" Met Production

Scenario 1 — The dry-run that lied

Our first-wave scripts were marked dry-run validated days before the real run — which crashed on the first item. The target repository stamps created_at inside a DynamoDB UpdateExpression; legacy rows carried their own. Same attribute, same expression, twice:

ValidationException: Two document paths overlap with each other; path one: [created_at], path two: [created_at]

A dry-run exercises source traversal, transformation logic, and accounting. It does not exercise the target's actual write semantics — conditional expressions, repository-stamped attributes, index constraints. The fix was two lines. The lesson was structural: push a small set of real rows through the real write path before any bulk run.

Scenario 2 — The column that lied about its type

Same evening, second crash: a GSI key expected a string, and the legacy attribute was a string on almost none of the rows — a single-element list on the vast majority, an empty list or absent on the rest. Years of schema drift, invisible in any hand-inspected sample: every row we had happened to look at was fine. Before fixing anything, we scanned the entire column and histogrammed its DynamoDB types. The survey proved no multi-owner lists existed — turning the fix into two safe lines instead of a product decision made blind. Samples reassure; surveys inform. Any attribute about to become a key gets surveyed across every row.

Scenario 3 — The checkpoint that would have poisoned another environment

Checkpoint files were keyed by job name alone, because when they were written only one target environment existed. Running the same job against a second environment would have silently resumed from the first one's cursor. The bug wasn't in checkpointing; the bug was in state identity. Any state a migration persists — checkpoints, quarantine lists, skip notes — must carry the environment in its identity, because you will always eventually run the job against a second environment.

Scenario 4 — The credentials that didn't outlive the backfill

Credential lifetime is a migration dependency, not an operator detail. Long scans meet it head-on: first a login token expired mid-scan — retry logic re-resolved the session, the checkpoint held, resume worked. Then, overnight, the login session itself lapsed: every "refresh" returned the same dead token, and the auto-resume loop faithfully burned its retries against it while a human slept. Treat it as a dependency and the options rank themselves: credentials whose lifetime exceeds the run; failing that, re-authenticate immediately before launch; and size any retry loop so attempts × backoff outlives the hours nobody is watching. Write the resume loop anyway — checkpointed idempotent passes make retries free.

Scenario 5 — The proof that depended on the write path

"Run twice, second run writes zero" is only a proof for put_if_absent jobs. Upsert jobs rewrite the same rows every pass — their write counter proves nothing. For upsert paths, make the no-op explicit: compare a deterministic content hash or modified-timestamp before writing, and count skips; failing that, the proof is a stable target count plus spot-checked rows.

One boundary worth stating honestly: counts prove population-level completeness; they do not prove row-level equivalence. Where the transformation was one-to-one, we additionally compared deterministic identifiers and selected content hashes. Where one source row fanned out into multiple target rows, the job declared its expected cardinality explicitly — which is also why naive counter-vs-count reconciliation once flagged a phantom discrepancy of exactly the fanned-out rows. Idempotency needs a proof, not a claim — chosen per write path, before the run.

Scenario 6 — The plan that aged out

A migration plan is an executable assumption about a dataset whose shape keeps changing. Ours recommended bounding the largest table by a time window; by run day the table had grown enough that the window kept nearly all of it, and even our skip-lists grew between rehearsal and cutover. Plans decay; re-measure on run day.

4. Build for the Operator, Not Just the Run

Four investments, each costing under an hour, each repaid within a day:

• A challenges log. One append-only markdown file: every crash, count anomaly, and decision, with its fix and lesson. The difference between a rehearsed prod cutover and a re-discovery tour.
• An interactive runner. Explains each step, asks before executing, tees output to timestamped logs, and states each step's pass condition — so a junior engineer knows what good looks like without paging you.
• A resume mode distinct from run. Our run deliberately restarted counters; when a long job was killed mid-flight, restarting would have discarded the cursor. The mode you need at 2 a.m. should exist before 2 a.m.
• Account guardrails in the wrapper. Assert both AWS profiles resolve to the expected account IDs before any job runs. Four lines that make "migrated into the wrong account" structurally impossible.

Positions Worth Defending

1. A dry-run validates logic, not the write path. Real rows through the real write path, before any bulk run.
2. Samples reassure; surveys inform. Any attribute becoming a key gets a full-column type survey.
3. Idempotency needs a proof, not a claim. Put-if-absent proves itself; upserts need a content hash, a timestamp guard, or a stable target count.
4. Persisted state carries its environment. Checkpoints without scope are future cross-environment corruption.
5. Plans decay; re-measure on run day. Every volume-based number in a migration plan is wrong by cutover — the only question is by how much.

Nothing here is exotic.

That's the point.

The failures that actually happen are mundane. A duplicated attribute. A stale checkpoint. An expired credential. A type that was never what the sample said it was.

The contract is what turns those failures from incidents into log entries.

A note on stakes: at Hoomanely, the data layer we moved carries what our smart bowl and the EverWiz app know about the pets in our users' lives. When a migration is custody of a pet's history, "probably all arrived" is not a standard — provably recoverable and provably complete is.