Reconciled, Not Verified: The Defects That Survive a Clean Count
How a migration can pass reconciliation at every environment and still ship a broken system — and what we verify instead.
A matching count proves the rows arrived. It does not prove the system can find them, identify them, or alert anyone when they stop working.
We learned that the hard way, moving a pet-care platform's entire data estate through three environments in one continuous push — dev, then staging, then production. Every hop reconciled cleanly on the first pass: table counts matched their source snapshots, a dry-run file sync reported nothing missing, identity counts squared. Three times the migration looked finished.
Three times something was broken that reconciliation had no opinion about: a search index quietly rejecting every write, users signing in to accounts that appeared to have no history, and a production service crash-looping with no alarm wired to say so. None surfaced as an error. Every row was exactly where it belonged in all three cases.
The Short Version
Reconciliation tells you whether the data arrived. Verification tells you whether the system can still use it.
A migration can have matching row counts while search returns nothing, identity resolution points at the wrong record set, or production has no alert capable of detecting an outage. Reconciliation answers "did it arrive?" — a data-plane question. Verification answers "does it still work?" — an access-plane question, and the one users actually experience.
We now treat migration completion as four independent proofs: data, index, identity, and operations.
Show ImageFigure 1: Reconciliation proves data-plane properties. Every defect we shipped lived in the access plane, which counts cannot express.
1. Data Migration Reconciliation Is Not Migration Verification
Post-migration validation is almost always designed around the data plane: did every row make the trip? It's the natural question because it's the measurable one — counts are cheap, comparable, and satisfying.
But a migration isn't complete when the data lands. It's complete when the system that reads that data still works: the index that makes it findable, the identity resolution that makes it theirs, the operational signals that tell you when either stops being true. A row count is structurally incapable of expressing those properties. Worse, a clean reconciliation manufactures confidence — it produces a green result that is technically accurate and practically misleading.
So the engineering problem isn't "did the rows arrive." It is whose question does each verification actually answer, and which questions is nobody asking?
2. Three Ways "Done" Lied
Defect 1 — The index that rejected its own writes
Community search returned nothing, no matter how many times the backfill reported completion. The destination was a vector-search collection whose type does not accept a client-supplied document id — every write carrying one was refused. The backfill wasn't failing loudly; it was failing per document, one quiet rejection at a time, and summing successes anyway.
What exposed it wasn't a counter comparison — it was a behavioral check: take the set of identifiers the application would use to retrieve these records, and ask the destination to return them. Almost nothing came back, while the job's own log claimed a complete run.
The platform wasn't misbehaving. It was correctly refusing an operation its collection type has never supported. The defect was in our wrapper's assumption that it would. A platform's constraints are documentation; code that assumes them away is where the bug actually lives.
Defect 2 — The login that resolved to the wrong account
Every migrated user carries a stable legacy identifier, stitched into their new sign-in token alongside a freshly issued session id. One line of code read the wrong one.
The causal chain matters, so here it is explicitly: the user's migrated records existed under legacy_id. Authentication succeeded normally. But the application then looked up that user's data using session_id — an identifier nothing had ever been filed under. Finding nothing, it rendered the account as though it had no history. A years-old account presented itself, convincingly, as brand new.
Show ImageFigure 2: The token carried both identifiers. Reading the session id instead of the legacy id produced a valid login into an empty-looking account — indistinguishable from success at every layer above it.
This is the defect class that concerns me most, because authentication succeeding and identity resolving to the right record set are different claims, and only one of them is visible in a status code. The verification that caught it was mundane: sign in as a migrated user, then pull that user's records from the table their session claims to own, and confirm the returned records are the ones that user should have.
Defect 3 — The outage with no alarm attached
A production service pegged its CPU under real load, failed its own health check, and was killed by the system meant to keep it alive — in a loop, serving errors throughout. Every fact needed to diagnose it was already in our metrics platform before anyone looked. Not one alarm in the account was wired to page a human.
The service was one capacity tier from breathing room; that was the cheap part. The expensive lesson was structural: full instrumentation with no alerting is operationally identical to no instrumentation. The signals existed. Nothing converted a signal into someone's attention.
3. What We Verify Instead
Verify behavior, not counters
A migration job's own success counter is not independent evidence. It proves only what the job believes it successfully did. Every check that found a real defect instead asked the destination a question the job couldn't answer for itself.
The principle worth carrying: the destination must verify the destination. Don't ask the migration job whether the migration worked — ask the system that consumes the migrated state.
Define a completion proof for every layer
Each layer fails differently, so each gets an explicit proof, written before the run:
| Layer | Weak completion signal | Stronger completion proof |
|---|---|---|
| Data | Row count matches | Source snapshot reconciles with destination, per table |
| Index | Writes submitted successfully | Expected identifiers are queryable from the destination |
| Identity | Accounts created | A migrated login returns that user's expected records |
| Operations | Metrics exist | An induced failure produces the expected alert |
session id instead of the legacy id produced a valid login into an empty-looking account — indistinguishable from success at every layer above it.
This is the defect class that concerns me most, because authentication succeeding and identity resolving to the right record set are different claims, and only one of them is visible in a status code. The verification that caught it was mundane: sign in as a migrated user, then pull that user's records from the table their session claims to own, and confirm the returned records are the ones that user should have.
Defect 3 — The outage with no alarm attached
A production service pegged its CPU under real load, failed its own health check, and was killed by the system meant to keep it alive — in a loop, serving errors throughout. Every fact needed to diagnose it was already in our metrics platform before anyone looked. Not one alarm in the account was wired to page a human.
The service was one capacity tier from breathing room; that was the cheap part. The expensive lesson was structural: full instrumentation with no alerting is operationally identical to no instrumentation. The signals existed. Nothing converted a signal into someone's attention.
3. What We Verify Instead
Verify behavior, not counters
A migration job's own success counter is not independent evidence. It proves only what the job believes it successfully did. Every check that found a real defect instead asked the destination a question the job couldn't answer for itself.
The principle worth carrying: the destination must verify the destination. Don't ask the migration job whether the migration worked — ask the system that consumes the migrated state.
Define a completion proof for every layer
Each layer fails differently, so each gets an explicit proof, written before the run:
| Layer | Weak completion signal | Stronger completion proof |
|---|---|---|
| Data | Row count matches | Source snapshot reconciles with destination, per table |
| Index | Writes submitted successfully | Expected identifiers are queryable from the destination |
| Identity | Accounts created | A migrated login returns that user's expected records |
| Operations | Metrics exist | An induced failure produces the expected alert |
Show ImageFigure 3: A migration is done when every layer passes its own proof — not when the slowest counter reaches 100%.
Test the access path the application actually uses
The strongest single test is an end-to-end user journey that crosses all four layers: sign in, search, open a record, view its history. If that passes for a migrated user, most of the defects above are structurally impossible. Synthetic checks against each layer tell you where something broke; the journey tells you whether anything did.
Treat alerting as migration scope
The alarms that were missing during the incident — unhealthy targets, zero healthy targets, elevated error rate — now exist. Adding them cost less than any of the three defects above, and closes the failure mode most likely to recur unnoticed. A migration that moves data into an environment nobody is watching has moved the data into the dark.
An implementation note: prefer the platform's native bulk path
Where the platform provided a native export/import mechanism, we preferred it over a hand-rolled scan-and-rewrite loop. It reduced application-side work and removed an entire class of throttling and retry behavior from our migration code. Custom code earns its place only where the platform genuinely can't help — for us, migrating identities silently, without triggering mail to every inbox — and there, parallelizing across workers turned a long serial job into a short one.
Migration Verification Checklist
Before declaring a migration complete:
- Source and destination snapshots reconcile, per table
- Expected identifiers exist in the destination index
- Representative records are retrievable through real application queries
- Migrated users authenticate successfully
- Authenticated users retrieve their own expected records
- A critical end-to-end workflow passes for a migrated user
- Health checks report healthy targets under real load
- Error-rate and availability alerts are configured
- An induced failure reaches the intended human
- Rollback and recovery behaviour has been exercised, not assumed
Positions Worth Defending
- A count matching is necessary, not sufficient. "Rows converted" and "rows correct and reachable" are different claims; every defect above lived in the gap.
- Authentication is not identity resolution. A success response against the wrong identifier is indistinguishable from correctness until you check whose data comes back.
- Destination-side evidence beats job-side counters. The system that consumes the data is the only honest witness to whether it arrived usefully.
- Instrumentation without alerting is not observability. A signal nobody is paged for is a signal that exists only after the fact.
- Every verification answers someone's question. Write down the question and who needs the answer — the questions nobody is asking become visible before the cutover, not after.
None of these defects needed a rewrite. Each was a one-line or one-config fix. All three needed someone to stop reading the summary and go look at the row.
At Hoomanely, this wasn't abstract data. It was pet weights, birthdays, and years of history — the information a pet parent expects to still be there after a migration. A system that reconciles perfectly while making that history unreachable hasn't succeeded.
The question worth asking at every cutover isn't whether the counters agree. It's whether the person on the other side can tell that the migration worked.
