How a migration can pass reconciliation at every environment and still ship a broken system — and what we verify instead. A matching count proves the rows arrived. It does not prove the system can find them, identify them, or alert anyone when they stop working. We learned that the hard
What a cross-account DynamoDB migration taught us about the gap between "validated" and "true." A dry-run validates your read path. Only the cutover validates your write path. Everything expensive lives in the gap between the two. We learned this while moving a few hundred DynamoDB tables
What six production incidents and one near-miss taught us about proving not only what a notification did, but why it didn't. A failed notification leaves evidence: an error, a retry, a DLQ entry, a provider response. A notification that never entered the system leaves nothing. That was the
Every system I have run at scale has taught me the same lesson in a different costume: failures don't hide in the components you own — they hide in the seams between them. This is the story of one such seam. Our notification pipeline had a monitored backend on