Our mobile measurement partner reported nearly every install as organic while we were paying for install campaigns on two major networks. The media-source field was never filled once. The cause was not one bug: 6 faults, 3 in app configuration and 3 in our own attribution code, each independently
Three background jobs, none of them serving a single user request, consumed nearly all of the DynamoDB read capacity on our production account. One scheduler table absorbed a GetItem count in the millions over a single week. A second job read a full batch from every index shard on every
Why an operator's already-open tab is the hardest client your platform has — and what it costs to keep it working. An operator opens a console screen, starts a support case, and types a note into a dialog. Two minutes later we deploy. Their next interaction behaves differently
Why our operator console has no database, no domain code, and no knowledge of any module until it asks: micro-frontends built on browser primitives, not a framework. Every admin console eventually drifts, because it is the one place where knowledge about a module lives outside that module. Ours was
How a migration can pass reconciliation at every environment and still ship a broken system — and what we verify instead. A matching count proves the rows arrived. It does not prove the system can find them, identify them, or alert anyone when they stop working. We learned that the hard
A data migration is only done when every row can be proven to have arrived and every failure can be recovered without making things worse. We learned that moving a few hundred DynamoDB tables and millions of items into a fresh AWS account with a redesigned schema: every row transformed,
What six production incidents and one near-miss taught us about proving not only what a notification did, but why it didn't. A failed notification leaves evidence: an error, a retry, a DLQ entry, a provider response. A notification that never entered the system leaves nothing. That was
Every system I have run at scale has taught me the same lesson in a different costume: failures don't hide in the components you own — they hide in the seams between them. This is the story of one such seam. Our notification pipeline had a monitored backend on