Designing for Absence: Engineering Accountable Notification Delivery

What six production incidents and one near-miss taught us about proving not only what a notification did, but why it didn't.

A failed notification leaves evidence: an error, a retry, a DLQ entry, a provider response. A notification that never entered the system leaves nothing.

That was the hardest reliability problem we found when rebuilding our notification platform. We eventually stopped treating it as a delivery problem and started treating it as an accountability problem:

Every non-send must be a recorded, queryable decision.

This post covers the problem behind the symptoms, the production scenarios that defined the solution space, and the architecture that answers both.

1. The Problem: Reliability Wasn't the Problem. Accountability Was.

The surface symptom was reliability: our primary flow was delivering to ~21% of its audience while a sibling campaign on identical infrastructure delivered at 82%. But underneath sat something worse — after the fact, "the user received nothing" and "we decided to send nothing" were indistinguishable. In one week, failed deliveries actually outnumbered successful sends, and no system could say which failures were decisions, which were defects, and which were decay.

That indistinguishability is an architectural flaw, not a bug. Fixing bugs removes instances; this required removing the class. A platform that can prove what it sent, when, and why — and equally prove what it deliberately withheld — is the actual product. Delivery rate improves as a consequence.

2. Six Incidents and One Near-Miss

Every requirement came from production — nothing here is speculative hardening:

• The silent zero. A filter edit reduced a campaign's audience to zero; the gap ran 3+ days and was found by a manual metrics pull. "We sent nothing" must be as visible as "we sent."
• The exclusion that looked correct. A reachability segment gated on historical attribute transitions, permanently excluding anyone whose flags never happened to toggle — visually identical to the filter working.
• The bypass nobody checked. "Safety alerts bypass quiet hours" existed as a constant that was defined and never read — leaving a water-empty alert suppressible by a user's night window.
• The triple send. A duplicated broadcast reached users three times in seven minutes — beyond SQS FIFO's five-minute dedup window.
• The 21:00 IST push for a US audience. The engine was fully timezone-parameterised, yet not a single device carried a timezone value.
• The tokens nobody could delete. Devices registered push tokens directly with the vendor; no delete path existed anywhere — the decay behind failures eventually outnumbering sends.
• The near-miss. A retry design that passed review twice and would have silently weakened our delivery guarantee. It gets its own section below.

3. The Design Principles

Every terminal path writes exactly one row. Each notification ends in exactly one append-only trail row — including the non-sends: deduped, suppressed:quiet_hours, suppressed:rate_limit, coalesced, suppressed:unreachable. Zero rows is a vanished notification; two rows corrupt every dashboard rate. The vocabulary is closed, and a closed vocabulary is only closed if a test enumerates it: a parametrised suite walks every terminal path, and the build fails if a new outcome ships without a scenario.

One distinction matters here: a terminal decision is not device delivery. SENT in our trail means the platform made and recorded its decision and handed the message to the provider; provider acceptance and device delivery are tracked as separate observable states fed back by receipts. Conflating the two is how dashboards lie.

UNKNOWN is a state, not a failure. Reachability is three-state — REACHABLE, UNREACHABLE, UNKNOWN — and UNKNOWN, the ordinary state for anyone no receipt has reported on, never gates anything. The dangerous implementation is the innocent-looking one:

python

if not reachable: suppress()   # conflates UNREACHABLE with UNKNOWN

That single conflation is how our vendor's segment silently excluded users forever. Absence of a signal is not a negative signal — a principle that generalises well beyond notifications, to fraud checks, feature gates, and access control alike. Three consecutive undeliverable receipts retire a token and flip one attribute; a single re-registration resets it. Gates arm only on evidence.

Criticality selects a transport, not a flag. A critical send rides the vendor's transactional path — which bypasses send windows by the vendor's own construction — and is never queued, merged, or quieted on our side. There is no branch to forget, because there is no branch.

Every gate declares its failure direction. Undeclared failure directions are decisions made by accident. Ours, chosen deliberately and pinned by tests:

The dedup layers themselves are owned, not delegated: an idempotency key derived from the fact being reported (event_type : subject : actor : addressee, all four load-bearing), claimed by a conditional write with no time window; a content fingerprint for callers without a key; a burst cap for the caller stuck in a loop.

Own the book of record. One canonical IANA timezone per person, with provenance (app outranks geo; upgrades never reverse), resolved in one function: channel override → canonical → tenant default → UTC. And our database is the canonical device book — token as sort key, so re-registration overwrites — while the vendor holds a projection we maintain. Whatever you cannot enumerate, you cannot clean; whatever you cannot clean will eventually outnumber the truth.

4. The Architecture

The boundary came first, and the scope sentence took three revisions to converge: domain services hand the notification service a message and a priority; the service is accountable for every notification reaching a deterministic, auditable terminal outcome — routed appropriately, at a sensible local hour, with provider delivery state observable. It does not decide whether a message is worth sending; that needs domain meaning, and it stays in the domain. Vendor coupling follows the same discipline — registration is data, behaviour is code: a new transport is one adapter file plus a registry entry, and a custom lint fails the build if a vendor's name appears anywhere else.

5. The Design We Almost Shipped

The near-miss deserves prominence because the wrong code looked correct and passed review twice. First instinct: claim the notification, then enqueue, so replays are refused. But claim-before-enqueue silently converts at-least-once into at-most-once — a transient delivery failure is never retried. The correct shape assigns each mechanism one job:

text

claim  = the idempotency boundary   → refuses replays
queue  = the failure-recovery boundary → redrives failures

Three supporting decisions carry their own lessons: message groups are per user and tier (or an urgent alert queues behind ordinary backlog); the visibility timeout is derived from worst-case handler duration, not chosen; and the queue ships dark — its age and DLQ alarms are a precondition for enablement, each deliberately fired once. A retry queue is a brand-new way for messages to stop arriving while every component reports success.

6. Proof, Not Confidence

The integration suite grew by roughly a third through this work — all tests against real services, zero mocks; a test that cannot reach its dependency skips rather than passing against a double. In a codebase whose defining failures were silent, a test that can lie quietly reproduces the original defect in miniature.

The honest limitations: the retry queue stays dark until its alarms have fired; merging is inert until a vendor-side bundle template exists; and the client release moving token registration to our API is what makes the device book true in production. The machinery is ahead of its data — by design, not accident.

7. Positions Worth Defending

1. Design for absence. If the system can explain what it did but not why it did nothing, it isn't fully observable.
2. UNKNOWN is not NO. Missing evidence must never become negative evidence.
3. Don't implement criticality as a flag. If correctness depends on remembering a branch, eventually someone won't.
4. Every gate needs a failure direction. Fail closed when duplicates are dangerous; fail open when silence is dangerous.
5. Own the book of record. If you cannot enumerate it, you cannot clean it; if you cannot clean it, it will eventually become your source of truth.

At Hoomanely, notifications aren't always conveniences. Some are safety signals concerning a living being. That's why our standard isn't simply "did we send it?" — it's "can we prove what happened, including why we didn't send it?" If it matters to a pet, it gets measured. Now, even the silences do.