Closing the Observability Gap Between "Sent" and "Delivered"

Every system I have run at scale has taught me the same lesson in a different costume: failures don't hide in the components you own — they hide in the seams between them. This is the story of one such seam. Our notification pipeline had a monitored backend on one side and a reputable delivery vendor on the other, and between them, a blind spot large enough to swallow months of silent failure: the majority of our push notifications dying on stale device tokens, and a critical tracking signal switched off for days without a single alert.

The remediation was not a rewrite. It was two hours of deliberate configuration — wiring Customer.io's reporting events and metadata into PostHog — guided by one architectural decision about where measurement should live. The specifics are ours; the pattern applies to any team whose only view of a delivery vendor is the vendor's own dashboard.

The Problem: A System Boundary Nobody Owned

Our backend emits events to Customer.io's Track API; Customer.io runs the campaigns and delivers through FCM and APNs. The division of labor was sound. What we had failed to assign was ownership of the question: what happened after we handed the message over?

When we audited the live campaign metrics through Customer.io's API — prompted by suspicion, not by any alert, which is itself the indictment — three findings surfaced:

1. Delivery had quietly collapsed. Our primary notification flow was reaching ~21% of its audience, consistently. In one week, undeliverable events (6,733) exceeded sends (6,153) — we were burning multiple dead device tokens per person. A sibling campaign on identical infrastructure delivered at 82%, which localized the fault: the platform was healthy, our audience data was not. The root cause was a lifecycle omission — device tokens were registered but never retired. Entropy did the rest.

2. A signal had gone dark without a sound. Our uninstall detection relies on a silent push to every user; the undeliverable responses are the signal. A routine filter edit zeroed that campaign's audience, and uninstall tracking went dark for three-plus days. No alarm fired because no alarm could fire — the data never left the vendor.

3. Engagement had no denominator. Mainline flows sat near 1% open-of-delivered, while one notification — our streak reminder — was achieving 12.5% opens and 15% conversions. A meaningful content insight, invisible for want of a funnel.

Three distinct symptoms, one structural cause: the feedback path from vendor to our systems existed on paper — PostHog had a registered webhook endpoint covering 54 Customer.io event types — but sat disabled, awaiting a signing secret. The capability had been provisioned; the loop had never been closed.

The Decision: Measurement Is a Plane, Not a Feature

The tempting response to an incident like this is to build something — a reconciliation service, a delivery tracker. We declined. The principle we applied instead: each concern gets exactly one home. Detection logic lives in our domain modules. Delivery lives in Customer.io. Measurement lives in PostHog — because it already held our product analytics, and because it offers the three capabilities a measurement plane needs in one place: a warehouse that syncs vendor objects as queryable tables, webhook-fed event tables for real-time telemetry, and SQL, dashboards, and alerting across both.

New services are a liability you staff forever. Configuration that closes an existing loop is leverage.

The Execution: Three Moves

Move 1 — Vendor metadata into the warehouse

Customer.io connected as a PostHog data warehouse source; six metadata tables (campaigns, broadcasts, newsletters, segments, sender_identities, transactional) now sync every six hours. This is what allows an alert to say which campaign flatlined by name — and what makes a watchdog on campaign edits possible at all.

Move 2 — Enable the reporting webhook

The unglamorous heart of the fix: paste the signing secret, enable the webhook. All 54 reporting event types now stream into per-channel tables — push_events, email_events — each row carrying delivery ID, campaign ID, customer identifiers, and timestamp. The full journey of any single notification is reconstructable:

sql

SELECT metric AS event_type, count() AS events
FROM `customerio.customerio.push_events`
WHERE timestamp >= now() - INTERVAL 7 DAY
GROUP BY metric
ORDER BY events DESC

Move 3 — Convert visibility into invariants

Data alone changes nothing; the value came from encoding what we learned as enforced invariants:

• The uninstall pulse got a heartbeat. Audience restored, plus a standing PostHog alert on entries = 0 for any running campaign whose silence is meaningful.
• Audiences were gated to reachability. Campaigns now target push-reachable users only; delivery moved from ~20% of sends to ~100% of eligible recipients, because dead tokens stopped counting as audience.
• Token lifecycle was closed. push_undeliverable events feed a weekly janitor that retires dead tokens via the Track API and flags users has_valid_token = false, which segments then exclude. The root cause — registration without retirement — no longer exists as a class.

The Results: What a Week of Truth Looks Like

(*Conversions are vendor-attributed from any interaction, not only tracked opens — the kind of nuance you only learn once you own the raw events.)

The counts matter less than what they made possible:

• Silent failure is now a contradiction in terms. The class of incident that hid our uninstall tracking for three days now pages us within one. Delivery drops, flatlines, and edit-induced audience wipes are HogQL alerts on real-time tables.
• Delivery rate regained meaning. Against an audience full of dead tokens, delivery rate is noise; against a reachability-gated audience, a drop is a genuine signal — and therefore worth alarming on.
• Content decisions acquired a denominator. The streak reminder's 12.5% open / 15% convert against ~1% for broadcast-style pushes is now a standing per-campaign dashboard. Our operating rule: any recurring notification that cannot clear 2% opens after four weeks is rewritten or retired. Judgment, informed by data — not vibes.

Principles Worth Keeping

1. A vendor's dashboard is a courtesy, not observability. It aggregates what the vendor chooses, on the vendor's timescale, with no alerting on your invariants. Raw events belong in a system you control.
2. Close the loop before you optimize anything inside it. Two hours of configuration surfaced every problem in this post and made each fix verifiable. Everything else was follow-through.
3. Lifecycles fail at retirement, not registration. Anything your system creates — tokens, sessions, subscriptions — decays into noise unless something is accountable for removing it.
4. If silence is a signal, instrument the silence. Any pipeline whose absence of output carries meaning needs an explicit zero-output alarm, or it will fail during routine maintenance — quietly, and at the worst possible time.
5. Own the seams. Our backend was monitored and our vendor was reputable; the hand-off between them belonged to no one. Assigning ownership of that boundary was the real fix — the webhook was merely its implementation.

At Hoomanely, we build connected products — the smart bowl, the EverWiz companion app — that help pet parents understand and care for their pets. A notification is frequently the moment our work reaches a human: a feeding insight, a health nudge, a streak kept alive. A push that dies silently on a dead token is a moment of care that never happens. Extending end-to-end observability to the notification pipeline holds it to the same standard we apply to our device telemetry: if it matters to a pet, it gets measured.