Is Your Health Check Measuring Health - or Just Activity?

Most monitoring conversations are about thresholds and routing - how many failures before we restart something, how long before we page someone, who gets woken up at 3am. That's real work. But it quietly assumes something worth verifying: that the signal you're measuring actually means what you think it means.

We recently spent a week chasing a fleet of connected devices that were restarting themselves every couple of hours and announcing each restart in Slack. The pattern was oddly precise - three restarts about ten minutes apart, then two hours of silence, then the same again. The devices turned out to be completely healthy. The monitoring wasn't. Two separate things had gone wrong, and neither one produced an error anywhere.

Your liveness signal shouldn't depend on your users showing up

The watchdog asked a reasonable-sounding question every five minutes: has any data arrived in the last thirty minutes? Camera image, thermal reading, proximity event - any one of them would do. If all three were stale, it assumed the pipeline was wedged and restarted the service.

The problem is that all three of those signals only exist when someone actually interacts with the device. No interaction, no capture, no data. Which means at three in the morning, a perfectly healthy device produced exactly the same readings as a dead one - and got restarted for it. Then again ten minutes later, and once more after that, before backing off for two hours and starting over.

Any liveness signal derived from user activity will fail you at precisely the quietest hour, which is also the hour nobody is watching.

Someone on the team had actually anticipated this. Buried in the firmware was a heartbeat - one row written every five minutes whether or not anything was happening, with a comment saying exactly why:

/* the pipeline watchdog uses proximity rows as a liveness signal.

• We therefore persist one row every 5 min regardless of trigger */

That's the right instinct. "Is work happening?" and "is this thing alive?" are two different questions, and only the second one can be trusted when the system is legitimately idle.

Check what else in your system touches your monitoring data

The heartbeat existed. The watchdog never saw a single one - every check came back reporting that no proximity data had ever been recorded.

The reason was a cleanup routine elsewhere in the codebase, running in a loop every two seconds, deleting those rows almost as fast as they were written. Its comment referenced the very same watchdog:

// Proximity rows are written as a liveness heartbeat (every 5 min)

// so the pipeline watchdog can see "the bus is alive."

DELETE FROM offline_queue WHERE event_type='proximity'

AND (status='uploaded' OR created_at < ... OR id IN (...));

Two engineers, both thinking about the watchdog, writing code that cancelled each other out.

The detail that makes this worth writing down is the condition on that delete. Rows were removed once they had been successfully uploaded. So if uploads had been failing, heartbeats would have accumulated and the watchdog would have stayed quiet. Because uploads were succeeding, the evidence was erased within seconds. The system restarted itself precisely because it was working correctly.

The fix was removing one condition and letting the existing age and row-count limits do the bounding they were already there to do:

-- Before
AND (status='uploaded' OR created_at < ...'-3 days' OR id IN (...))

-- After
AND (created_at < ...'-3 days' OR id IN (...))

There was a second failure hiding in the same area. A pipeline change had renamed one of the event types - a deliberate, correct change. But the watchdog still queried the old name, so it reported that subsystem dead for thirty-four days while it worked perfectly the entire time. Monitoring reads your data by name, which quietly turns every event name into an API contract. Nothing fails loudly when you break it.

The fix we almost shipped

The obvious fix, and the first one we built, was to stop sending an alert on restarts. The reasoning held together: the alerts are noisy, they fire on every restart, genuine reboots are rare, so only alert on genuine reboots. It would have passed review. It would have shipped. The Slack noise would have stopped.

It would also have been the worst possible outcome, because that alert was the only visible symptom of the real bug. Silence it and the fleet keeps restarting itself indefinitely - invisibly, and now permanently, because the one thing telling us it was happening had been switched off.

We had even iterated on it, replacing a rough first version with a more elegant one. Polishing the wrong solution while feeling productive about it.

What caught it wasn't a test or a code review. It was a colleague pointing out that the restarts themselves were necessary - the thing to fix was when we decided to restart, not whether we announced it.

The useful takeaway isn't "fix root causes," which everyone already agrees with in the abstract. It's a question worth asking before shipping anything that makes a symptom disappear: if this fix works, what will I no longer be able to see?

It all comes down to whether your instruments are telling you the truth

None of this was hard to fix. A condition removed from a query, a name updated, a restart rule rewritten to distinguish "quiet" from "broken." Nothing here required new infrastructure or a rewrite.

But monitoring occupies a strange position in a system: it's the thing you consult when you want to know whether everything else is working, which means when it fails, it doesn't fail loudly. It just quietly reports that everything is fine, or that everything is broken, with precisely the same confidence either way.