Rebooting a device sounds harmless until you remember what it's for. When a pet-monitoring station restarts, it isn't watching anything — no weight readings, no captures, no heartbeat. A safe operating-system update costs a reboot, and a reboot costs a small hole in a pet's record.

So a faster path is tempting: instead of replacing the whole operating system, just swap the handful of application binaries that actually changed, in place, with no restart at all. We built exactly that. It downloads, verifies, installs, and reports success — and today it is switched off on purpose, with the reason written into the file that would otherwise run it. This is the story of why the shortcut was the wrong tool, and why we left the evidence behind instead of quietly deleting it.

Context: Two Ways to Change a Device

The sanctioned path is a cold update. The device downloads a complete new root filesystem, writes it to the inactive one of two slots, reboots into it, and keeps it only if a health check passes — otherwise a watchdog rolls back to the slot that was working. It's safe precisely because the running system is never modified.

It's also heavy. A full image, a reboot, and a health-gate cycle is a lot of ceremony to fix a typo in one program.

The hot update was the answer to that. Its own header states the ambition plainly:

2	# ota-hot-update.sh — in-place binary update, no reboot
3	# Called by cronie every 6 hours.

No reboot, and a routine six-hour poll rather than a deliberate event. On paper it's the scalpel to the cold update's sledgehammer.

The Challenge: It Looked Completely Reasonable

The script isn't sloppy. It takes a single-instance lock, asks the backend whether an update exists for this specific device and version, and treats a failed check as a non-event rather than an error. Then it downloads the package and refuses to proceed unless the bytes are exactly what the server promised:

49	ACTUAL_SHA=$(sha256sum "$TARBALL" | cut -d' ' -f1)
50	[ "$ACTUAL_SHA" = "$EXPECTED_SHA" ] || { rm -f "$TARBALL"; die "SHA256 mismatch"; }

That's a real integrity check, and it fails closed — a mismatched download is deleted, not installed. Everything up to this point is the behaviour you'd want.

Then it stops the services that own the binaries, installs the new ones, restarts, and records the version. Mechanically, it reads like a correct update. The problems are not in what it does — they're in what doing it means.

What We Found: Two Problems, One Fatal

It broke the A/B guarantee

The install step writes the new binaries into the running system:

66	find "$EXTRACT_DIR" -type f -name "cm4_*" | while read f; do
67	    name=$(basename "$f")
68	    dest="/usr/bin/$name"
69	    install -m 0755 "$f" "$dest"
70	    log "Installed $dest"
71	done

That single line — writing into the active root filesystem — quietly dissolves the property the whole update system is built on. The A/B model works because each slot is an immutable, known artifact: it was produced by a reproducible build, it has a version, and if the other slot misbehaves you can fall back to it with confidence about exactly what you're falling back to.

Hot-patch the active slot and that stops being true. The running device now matches no image that was ever built, and no image you could rebuild. Rollback becomes a guess rather than a guarantee: revert to the other slot and the hot fixes silently vanish; stay where you are and you're running an artifact that exists nowhere but on that one device.

Multiply that across a fleet and the devices quietly drift apart — which is the exact failure mode a purpose-built operating system was adopted to eliminate.

It failed silently

The second problem is subtler and, in some ways, worse. Before replacing a binary the script stops the service that's running it:

54	# Stop services that own the binaries we are replacing
55	for svc in cm4-heartbeat weight-monitoring cm4-upload-monitor cm4-trigger-poller; do
56	    /etc/init.d/$svc stop 2>/dev/null || true
57	done
58	sleep 2

Those are legacy init-script paths. On the current operating system the services are managed by a different init system entirely, so those files simply don't exist — and look at how the failure is handled. The error output is discarded by 2>/dev/null, and then || true converts the failure into a success.

The result is a script that cannot stop the services, does not notice, and reports that it did. It then overwrites binaries underneath processes that are still running, waits two seconds, "restarts" services it never stopped, writes a new version number, and logs Hot update complete.

That is the most dangerous shape a bug can take: not a crash, but a confident, well-logged lie. Every signal a human would check says the update worked.

The Approach: Turn It Off, and Say Why

The decision was to disable the hot path rather than ship it. What makes it a good decision is where the disabling lives — in the scheduling file itself, with the reasoning and the conditions for bringing it back:

1	# Everbowl OTA hot update — DISABLED.
2	# ota-hot-update.sh uses SysV /etc/init.d (no-op on systemd) AND modifies the
3	# active rootfs in-place (breaks the A/B model). Re-enable only after it is
4	# rewritten to use systemctl + a non-rootfs-mutating strategy.
5	# 0 */6 * * * root /usr/bin/ota-hot-update.sh

Five lines carrying both faults, the verdict, and the re-entry criteria. The schedule that would have run it every six hours is right there, commented out and intact.

This matters more than it looks. A feature disabled without explanation is a trap for the next engineer, who finds a commented-out line, assumes it was tidiness or a stale experiment, and switches it back on. Writing down why converts a mystery into a decision — and names precisely what would have to change for the shortcut to become safe.

Results: The Slow Path, Deliberately Triggered

What ships is the cold path alone, and it isn't a background poll — it's something a person or the cloud asks for on purpose. Its header lists the ways: a cloud push, an app-initiated request over the local wireless link, or a physical gesture on the device itself.

That last one is my favourite piece of field engineering. A small service watches a pin and counts presses:

9	CLICK_WINDOW=60
10	CLICK_THRESHOLD=5
11	COLD_UPDATE=/usr/bin/ota-cold-update.sh

Five presses inside sixty seconds — deliberate enough that it can't happen by accident, simple enough that anyone can do it standing in front of the device with no tools and no network.

The outcome is a fleet where every device is still exactly an image that was built, versioned, and can be rebuilt — and where updating is an event with a health gate and a way back, not a silent nightly mutation.

Why It Matters at Hoomanely

Hoomanely is reinventing healthcare for pets — replacing reactive, imprecise care with continuous, clinical-grade monitoring that catches problems early. Our devices form a Physical Intelligence ecosystem: sensors fused at the edge, feeding the Biosense AI Engine that turns raw signals into personalized, preventive insights.

A device that lives in someone's home for years must be able to change safely, and "safely" has a specific meaning: we can always say exactly what software a given device is running, and we can always get it back to something that worked. A shortcut that trades that guarantee for a few saved seconds isn't a shortcut — it's borrowing against every future update.

Choosing the slower path, and documenting the refusal, is the same instinct that runs through the rest of our stack: prefer the honest mechanism to the fast one, and never let a system report success it hasn't earned.

Key Takeaways

• Speed that breaks an invariant isn't speed. In-place binary patching saves a reboot but destroys the property that makes A/B rollback trustworthy — that each slot is exactly a build you can reproduce.
• A slot is only a fallback while it's unmodified. Once the running system matches no artifact you built, "roll back" stops being a guarantee and becomes a guess.
• Beware double-silencing. Discarding stderr and forcing success turns a failed command into an invisible one — here it meant overwriting binaries under live processes while logging completion.
• The worst bug is a confident lie. A crash gets investigated; a script that reports success it never achieved passes every check a human would run.
• Disable loudly. Leave the code, comment out the trigger, and write down both the faults and the conditions for re-enabling — otherwise the next engineer will simply switch it back on.

Author's Note

This lives in the update machinery of everOS, the operating system behind Hoomanely's Everbowl. It's a small story about a script that never runs, and it's one of my favourites — because the engineering worth showing isn't only what we shipped, it's what we built, understood properly, and then chose not to turn on. The comment explaining that choice will outlast anyone's memory of making it.