Publishing an operating system to one bench device is a task. Publishing it to a fleet of devices sitting in customers' kitchens — each polling the same update feed, each capable of being permanently broken by a bad write — is a discipline. The scary part isn't the update mechanism; that's a solved problem with two partitions and a fallback. The scary part is the rollout: how you make a change reach exactly the devices you intend, in the order you intend, with proof at every step that nothing else was touched.

Our operating system has rolled across the fleet through more than a dozen releases, and every one left a script behind. Read in sequence, those scripts are the method. This is how a release travels from one bench device to every home — and why no device that wasn't meant to update ever downloaded a single byte.

The Problem: One Feed, Every Device Listening

The update path is deliberately simple: each device runs a timer that polls a feed once an hour and runs whatever active update script it finds. There is one active entry per device class. That simplicity is a feature — one channel, one place to look, no back doors.

It also means every device hears every publish. Push a release intended for two test units and the entire fleet pulls the same script. Devices still on the legacy operating system pull it too. If the script does the wrong thing on the wrong machine, the blast radius is the whole fleet, instantly.

So the real engineering problem isn't "how do we update a device" — it's "how do we publish to everyone and have almost everyone correctly do nothing." Every safeguard below follows from that inversion.

The Approach: Widen the Circle, Guard the Edges

A release moves outward in stages, and the artifact trail shows it plainly. First a single-device canary. Then a stage-1 to a couple of allow-listed test units. Then a stage-2 dry-run on a handful more — described in the script header as the run "that must pass before pushing to user bowls." Only then the fleet.

Each stage is the same script with a wider allow-list, delivered over the same hourly timer the customers' devices use. That last point matters: the test units exercise the identical delivery path, so by the time a release reaches a home, the exact bytes and the exact mechanism have already succeeded end-to-end on hardware.

The Process: Three Guards Before a Byte Is Downloaded

The heart of the method is that every script opens with three checks, and a device that fails any of them exits before it has downloaded anything. The order is deliberate: cheapest and broadest first.

Guard 0 — is this even the right operating system? The feed is shared with devices still running the legacy system, whose updater would happily pull and run this script. It bails instantly on anything that isn't ours:

18	# GUARD 0 — everOS-only (a Debian bowl's updater would also pull this; bail instantly).
19	[ -d /etc/everos ] || { echo "[everos_ota_s1] not everOS — skip"; exit 0; }

Guard 1 — is this device on the list? The device reads its own identity from local config and checks it against the release's allow-list. Anything not named leaves immediately — no download, no disk write, nothing:

22	ID=$(grep -E '^device_id' /data/config.txt 2>/dev/null | cut -d= -f2 | tr -d ' ')
23	echo "$ALLOWED" | tr ' ' '\n' | grep -qx "$ID" || { log "device '$ID' not in stage-1 list — skip"; exit 0; }

(The comment on line 21, which names the specific test units, is omitted.)

Guard 2 — has this already been done? The script is idempotent. A marker file records completion, and a device already running the target version simply drops the marker and leaves. An hourly timer means this script runs again and again; it must be safe to run a hundred times:

25	# GUARD 2 — idempotent.
26	[ -f "$MARK" ] && { log "already applied $TARGET_VER — skip"; exit 0; }
27	[ "$(cat /data/ota/installed_version 2>/dev/null)" = "$TARGET_VER" ] && { touch "$MARK"; log "already on $TARGET_VER — skip"; exit 0; }

Notice that every guard exits with status zero. That's not carelessness — it's the whole point. To the updater, "correctly did nothing" and "succeeded" look the same, so a non-target device marks the version handled and never re-examines it.

Verify Everything, Write Only the Inactive Slot

A device that passes all three guards still hasn't written anything. Next comes the integrity chain, and it fails closed at every link. The downloaded image is checked against a published checksum, then its signature is verified against a public key baked into the device. Any mismatch deletes the download and refuses:

39	GOT=$(sha256sum "$NEW" | cut -d' ' -f1)
40	[ "$GOT" = "$EXP" ] || { log "sha256 mismatch exp=$EXP got=$GOT"; rm -f "$NEW" "$SIG"; exit 1; }
41	openssl dgst -sha256 -verify /etc/everos/img-sign.pub -signature "$SIG" "$NEW" >/dev/null 2>&1 \
42	    || { log "SIGNATURE INVALID — refusing"; rm -f "$NEW" "$SIG"; exit 1; }
43	log "sha256 + signature verified ($GOT)"

Later releases tightened this further for compressed images — verifying the archive's checksum, decompressing, then verifying the decompressed checksum and signature before anything touches storage. The later script's own header states the contract:

9	# Brick-safe: nothing written until BOTH the .gz checksum AND the decompressed sha+signature pass;
10	# writes only the INACTIVE slot; one-shot tryboot auto-reverts if the new slot fails to boot.

Only now does the write happen — and only ever to the partition that is not running. The script derives the inactive slot from the live command line, and if it can't determine one with certainty, it aborts with the active slot untouched:

45	ACT=$(grep -o 'root=[^ ]*' /boot/cmdline.txt | cut -d= -f2)
46	case "$ACT" in
47	  *p2) IN=/dev/mmcblk0p3 ;;
48	  *p3) IN=/dev/mmcblk0p2 ;;
49	  *)   log "cannot determine slot from '$ACT' — abort (active slot untouched)"; exit 1 ;;
50	esac

The reboot itself is a one-shot trial boot into the new slot. If the new system fails to come up healthy, the bootloader firmware falls back to the previous slot on its own — no code of ours needs to run for the device to recover:

63	log "rebooting via tryboot -> v$TARGET_VER on $IN (auto-reverts if boot fails). DO NOT power off."
64	sleep 3
65	reboot "0 tryboot"

Publish, Then Prove the Publish

A rollout can also fail on the publishing side — a typo in a version string, a stale entry left active, a corrupted upload. So publishing isn't a fire-and-forget upload; a tool publishes the script and then round-trips it to prove the fleet will see exactly what was intended.

It queries the feed the way a device would, to confirm the new entry is the one marked active. Then it downloads the published copy back and compares checksums against the local file:

36	  [ "$(sha256sum /tmp/ota_rt.sh | cut -d' ' -f1)" = "$(sha256sum "$F" | cut -d' ' -f1)" ] \
37	    && echo "  SHA MATCH — published byte-identical" || echo "  SHA MISMATCH — do not trust this entry"

The tool's header also records a hard-won operational rule about the updater's own bookkeeping:

8	# One active entry exists per deviceType#subCategory; publishing auto-deactivates
9	# the previous one. Always use a FRESH version string (the updater marks any
10	# exit-0 script as installed and skips that version forever).

That last sentence is the kind of detail that only exists because someone hit it. Because guarded devices exit zero, they record the version as done. Re-publish a corrected script under the same version and the fleet will ignore it forever. Every publish gets a new string.

Every Release, Accounted For

Finally, none of this is trustworthy unless you can say precisely what you shipped. The build manifest keeps a ledger: every released image is built from a tagged commit, and its exact filesystem checksum is recorded alongside what changed.

178	Firmware releases must be built from a tagged commit. Shipped rootfs sha per version:
179	- everos-1.0.5 rootfs sha = `4e3b83ce6aa9a62bf804d42cb08ec064cad9f8e14827be9258d0f7797f097ce8`
180	- everos-1.0.6 rootfs sha = `256b10f6de75594f80961a97de91c3d423f65513c3204a7baccc421b2548c250`
181	- everos-1.0.7 rootfs sha = `73feef63d94168b9b81470f2f3f975e4aaf2696383bfd0e8ded11960f8dc516e`  (= 1.0.6 + Fix-B US WiFi regdomain)

And the rule that makes the ledger mean something:

192	(Tag each release commit `vX.Y.Z`; rebuild from the tag on a clean checkout and confirm the rootfs sha matches above.)

The checksum a device verifies on download is the same checksum recorded here against a git tag. That closes the loop from a specific commit, through a reproducible build, to the exact bytes running in a specific home. You can only safely roll out what you can positively identify.

The Results

The outcome is a fleet that has taken more than a dozen operating-system releases in the field without a single device being lost to an update. Non-target devices never downloaded anything. Target devices never wrote anything they hadn't fully verified, never wrote to the partition they were running from, and always had an automatic way back.

Just as valuable is what the method produced as a side effect: a complete, readable history. Each stage of each release is a script in the repository, and each release is a line in the manifest with its checksum and its reason. Six months from now, anyone can reconstruct exactly what reached which ring of the fleet, in what order, carrying which bytes — from the files alone.

Why It Matters at Hoomanely

Hoomanely is reinventing healthcare for pets — replacing reactive, imprecise care with continuous, clinical-grade monitoring that catches problems early. Our devices form a Physical Intelligence ecosystem: sensors fused at the edge, feeding the Biosense AI Engine that turns raw signals into personalized, preventive insights.

A health device has to keep improving for years, and every improvement has to reach every home. The rollout discipline above is what lets us ship those improvements continuously without ever gambling with a family's device — the same care that goes into a sensor reading goes into how a release travels.

It also embodies how we think about trust. We don't ask customers to believe an update is safe; we build so that a device can prove the bytes it received are the bytes we tagged, and so that a device that shouldn't update simply, quietly, doesn't.

Key Takeaways

• Invert the problem. When one feed reaches every device, the design goal is "publish to all, and make almost everyone correctly do nothing."
• Guard before you download. Three cheap checks — right OS, on the allow-list, not already done — let non-target devices exit before a byte moves.
• Exit zero on purpose. Guarded devices must look like successes, or an hourly timer will retry them forever — which also means every publish needs a fresh version string.
• Verify everything, write only the inactive slot, trial-boot. Checksum, signature, an untouched active partition, and a one-shot boot that reverts by itself.
• Prove the publish and ledger the release. Round-trip every publish for byte-identity, and record every shipped checksum against a git tag so a release is never anonymous.

Author's Note

This rollout method carried everOS, the operating system behind Hoomanely's Everbowl, across the fleet release after release. There is nothing exotic in it — allow-lists, checksums, a trial boot, a ledger — and that is exactly why it works. The scripts left behind read less like tooling and more like a lab notebook: each one a record of a release earning its way, ring by ring, from a single bench device to every kitchen it now runs in.