The Build That Verifies What It Ships

A firmware build has one job that is easy to state and hard to guarantee: the image that comes out must contain every feature you believe you put in. "The build passed" feels like proof. It isn't. A build system reports on the steps it ran — each recipe compiled, each package installed — not on whether the result is the product you intended to ship.

everOS closes that gap with a capability most embedded builds lack: the image verifies its own contents before it is allowed to exist. A final stage inspects the finished filesystem against a written contract of what must be present, and a build that cannot honour the contract fails instead of producing an artifact. This post is about how that contract is built, why it inspects the artifact rather than the intentions, and how the same principle carries through to the running device.

The Concept: Verify the Artifact, Not the Steps

Every stage of an image build can succeed while the outcome is wrong. A recipe can compile cleanly and never be pulled into the image. A dependency can resolve to a subpackage the image excludes. A build flag can quietly switch a feature off inside a binary that still links and installs. None of these produce an error, because at every step the tool did exactly what it was asked.

The contract inverts the question. Instead of asking did every step succeed, it asks does the finished filesystem contain each thing the product requires — and it asks this at the one moment the answer is knowable: after the rootfs is fully assembled and before it is packed into an image.

That moment is a standard hook in the build system, and the contract is a shell function attached to it:

69	everos_assert_contract() {
70	    R="${IMAGE_ROOTFS}"
71	    fail=0
72	    _need() {   # _need <path> <why>
73	        if [ ! -s "$R$1" ]; then bbwarn "CONTRACT: missing $1 — $2"; fail=1; fi
74	    }

Two details carry a lot of weight. The check is -s, not -e: a file that exists but is empty counts as missing, because an empty unit file or a zero-byte binary is a feature that will fail at first use. And every entry takes a <why> — the contract documents its own reasoning, so the list is readable by anyone, not just the person who wrote it.

Why It Matters: A Fleet OS Has No Second Chance

On a developer's laptop, a missing feature is discovered in minutes. On a device in a customer's home, it is discovered when a pet parent tries to use it — and for some features, not even then. A commit step for updates, a provisioning flow, a detector inside a vision pipeline: these fail quietly, and the device looks perfectly healthy while doing so.

The class of features that can vanish without an error is broad, and the contract's own header lists the kinds it exists to catch:

Each of these builds green. The contract's premise, written directly above the function, is that this class needs a structural answer rather than more vigilance:

66	# Each was found by accident, weeks late. A build that cannot produce the feature
67	# must fail here rather than ship silently. Add a line whenever a feature is the
68	# kind that can vanish without an error.

That last sentence is the operating rule. The contract is not a fixed list; it grows with the product, one line per feature whose absence would be silent.

How It Works: A Declarative Shipping Contract

The body of the contract reads like a manifest of what the product is. Each line names a critical file and the feature it embodies:

75	    _need /usr/bin/cm4_ml_pipeline            "ML/bowl-detection binary"
76	    _need /usr/lib/everbowl-ble/everbowl_gatt_ble.py "BLE onboarding GATT server (EVOS-042)"
77	    _need /usr/lib/systemd/system/everbowl-ble.service "BLE onboarding unit (EVOS-042)"
78	    _need /usr/bin/nmcli                      "nmcli shim the BLE provisioning flow calls"
79	    _need /usr/bin/ota-confirm.sh             "A/B OTA commit step"
80	    _need /usr/bin/everbowl-firmware-updater  "the feed-OTA channel"
...
83	    _need /usr/bin/everos-boot-marker.sh      "boot_count / last_boot_reason writer"

(Two lines for a separate subsystem are omitted here.)

Read top to bottom, this is the product's promise to itself: the vision pipeline, the phone onboarding flow with the unit that starts it and the shim it calls, both halves of the update channel, and the boot-attribution writer. If any of them is absent from the finished filesystem, the image is not everOS.

Three refinements lift it above a simple file-exists loop. First, layout variance is handled explicitly — a tool may live in one of two directories depending on how the filesystem is merged, and the contract accepts either:

88	    # resize2fs lives in /sbin or /usr/sbin depending on usrmerge.
89	    if [ ! -s "$R/sbin/resize2fs" ] && [ ! -s "$R/usr/sbin/resize2fs" ]; then
90	        bbwarn "CONTRACT: missing resize2fs — A/B slots will keep the image filesystem size (A4)"; fail=1
91	    fi

Second, some features are not files at all. A detector compiled into a vision binary leaves no separate artifact to check — the binary exists whether or not the detector is inside it. So the contract inspects the binary's contents for the detector's own symbol:

92	    # ArUco is a compiled-in code path, not a file: check the binary carries it.
93	    if ! grep -qa 'aruco_v2' "$R/usr/bin/cm4_ml_pipeline" 2>/dev/null; then
94	        bbwarn "CONTRACT: cm4_ml_pipeline has no ArUco detector — bowl-order ships dead (EVOS-034)"; fail=1
95	    fi

Third, the outcome is binary and final. Every failed check emits a warning naming the feature and the reason, and then the build stops. There is no artifact to accidentally publish:

96	    if [ "$fail" != "0" ]; then
97	        bbfatal "everOS rootfs contract check FAILED — see the CONTRACT warnings above. The image would ship with a feature silently missing."
98	    fi
99	    bbnote "everOS rootfs contract check passed"
100	}
101	ROOTFS_POSTPROCESS_COMMAND += "everos_assert_contract;"

The last line is what makes it structural rather than optional: it is appended to the rootfs post-process, so it runs on every image build, for every board, with no way to forget it.

Applications: The Same Principle, Running on the Device

A build that proves what it ships has a natural counterpart: a device that proves what happened to it. everOS carries the same instinct into the field with a small script that runs once per boot and gives every device two facts the fleet dashboard had never been able to fill — how many times it has booted, and why the last boot happened.

The counter is designed to survive the things that normally reset it. It lives on the persistent data partition rather than in a rootfs slot, so an A/B update leaves it intact, and it is written crash-safely:

21	# ── boot_count: monotonic, persistent across updates (lives on /data, not in
22	#    the rootfs slot, so an A/B update does not reset it).
23	COUNT=$(cat "$OTA/boot_count" 2>/dev/null)
24	case "$COUNT" in
25	    ''|*[!0-9]*) COUNT=0 ;;          # absent, corrupt, or non-numeric -> restart at 0
26	esac
27	COUNT=$((COUNT + 1))
28	# Temp + rename: a power cut during this write must not leave a truncated file
29	# that the next boot then reads as 0 and silently restarts the count.
30	printf '%s\n' "$COUNT" > "$OTA/boot_count.new" 2>/dev/null &&
31	    mv "$OTA/boot_count.new" "$OTA/boot_count" 2>/dev/null

The attribution is where the design judgement shows. Each boot is explained in order of certainty: a marker left by the update system when it committed or reverted a slot, then a kernel-panic record that survived in persistent store, then a watchdog stamp — but only if it falls within ten minutes of this boot. And when none of those apply, the script makes a deliberate choice:

37	# Anything else we genuinely cannot distinguish from a power cut, so we say
38	# "unknown" rather than guessing "power_loss" -- a wrong attribution here would
39	# send someone to a customer's house for a mains problem that was ours.
40	REASON=unknown

A field that says unknown is honest. A field that guesses is a diagnostic that can dispatch a technician to the wrong problem. The same value system that refuses to ship an image it cannot verify refuses to report a cause it cannot prove.

Each marker is also consumed exactly once, so an update-triggered reboot is attributed to the update, and the boot after it is not:

42	if [ -f "$OTA/next_boot_reason" ]; then
43	    # Written by the previous boot's ota-confirm; consume it so it is reported
44	    # exactly once and the boot after this one is not also blamed on the update.
45	    R=$(cat "$OTA/next_boot_reason" 2>/dev/null)
46	    [ -n "$R" ] && REASON="$R"
47	    rm -f "$OTA/next_boot_reason" 2>/dev/null
48	elif [ -s /sys/fs/pstore/dmesg-ramoops-0 ] 2>/dev/null; then
49	    # A kernel panic record survived the reboot. Strongest possible signal.
50	    REASON=kernel_panic

Why It Matters at Hoomanely

Hoomanely is reinventing healthcare for pets — replacing reactive, imprecise care with continuous, clinical-grade monitoring that catches problems early. Our devices form a Physical Intelligence ecosystem: sensors fused at the edge, feeding the Biosense AI Engine that turns raw signals into personalized, preventive insights.

That promise rests on devices in homes doing exactly what we believe they do. A build that proves its own contents means every image that reaches a family carries the full product — the onboarding flow, the vision pipeline, the update path — not a plausible-looking subset. And a device that can account for every boot means our fleet view reflects reality rather than absence of data.

Both are the same engineering value applied at two points in the lifecycle: never report success you cannot verify. It is a quiet discipline, and it is what allows a small team to ship an operating system to homes with confidence.

Key Takeaways

• Verify the artifact, not the steps. Green build stages prove the tools ran; only inspecting the finished filesystem proves the product is complete.
• Make the contract declarative and self-documenting. One line per critical feature, each with its reason, readable by anyone on the team.
• Go beyond "file exists." Check non-empty, accept known layout variants, and grep binaries for compiled-in features that leave no separate file.
• Fail hard, at build time. A missing feature should stop the build, so an incomplete image can never become a downloadable artifact.
• Carry the principle to the device. A crash-safe, update-surviving boot counter and certainty-ordered attribution — with an honest unknown — give the fleet facts instead of guesses.

Author's Note

These two small pieces of everOS — a contract the build enforces on itself, and a script that lets every device explain its own reboots — do almost nothing on a good day. The build passes, the counter increments, the reason says ota_commit. Their value is entirely in the days that are not good, when they turn a silent gap into a hard stop or an honest answer. Building for those days is most of what shipping to real homes turns out to be.