Every over-the-air update hides a bootstrapping problem: the software that applies updates is itself software, and can be broken by an update. When a bad image lands on a sensor module inside a pet feeding station — a device with no screen, no buttons, no port anyone will plug into — nobody is there to hold down a recovery combination. The board is scrap, or a technician visits a kitchen.

The way out is to carve off a small piece of flash, write-protect it, and make it the one thing an update can never touch. On our camera-and-thermal sensor module that piece is a recovery stub: it runs first on every reset, decides whether the application is trustworthy, and if it isn't, sits and waits to be reflashed. Here's how it works — including how we placed its state inside flash that had already been physically damaged.

The Problem: Trusting the Thing That Applies Updates

A microcontroller doesn't have the luxury of the two-slot, health-gated update machinery a full Linux system enjoys. It has one flash array, a vector table at a fixed address, and whatever discipline you impose on the layout yourself.

The naive arrangement puts the update logic inside the application. It works right up until the moment it doesn't: ship one image that crashes before it reaches the update code, and the device can never be reached again. The updater has to live somewhere an update cannot reach.

That gives the stub a hard design constraint. It must be small enough to fit in a write-protected region, simple enough that it is essentially never revised, and fast enough that it costs nothing on the millions of boots where nothing is wrong.

The Approach: A Stub That Never Changes

The stub's own documentation states the budget it holds itself to, and the only three circumstances under which it refuses to launch the application:

5	 * This stub runs at 0x08000000 on every reset. Its normal path is:
6	 *   1. Read boot status flag           (~1µs, memory-mapped flash read)
7	 *   2. Validate active partition        (~10µs, check stack ptr + reset vector)
8	 *   3. Jump to application             (immediate)
9	 *
10	 * Total overhead on normal boot: < 1ms (no PLL, no peripherals, runs on HSI).
11	 *
12	 * Recovery mode is entered ONLY when:
13	 *   - boot_state == FORCE_RECOVERY  (set by CAN/UART safe mode command)
14	 *   - boot_state == BOOT_FAILED     (boot attempt counter exceeded)
15	 *   - Active partition validation fails AND standby also fails

Note what it deliberately does not do on a healthy boot: no clock tree to configure, no peripherals to initialise, no drivers. It runs on the internal oscillator and gets out of the way in under a millisecond. Every line of setup it skips is a line that cannot fail.

Around it sits a deliberate flash map: the write-protected stub at the base, a small boot-status sector holding a checksummed record of what should happen next, and then two application partitions — an active one and a standby one — so a failed update has somewhere to fall back to.

The Process: Four Decisions That Make It Safe

Validate before you jump

Before handing control to an application, the stub asks a cheap question: does this even look like code? It reads the first two words of the candidate's vector table and sanity-checks them:

456	static bool validate_partition(uint32_t addr) {
457	  uint32_t sp = *(volatile uint32_t *)(addr);     /* Stack pointer */
458	  uint32_t rv = *(volatile uint32_t *)(addr + 4); /* Reset vector */
459	
460	  /* Stack pointer must be in RAM range (640KB SRAM) */
461	  if (sp < 0x20000000 || sp > 0x200A0000) {
462	    return false;
463	  }
464	
465	  /* Reset vector must be in flash range and must be odd (Thumb mode) */
466	  if (rv < 0x08000000 || rv > 0x08200000 || (rv & 1) == 0) {
467	    return false;
468	  }
469	
470	  return true;
471	}

The last condition is my favourite piece of processor lore in the whole file. On this architecture every valid function address has its lowest bit set, because that bit selects the instruction set. An even reset vector is therefore not merely wrong — it is impossible. Two comparisons and a parity check reject erased flash, a truncated download, or garbage, in about ten microseconds.

Count the attempts, then roll back

A freshly-installed image is marked pending, not confirmed. The stub increments a counter every time it launches such an image, and only the running application can clear it — by declaring itself healthy. If the application never gets far enough to do that, the counter climbs until the stub gives up on it:

331	  if (status_valid && status.boot_state == BOOT_STATE_PENDING) {
332	    TRACE("[BOOT] T6 -> PENDING path\r\n");
333	    status.boot_attempt_count++;
334	
335	    if (status.boot_attempt_count > status.max_boot_attempts) {
336	      /* Too many failed attempts — try the other partition */
337	      uint8_t other = (status.active_partition == 0) ? 1 : 0;
338	      uint32_t other_addr = (other == 0) ? PARTITION_A_ADDR : PARTITION_B_ADDR;
339	
340	      if (validate_partition(other_addr)) {
341	        /* Roll back to the other (previously confirmed) partition */
342	        status.active_partition = other;
343	        status.boot_state = BOOT_STATE_CONFIRMED;
344	        status.boot_attempt_count = 0;
345	        status.recovery_trigger = RECOVERY_TRIGGER_BOOT_FAIL;

This is a health gate built out of nothing but a counter in flash. It catches the failure mode that validation cannot: an image that is structurally perfect and still cannot run. It also records why the rollback happened, and marks the offending version as failed in a small on-flash history, so the reason survives the reboot.

Make "power died halfway" a real state

The boot state isn't a boolean; it's an enumeration that names each situation the stub can wake up into:

77	  BOOT_STATE_CONFIRMED = 0x00,         /**< Firmware is confirmed healthy */
78	  BOOT_STATE_PENDING = 0x01,           /**< New firmware, not yet confirmed */
79	  BOOT_STATE_FORCE_RECOVERY = 0x02,    /**< CAN/UART safe mode command received */
80	  BOOT_STATE_BOOT_FAILED = 0x03,       /**< Boot attempt counter exceeded */
81	  BOOT_STATE_COPY_IN_PROGRESS = 0x04,  /**< Power-loss guard: B→A copy interrupted */
82	  BOOT_STATE_EMPTY = 0xFF              /**< Erased / uninitialized flash */

Two of these are worth pausing on. COPY_IN_PROGRESS is written before the stub starts copying an image between partitions, so if power fails mid-copy the next boot knows the partition is half-written and simply redoes the copy rather than launching rubble. It's crash-consistency, in a bootloader, in one byte.

And EMPTY is 0xFF — which is exactly what erased flash reads back as. A blank sector therefore isn't an error to be special-cased; it's a legitimate, self-describing state meaning "nothing has been recorded here yet." That's the sort of small alignment with the hardware that keeps a bootloader short.

Writing Around Damaged Flash

Now the part that took real forensics. During development, an earlier update bug wrote to the boot-status sector without erasing it first, while the flash banks were logically swapped. On this part, writing over already-written cells corrupts their error-correction codes permanently — a double-bit error that no amount of re-erasing will fix. Specific words in flash were simply dead, on the affected boards, forever.

Rather than discard the hardware, the team mapped the damage and designed around it. The header carries the minefield:

35	 * Known damaged ECC cells (accumulated from incorrect write-without-erase
...
38	 *   Physical Bank 1 hardware defects (offset within bank):
39	 *     0x08008090  (offset 0x8090, ECC word 0x0809)  — do not write here
40	 *     0x08009090  (offset 0x9090, ECC word 0x0909)  — do not write here
41	 *
42	 *   Physical Bank 2 induced defect (from write-without-erase with SWAP_BANK=1):
43	 *     0x08108140  (offset 0x8140, ECC word 0x0814)  — do not write here

Then the placement is chosen to thread between them — accounting for the structure's own size, the position of its checksum field within it, quadword alignment, and the fact that a bank swap changes which physical cells a logical address lands on. The comment verifies both configurations explicitly:

52	 *   Bank 1 (SWAP_BANK=0): offset 0x81B0-0x824F avoids 0x8090, 0x9090, 0x8140 ✓
53	 *   Bank 2 (SWAP_BANK=1): offset 0x81B0-0x824F avoids 0x8140 (only known defect) ✓

Two ticks at the end of a comment, standing in for a lot of arithmetic. The lasting value wasn't rescuing a few boards — it was the discipline that came out of it: erase before you write, know which physical cells a logical address touches when banks can swap, and protect your critical record with a checksum whose own bytes you've verified are landing somewhere healthy.

The Results

What ships is a sensor module that is genuinely difficult to brick. A structurally invalid image is rejected before it ever runs. A structurally valid image that cannot function is abandoned after a bounded number of attempts and rolled back to the last one that worked. An interrupted copy is finished on the next boot instead of being launched half-written. And if every application on the device is unusable, the stub still comes up and still listens — over the internal bus or a serial line — for a fresh image.

The cost of all that insurance, on the overwhelming majority of boots where nothing is wrong, is under a millisecond. That asymmetry is the whole design: pay almost nothing every time, and have a way home the one time it matters.

Why It Matters at Hoomanely

Hoomanely is reinventing healthcare for pets — replacing reactive, imprecise care with continuous, clinical-grade monitoring that catches problems early. Our devices form a Physical Intelligence ecosystem: sensors fused at the edge, feeding the Biosense AI Engine that turns raw signals into personalized, preventive insights.

Those sensors are microcontrollers buried inside appliances in people's homes. They have to keep improving for years — new capture logic, new calibration, fixes — and every one of those improvements is a chance to brick a board that nobody can reach. A recovery path that an update cannot destroy is what makes shipping firmware improvements a routine act rather than a gamble.

It's also what keeps the promise of continuous monitoring honest. A pet's health record can survive a device rebooting; it cannot survive a device that never comes back.

Key Takeaways

• The updater must live where updates cannot reach. Put recovery logic in a small, write-protected region that is effectively never revised, and keep the application ignorant of it.
• Validate an image before trusting it. Checking that the initial stack pointer lands in RAM and the reset vector is a legal odd (Thumb) flash address rejects erased or truncated firmware in microseconds.
• A counter in flash is a health gate. Mark new firmware pending, increment on each launch, and let only a healthy application clear it — that catches images that are valid but non-functional.
• Name every state, including the awkward ones. An explicit "copy interrupted" state makes power loss mid-write recoverable, and letting erased flash (0xFF) mean something keeps the code short.
• Erase before you write, and know your physical geometry. Writing over written cells can destroy their error-correction permanently — and when banks can swap, a logical address is not a fixed physical cell.

Author's Note

This recovery stub guards the camera-and-thermal sensor module inside Hoomanely's Everbowl, one of the physical-intelligence devices in our ecosystem. It is a few kilobytes of deliberately boring code that runs for under a millisecond and then steps aside, and its entire purpose is to be the thing still standing when something else has gone wrong. The comments mapping dead flash cells are my favourite artefact in the codebase — a mistake, turned into a map, turned into a rule.