OTA firmware updates: how IoT devices update in the field without bricking
An OTA (over-the-air) firmware update replaces the software on a device remotely, without a cable or a technician. Done well, it is the mechanism that lets a product improve for years after it ships. Done badly, it is the fastest way to turn a fleet into bricks. A dependable IoT OTA update system needs five parts: a signed and versioned image, a transport that survives bad networks, an A/B partition layout so the old firmware is never overwritten in place, a health gate that decides whether the new firmware stays, and a rollout controller that updates a few devices before updating all of them. This guide covers each part, the failure modes that matter, and how to design the system so a failed update costs a retry instead of a return.
The short version
- An OTA update is six steps: publish, notify, download, verify, switch, confirm or roll back. Most failures live in download and in a missing rollback.
- Write the new image to a slot the device is not running from (A/B), and never write to the bootloader.
- The new firmware must prove it works within a deadline, commonly 5 to 15 minutes, or the bootloader reverts; a boot counter of 3 catches boot loops.
- Update a canary group first, then 5 to 10 percent, then the fleet, with a stop rule such as confirmation rate below 98 percent.
How an OTA firmware update works
Every OTA firmware update follows the same sequence, whether the device is a microcontroller with 512 KB of flash or an embedded Linux board.
- Publish. A build server produces a firmware image, signs it with a private key, and records its version, hash and target hardware revision in an update manifest.
- Notify. Devices learn an update exists, either by polling an update server on a schedule or by receiving a push message over MQTT or a similar channel.
- Download. The device fetches the image in chunks, verifies each chunk, and writes it to an inactive storage slot. Downloads must resume after a dropped connection rather than restart.
- Verify. The device checks the signature and hash of the complete image against the manifest before anything else happens.
- Install and switch. The device marks the new slot as the boot target and reboots.
- Confirm or roll back. On first boot the new firmware must prove it works within a time limit. If it does, it is confirmed as the permanent image. If it does not, the bootloader switches back to the previous slot.
Steps 3 and 6 are where most systems fail. Step 3 fails when the download writes over the running firmware and power is lost halfway. Step 6 fails when it does not exist: the device boots the new image, something is broken, and there is no path back.
The A/B partition layout
The A/B (dual-slot) layout gives the device two copies of the firmware. Slot A holds the running image; the update is written to slot B. Only after slot B is complete and verified does the bootloader switch to it. Slot A is untouched, so if slot B fails, the device boots A again.
The cost is flash. A/B doubles the space needed for the application, which is why small microcontrollers often ship with in-place updates and a prayer. On parts with 1 MB or more, or with external flash, A/B is the right default. On STM32 parts with dual-bank flash, the hardware can swap banks atomically; our post on firmware updates with bank switching on STM32H5 walks through that implementation, and designing an A/B swap partition for a brick-proof bootloader covers the layout decisions for parts without hardware support.
For embedded Linux, the same idea applies to root filesystems: two root partitions, a bootloader (U-Boot or GRUB) that reads a boot flag, and an update tool such as RAUC, Mender or SWUpdate that writes the inactive partition. The post on A/B OTA updates that cannot brick a device describes how we run this on a Linux fleet.
The bootloader is the part you cannot update
The bootloader decides which slot to boot and must itself be correct forever, because an OTA update that breaks the bootloader has no recovery path. Three rules follow.
Keep the bootloader small and dumb. It should verify a signature, read a boot flag and jump. It should not parse networks, file systems or configuration formats.
Protect it from writes. Lock the bootloader region with the MCU’s flash protection or the SoC’s secure boot chain, so no application bug or update can overwrite it. The post on the recovery bootloader that cannot be overwritten describes how we enforce this.
Give it a last resort. A recovery mode, entered by a button hold or by a boot counter that exceeds its limit, that can download a known-good image over a simple transport. Field experience with this layout is in field-proven OTA partition strategies and brick-proofing.
Signing, versioning and the manifest
A device must never run an image it cannot verify. Sign every image with an asymmetric key (Ed25519 or ECDSA P-256 are standard choices); store only the public key on the device; keep the private key in a hardware security module or a cloud KMS, never on a developer laptop. Verify the signature before writing the boot flag, not after.
The manifest ties the image to a context: version number, hardware revision it applies to, minimum version it can upgrade from, image hash, and signature. Two manifest rules save fleets. First, enforce anti-rollback: a device must refuse any image with a version lower than its current one unless a specific downgrade flag is signed into the manifest, otherwise an attacker can replay an old vulnerable image. Second, check hardware revision before download, so a revision B device never flashes a revision A image that drives a pin the wrong way.
The health gate: how the device decides the update worked
Booting is not success. The new firmware must demonstrate that it works, and the definition of “works” has to be decided before the update ships. A health gate is a checklist the new firmware runs on first boot, with a deadline. Typical checks: the watchdog is being fed, the main application task started, the device connected to its network, the device reached the update server and reported its new version, and the primary sensor returned a plausible reading.
If every check passes within the deadline (commonly 5 to 15 minutes), the firmware calls a confirm function that writes a flag the bootloader reads. If the deadline passes without confirmation, or the device reboots more than N times (a boot counter in persistent memory, typically N equals 3), the bootloader reverts to the previous slot. The rules around this gate decide whether a bad update becomes a support incident or a non-event. The devices in our fleet that fix themselves this way are described in the self-healing fleet.
Staged rollouts: never update everyone at once
A firmware image that passed every test still meets conditions in the field that the lab did not have: a router model that drops long TLS sessions, a sensor batch with slightly different timing, a home where the device loses power at 3 a.m. every night. Staged rollouts find these before they affect the whole fleet.
The standard pattern is canary, stage, fleet. Push to an internal canary group (your own devices and a few opted-in users), watch confirmation rate, crash rate and connectivity for 24 to 72 hours, then widen to 5 to 10 percent, then to everyone. Define the stop condition in advance: for example, halt the rollout if the confirmation rate for the new version falls below 98 percent or if the rollback count exceeds a threshold. The rollout controller should be able to pause and resume without touching devices that already updated. Our post on shipping an OS to live homes with canary, stage and fleet rollouts describes how we run it.
Schedule updates for when the device is idle and powered. A device in the middle of its primary job should defer, and a battery-powered device should require a minimum charge level before writing flash.
Failure modes and what prevents them
The last row deserves emphasis. Hot updates and manual flashes over SSH are tempting shortcuts during an incident, and they destroy the one property a fleet needs: knowing which code every device runs. We removed that path on purpose; the reasoning is in hot updates: the shortcut we disabled.
How EverBowl™ updates safely
EverBowl™ learns how a dog eats and drinks over months, and that record is only useful if the firmware behind it keeps improving without putting a single bowl at risk. Every update is signed, written to an inactive slot, confirmed by a health gate on first boot and rolled out in stages, so an update that fails in one home reverts on its own and never reaches the next. EverBowl does not diagnose anything; the value is the record, and a bowl that is offline after a bad update is a gap in that record. ## Conclusion
A dependable OTA firmware update system is mostly about what happens when things go wrong. Sign and version every image, write it to a slot the device is not running from, make the new firmware prove itself before it is kept, protect the bootloader from every write path, and update a handful of devices before you update all of them. Each of those decisions is cheap at design time and expensive to retrofit after the first bad update reaches the field.
FAQs
What is an OTA firmware update?
An over-the-air firmware update delivers new device software remotely over WiFi, cellular or another wireless link, verifies it on the device, and installs it without physical access. It is how connected products fix bugs and add features after they ship.
What is an A/B partition in OTA updates?
A layout with two firmware slots. The device runs from one slot while the update is written to the other; the bootloader switches only after the new image is verified, and switches back if the new image fails. It prevents a failed or interrupted update from leaving the device unbootable.
How does firmware rollback work?
The bootloader keeps a boot counter and a confirmation flag. If the new firmware does not confirm within a deadline, or the device reboots more than a set number of times, the bootloader boots the previous slot again.
How should OTA updates be secured?
Sign images with an asymmetric key kept in an HSM or cloud KMS, verify the signature on the device before switching slots, enforce anti-rollback so older versions are refused, deliver over TLS, and lock the bootloader region against writes.
What is a staged rollout?
Releasing an update to a small group first (a canary), monitoring confirmation and failure rates, then widening to a larger stage and finally the whole fleet, with predefined conditions that pause the rollout automatically.
Are delta OTA updates worth it?
They reduce download size by 70 to 90 percent for small changes but require a patch algorithm on the device and an exact match with the base image. They pay off on cellular or metered links; a first product can usually skip them.
