Field-Proven OTA : Partition Strategies and Brick-Proofing

Designing for interrupted updates and power-loss resilience in real-world embedded systems

In the unforgiving world of IoT tracking devices, where power interruptions, network failures, and harsh environments are the norm rather than the exception, implementing robust Over-The-Air firmware updates can be the difference between a successful product and expensive field failures. After deploying thousands of tracking devices in mission-critical applications, we've learned that the traditional OTA approach simply isn't enough when real-world conditions meet Murphy's Law.

Why traditional OTA fails in the field

Most embedded developers start with a simple approach: download firmware, overwrite existing code, reboot. That works perfectly in the controlled environment of a lab, but fails badly in the field, in GPS trackers bouncing around, asset monitors in remote locations with unstable power, or tracking collars where battery depletion can strike mid-update.

The dual-bank solution

The most battle-tested approach for mission-critical devices is dual-bank (A/B) partition architecture with an always-on bootloader. This is the strategy we've implemented on our STM32-based platform, and it has achieved zero field bricks across thousands of deployed units.

The design is simple: the active firmware is never touched during updates. New firmware downloads to the staging bank, gets validated, and only becomes active after successful verification. If anything goes wrong, power loss, corruption, an invalid CRC, the device simply keeps running the proven firmware from the active bank.

Bank switching happens atomically through a single option byte modification:

// STM32H5 Implementation - Atomic Option Byte Modification
void vbus_ota_switch_bank(void) {
    HAL_FLASH_OB_Unlock();
  
    // Toggle SWAP_BANK bit atomically
    if (FLASH->OPTSR_CUR & FLASH_OPTSR_SWAP_BANK_Msk) {
        CLEAR_BIT(FLASH->OPTSR_PRG, FLASH_OPTSR_SWAP_BANK_Msk);
    } else {
        SET_BIT(FLASH->OPTSR_PRG, FLASH_OPTSR_SWAP_BANK_Msk);
    }
  
    HAL_FLASH_OB_Launch();  // Atomic commit + reset
}

Our implementation uses a three-stage validation process: pre-flash validation through CRC verification before any write operations, post-flash validation through stack pointer and reset vector checks, and runtime validation through a health heartbeat within the first minute or two of first boot. If the new firmware fails to send that heartbeat within the trial window, the bootloader automatically reverts to the previous bank, no cloud connectivity required for recovery.

CAN FD optimization for transfer

Modern tracking applications often need high-bandwidth firmware updates. Our implementation uses CAN FD with 64-byte frames instead of classic CAN's 8-byte limit, which meaningfully improves throughput while keeping industrial-grade reliability. Bitmap-based chunk tracking handles out-of-order delivery and duplicates gracefully, STM32H5-optimized flash operations use 16-byte aligned writes for maximum performance, and intelligent buffering reduces flash wear while maintaining data integrity.

Storage and network resilience

Dual-bank architecture needs roughly 2x flash storage for firmware, but external QSPI flash makes that cost-effective: internal flash holds the critical bootloader and application code in dual-bank form, while external QSPI handles data logging, configuration, and staged firmware, keeping only the critical code paths on the more expensive internal flash.

Tracking devices often work in challenging RF environments, so we implemented progressive retry with exponential backoff:

// Progressive retry with exponential backoff
uint32_t retry_count = 0;
const uint32_t MAX_RETRIES = 5;
uint32_t retry_delay = 1000; // Start with 1 second

while (retry_count < MAX_RETRIES && !transfer_complete) {
    if (attempt_chunk_download(chunk_id) == SUCCESS) {
        retry_count = 0;  // Reset on success
        retry_delay = 1000;
    } else {
        retry_count++;
        retry_delay = min(retry_delay * 2, 30000);  // Cap at 30s
        vTaskDelay(pdMS_TO_TICKS(retry_delay));
    }
}

Field deployment also taught us the value of comprehensive failure telemetry: tracking update attempt success and failure rates by device model and firmware version, voltage monitoring during critical update phases, flash health monitoring for wear leveling and bad blocks, and network quality metrics for signal strength and packet loss during transfers.

Regulatory compliance

The current regulatory landscape, particularly the EU Cyber Resilience Act, mandates robust security update mechanisms throughout a product's lifecycle. Dual-bank architecture naturally supports this through authenticated updates with cryptographic signature verification before activation, automatic rollback on validation failure, complete audit trails of update history and validation logs, and field-proven upgrade paths for long-term security patch support.

Performance in the field

Our production implementation delivers strong real-world numbers: a zero brick rate across thousands of deployed tracking devices, a first-attempt success rate well above 99%, and a high recovery rate from power-interrupted updates. Transfer performance runs a few minutes for a 1MB firmware update over CAN FD, with a meaningful reduction in flash wear compared to earlier implementations, and automatic rollback completing in under 30 seconds on failure detection.

Key takeaways

Invest in dual-bank architecture early, retrofitting it later is far more expensive than designing it in from the start. Plan for worst-case scenarios, if it can fail in the field, it will, so design recovery mechanisms accordingly. Validate everything, twice, CRC checks, stack pointer validation, and runtime health monitoring are non-negotiable. Optimize for your transport, whether CAN FD, cellular, or LoRaWAN, and tailor the transfer strategy to that medium's strengths and limits. And monitor and iterate, field telemetry from deployed updates gives invaluable insight for improving the OTA strategy over time.

Why it matters at Hoomanely

Our tracking and monitoring solutions have to thrive in the unpredictable conditions where they actually operate, not just in the lab. The field-proven OTA strategies here are integral to our platform, keeping tracking solutions secure, updatable, and resilient throughout their operational lifetime, so pet families can rely on continuous, uninterrupted monitoring.