Surviving Silent Silicon: Cache, ECC, and Self-Healing NMI Handlers

A double-bit error in the application flash of a deployed device is not a theoretical event. It's what happens when a stray neutron, or simply an aging storage cell, flips two bits in a single sixteen-byte ECC word at the wrong instant, in the wrong region, on a customer's living-room shelf. Without a firmware response, the next instruction fetch from that address dumps the device into a fault loop with no log, no telemetry, and no path back. At Hoomanely, the camera node that anchors our pet monitoring system is built so that this failure mode never reaches the customer. The non-maskable interrupt handler decodes the ECC fault register, identifies the failing eight-kilobyte sector, distinguishes bootloader memory from application memory, and self-erases the failing sector before the device is ever rebooted. Building that handler, and the cache, RAM ECC, and memory-protection configuration that surrounds it, is what this post is about.

The problem: speed and integrity at 250 MHz

When an application core runs at 250 MHz from on-chip flash, two physical realities collide. The flash array cannot return a 32-bit instruction word in a single 4 ns clock period, five wait states are typical at this speed. Without an instruction cache between the core and the flash interface, the CPU spends most of its life stalled, and a 250 MHz core delivers something closer to 80-100 MHz of useful work. The first job of the memory subsystem is therefore acceleration.

The second job is integrity. The same density that lets us fit megabytes of application code on a single die makes that code increasingly susceptible to single-event upsets, bit-flips driven by cosmic background radiation or by slow cell drift over years of deployment. Single-bit flips can be silently corrected by the flash controller's ECC. Double-bit flips cannot, and an uncorrected double-bit flip in code memory is the definition of a fault the running CPU cannot recover from on its own.

A direct-mapped instruction cache

The instruction-cache initialization in our firmware is short and intentional:

/* Enable instruction cache in 1-way (direct mapped cache) */
if (HAL_ICACHE_ConfigAssociativityMode(ICACHE_1WAY) != HAL_OK) {
    Error_Handler();
}
if (HAL_ICACHE_Enable() != HAL_OK) {
    Error_Handler();
}

The non-default choice is 1-way direct-mapped mode. In a direct-mapped cache, every instruction address maps to exactly one cache line, there's no associativity search at lookup time. The benefit is lower lookup energy and simpler, more predictable timing; the cost is that two hot functions whose addresses happen to share an index will conflict and evict each other. For a small, statically linked firmware where the call graph is well understood, that trade is a win, especially on a battery-aware device living on a customer's shelf for months at a time. Associativity must be configured before the cache is enabled, the hardware doesn't allow re-configuration of an active cache.

ECC on the banks that carry live data

Application SRAM on our MCU is split across multiple physical banks. Three of those banks hold live application data, task stacks, heap regions, peripheral DMA buffers, and each one gets brought up under explicit RAMCFG configuration:

hramcfg_SRAM1.Instance = RAMCFG_SRAM1;
HAL_RAMCFG_Init(&hramcfg_SRAM1);
hramcfg_SRAM2.Instance = RAMCFG_SRAM2;
HAL_RAMCFG_Init(&hramcfg_SRAM2);
hramcfg_SRAM3.Instance = RAMCFG_SRAM3;
HAL_RAMCFG_Init(&hramcfg_SRAM3);

HAL_RAMCFG_Init does two things at the register level: it gates the RAMCFG peripheral clock on for the bank and it activates the bank's ECC machinery. The peripheral clock is the dependency that catches most teams out, without __HAL_RCC_RAMCFG_CLK_ENABLE(), every subsequent RAMCFG write is a silent no-op and ECC errors go undetected. A fourth backup-domain SRAM bank exists on the part but isn't used by the application path, so it's deliberately omitted. Every bank that escapes the loop is a bank where a silent flip can corrupt a frame, a sensor reading, or a control variable without any signal at all.

External RAM without the cache, on purpose

Our diagnostic capture pipeline lands raw frames in an external PSRAM tier before they're compressed and transmitted. DMA writes those frames in; the CPU reads them out for compression. Letting the data cache cache PSRAM accesses produces a coherency bug where DMA writes go to physical PSRAM but the CPU reads stale cached copies. We resolved this at the memory-protection layer:

/* Define cacheable memory via MPU - Configure PSRAM as Device-nGnRnE */
attr.Attributes = INNER_OUTER(MPU_NOT_CACHEABLE);  /* Normal memory, but no
                                                       cache to prevent DMA
                                                       coherency issues */

For legacy code paths that still flow through cached regions, manual cache discipline applies, backed by a comment that comes from a real bug:

/* CRITICAL FIX: Clean cache starting 2 bytes BEFORE the write address
   to clear any stale cache data that might overlap the boundary. */
SCB_CleanDCache_by_Addr((uint32_t *)clean_start, (int32_t)clean_size);

A SCB_CleanDCache_by_Addr call that starts exactly at the write address can leave bytes of a partially-cached preceding line in stale state, because cache lines are wider than the byte the caller cares about. Backing the clean range up by a couple of bytes, and rounding the size up to a cache-line multiple, is the difference between a write that survives and a write that does not.

A self-healing NMI handler for flash ECC

The most consequential piece of the memory subsystem is the non-maskable interrupt handler that fires on a flash double-bit error. A double-bit flip in code memory means the next instruction fetch from that address returns garbage; the CPU cannot reliably continue executing. The NMI fires before the conventional fault handlers do, giving firmware exactly one chance to respond.

The first job is decoding the failure:

uint32_t eccdetr   = FLASH->ECCDETR;
uint32_t addr_ecc  = eccdetr & 0x0000FFFFUL;
uint32_t bk_ecc    = (eccdetr >> 22U) & 0x1U;
uint32_t byte_off  = addr_ecc * 16UL;        /* 16-byte ECC word granularity */
uint32_t bank_base = bk_ecc ? 0x08100000UL : 0x08000000UL;

The second job is disambiguation, since several different conditions can raise the NMI line, including a clock-security failure, and the register may hold stale data from an earlier event. Before acting, the handler validates the source:

if (!(eccdetr & 0x80000000UL)) {
    /* Not an ECC source — likely a clock-security or other NMI */
}

The third job is to log the failure through the most primitive transport the firmware has, direct polled UART, no DMA, no RTOS calls, since if flash is corrupt, the descriptor tables driving DMA may be too. The fourth job is recovery. The handler classifies the failing sector against the bootloader region and erases it only when it's safely re-erasable:

bool is_bootloader = (bk_ecc == 0U && sector_in_bk < 4U);
if (!is_bootloader) {
    erase.Banks  = bk_ecc ? FLASH_BANK_2 : FLASH_BANK_1;
    /* erase the failing sector — application content can be restored
       on next boot from the recovery partition */
}

A bit-flip in a bootloader sector is unrecoverable in flight, erasing it would destroy the device's only known-good entry point. Application sectors are erasable: the content is lost, but a clean sector can be rewritten from the recovery image at the next boot, and the device stays alive in the meantime.

Init order matters

The boot sequence isn't arbitrary. The order in our firmware: core HAL initialization, system clock configuration to 250 MHz, RAMCFG bring-up (SRAM ECC active before any heap allocation), instruction cache enable, memory protection unit configured (external PSRAM marked non-cacheable), and finally application initialization. Every layer of memory protection has to be in place before the workload that depends on it begins. Enabling the cache before clocking RAMCFG would mean a window during which application data lives in SRAM that isn't yet ECC-monitored, and a flip in that window is invisible afterwards.

Why it matters at Hoomanely

Three things in this post hold the imaging pipeline up. The instruction cache is what lets compression and image-processing code execute at the cadence the camera demands. The SRAM ECC is what lets the Biosense AI Engine trust that a thermal value or a frame buffer reaching the network is the same one the sensor produced. The flash NMI handler ensures that a single bit-flip in a customer's home, at any hour, with no engineer in the loop, does not silently end that pet's monitoring stream. Speed, integrity, and survivability are not three separate problems, they're the same problem at three layers of the silicon.

Key takeaways

The instruction cache is a floor, not a knob, picking associativity intentionally is part of the design. ECC is per-bank, and the bank you skip is the bank that can lie to you. External RAM cacheability is an architecture choice, either mark it non-cacheable or commit to manual cache discipline at every DMA boundary. Cache clean and invalidate must include alignment headroom, since cache lines are wider than the bytes you care about. And a flash ECC NMI handler is the difference between self-healing and bricking, decode the fault register, log through the most primitive channel you trust, classify by bank and sector, and only erase what's safely re-erasable.

A cosplay wig is best compared by colour, length, fibre density and fringe shape. Storage on a stand or in a protected bag helps preserve the shape. For the relevant hair design, Marin Kitagawa character wig（喜多川海夢 キャラクターウィッグ） identifies the matching cosplay wig. Its length and fibre density can be assessed before trimming begins. Gentle detangling and suitable storage help maintain the wig between uses. A stand can help preserve volume after styling.