Building a Multi-Gigabyte Flash Storage Pipeline for Real-Time Sensor Fusion

Building a Multi-Gigabyte Flash Storage Pipeline for Real-Time Sensor Fusion

By Vaishak C

In the world of IoT and edge AI, sensor fusion systems generate large amounts of data that must be processed, stored, and transmitted in real time. Our challenge was building an embedded storage pipeline capable of handling continuous multi-sensor data streams, thermal imaging, high-resolution camera feeds, and proximity measurements, while meeting strict latency requirements for time-critical applications.

The numbers were demanding: multiple gigabytes of sensor data per hour, with individual camera frames reaching 529KB and thermal datasets needing sub-100ms storage latency. Traditional embedded storage approaches couldn't meet these demands without sacrificing either performance or reliability.

Architecture overview: multi-layer storage strategy

Most embedded systems rely on simple filesystems or basic flash management, but high-throughput sensor fusion needs something more sophisticated. We needed burst handling for multiple simultaneous sensor readings, a buffering strategy for transmission delays, reliability against power failures and corruption, and minimal impact on real-time sensor processing.

We designed a three-tier architecture, each layer optimized for its role.

Tier 1: PSRAM buffer layer. High-speed temporary storage for active sensor captures, using 32MB of external PSRAM for large camera frame buffering, ring buffer management for continuous data streams, and sub-10ms access times for real-time processing.

Tier 2: flash storage layer. 128MB of OCTOSPI flash for persistent storage, LittleFS with wear leveling, a circular buffer design for space efficiency, and DMA-accelerated writes for a meaningful performance boost.

Tier 3: transmission layer. CAN FD protocol for reliable data streaming, adaptive LZ4 compression for bandwidth optimization, priority queuing so time-critical data goes first, and automatic retry and error recovery.

OCTOSPI flash optimization

Moving from traditional SPI to OCTOSPI provided the bandwidth foundation for our high-throughput requirements. Standard SPI tops out around 50 Mbps; OCTOSPI delivers well over 200 Mbps theoretical bandwidth, which matters for sustaining multi-gigabyte-per-hour writes.

The real breakthrough was DMA-based flash operations. Traditional CPU-based writes created bottlenecks during large camera frame storage. Our implementation uses task-based completion notifications through FreeRTOS primitives, interrupt-driven DMA callbacks for non-blocking operation, automatic retry on error, and page-aligned writes for optimal flash controller performance:

// DMA completion handling with task notifications
static void DMA_CompletionCallback(void) {
    BaseType_t xHigherPriorityTaskWoken = pdFALSE;
  
    // Notify waiting task
    vTaskNotifyGiveFromISR(dma_completion_task, &xHigherPriorityTaskWoken);
    portYIELD_FROM_ISR(xHigherPriorityTaskWoken);
}

Moving from CPU-based writes to DMA-accelerated writes delivered a meaningful throughput increase while dramatically cutting CPU utilization during storage operations, freeing the core for sensor processing instead.

Raw flash management alone wasn't enough for our reliability bar, so LittleFS provided wear leveling through automatic block rotation, power-safe atomic writes that prevent corruption, compression support that reduces storage requirements, and flexible configuration tuned to our access patterns.

Circular buffer management

With our sustained throughput and 128MB of flash capacity, we had well under a full day of storage, insufficient for extended offline operation. The fix was intelligent circular buffer management that goes beyond simple overwrite logic.

Transmission-based cleanup tracks transmission status for each data segment and automatically reclaims space after successful CAN transmission, with thermal data (time-critical) prioritized for transmission first. Intelligent overflow handling triggers pre-transmission at 80% capacity, runs emergency cleanup for critical storage situations, and degrades gracefully by reducing capture frequency rather than losing data outright.

// Space management with transmission awareness
typedef struct {
    uint32_t sequence_id;
    transmission_status_t status;  // PENDING, SENDING, SENT, FAILED
    uint32_t content_size;
    uint32_t content_offset;
    uint8_t retry_count;
} storage_entry_t;

In production, this delivers an average write latency in the low tens of milliseconds for 529KB camera frames, high effective storage utilization, zero corruption events across an extended field deployment, and a high first-attempt transmission success rate.

Handling real-world challenges

LittleFS cache assertions became our biggest production challenge. Under heavy load, the filesystem would occasionally hit internal assertion failures, needing a staged recovery process: a graceful cache reset attempting to clear and reinitialize the cache, an aggressive recovery reallocating buffers and restarting the filesystem, an emergency mode forcing a remount with data preservation, and a hardware reset as the last resort.

We also built comprehensive hang detection: operation timeouts limiting storage operations, watchdog integration for hardware-level recovery, progressive retry logic with exponential backoff, and continuous health monitoring.

Across an extended field deployment, uptime stayed high excluding planned maintenance, sustained throughput matched design targets with margin at peak, storage latency stayed in the tens of milliseconds at the 95th percentile, and the handful of cache reset events that did occur resulted in zero data loss. Flash wear stayed low relative to rated erase cycles even after writing many terabytes cumulatively.

Why it matters at Hoomanely

This storage pipeline is the technological foundation of our push to move pet healthcare from reactive to preventive care. Our edge AI platform continuously monitors pets through multi-sensor fusion, generating clinical-grade intelligence at home: continuous 24/7 thermal, camera, and proximity data collection, early detection of health anomalies before symptoms appear, personalized health baselines, and technology that strengthens the human-pet bond through better understanding. The robust storage architecture ensures no data loss during critical health monitoring, while sub-100ms latency supports real-time health alerts.

Key takeaways

DMA-accelerated OCTOSPI flash operations deliver a meaningful performance boost with dramatically reduced CPU load. Sub-100ms storage latency is achievable for time-critical sensor fusion applications. Multi-tier architecture is essential for high-throughput embedded storage under real constraints. Proactive space management is more reliable than reactive overflow handling. And recovery mechanisms have to be tested extensively under real-world failure conditions, not just in the lab, since compression algorithms and transmission-aware storage management both meaningfully improve system efficiency once they're battle-tested.

Read more