Ghost in the Shell: Validating Firmware Logic with Mock Sensor Pipelines

Firmware that passes physical hardware testing can still ship with bugs that vanish the moment you go looking for them, only to appear disastrously in the field under chaotic conditions. We call these Heisenberg bugs. The fix we landed on, which we call the Ghost strategy, introduces an injection point at the seam between the driver and logic layers, feeding the logic layer synthetic events indistinguishable from real physical triggers. A random trigger task running directly on the microcontroller inside the RTOS simulates an erratic external environment, uncovering race conditions, buffer overflows, bus contention, and logic fallthroughs that physical sensor inputs almost never reproduce reliably. Running the device headless overnight compressed weeks of real-world usage into a single night. Because the scaffolding code was written with the same rigor as production firmware, the Ghost can be re-enabled at any time to reproduce field conditions on a desk, no physical hardware required.

The paradox of hardware-dependent firmware

Embedded systems engineers live with a persistent paradox: firmware exists to control and react to physical hardware, yet that same dependency often becomes the biggest bottleneck in software velocity. This creates a hardware-loop latency that stifles iteration. To validate a simple logic change, a developer might need to flash a device, physically manipulate a sensor, observe the result, and repeat.

That friction gets worse in systems with complex, high-throughput sensor pipelines, thermal cameras, LiDAR arrays, proximity-triggered edge devices. The happy path is easy to test. The edge cases, sensor timeouts, bus contention, buffer overflows during rapid bursts, are notoriously hard to reproduce physically. How do you reliably wave a hand in front of a proximity sensor exactly 4 milliseconds after a flash memory write cycle begins? A human hand simply isn't that precise.

So firmware often ships with bugs that disappear when you look for them slowly, but appear disastrously in the field under real chaos.

This post covers the methodology we used to validate a high-performance heterogeneous imaging pipeline. By decoupling business logic from the physical world and building a software-defined simulation of external events, a "Ghost" in the shell, we validated memory safety, concurrency, and logic flow under conditions that would have been nearly impossible to reproduce with physical inputs alone.

The system under test

To understand why this approach mattered, it helps to look at the system in the abstract. The device was a dual-sensor imaging unit for edge deployment, with a high-resolution thermal sensor and a visual camera sensor. It ran in a high-speed burst mode: capturing rapid sequences of data, buffering them in volatile memory, offloading to non-volatile storage, and eventually transmitting to a host over a high-speed bus.

The pipeline had four distinct, asynchronous, competing stages. Trigger is an external asynchronous event, an interrupt from a proximity sensor, that starts the sequence. Acquisition is the parallel capture of thermal matrices and visual frames under strict timing constraints, since the thermal sensor needs a rigid refresh rate while the visual camera pushes megabytes through a DMA channel. Storage is a high-speed move into a circular buffer, followed by a lower-priority background task draining that buffer to persistent storage. Transmission is the asynchronous packetization and sending of stored data.

The complexity doesn't live in any one stage, all of them are well understood individually, it lives in their intersection. What happens if a new trigger arrives while storage is 95 percent full? What happens if transmission locks storage for a read exactly when acquisition tries to write new data? These race conditions are where hard faults, deadlocks, and silent data corruption breed.

Validating these intersections with a physical sensor is erratic at best. A developer can't reliably generate a burst of triggers with the millisecond precision needed to hit these specific race windows. To really validate the firmware, we had to remove the physical world from the equation.

The philosophy of the Ghost: internal stimulus injection

The core idea was simple: treat sensor inputs as data streams, not physical obligations. In a well-architected firmware codebase, the driver layer translates physical interrupts into system events, and the logic layer consumes those events without caring where they came from, only that they occurred.

By introducing an injection point at the seam between these layers, we could feed the logic layer synthetic events indistinguishable from real physical triggers. We called this the Ghost strategy.

The random trigger task. We implemented a dedicated task inside the RTOS. This task served as the Ghost's user. It wasn't a unit test running on a PC host, it was a living task running on the actual microcontroller, competing for CPU cycles and bus access exactly like any other task.

Its job was to simulate an erratic, sometimes aggressive external environment, running on a simplified but highly configurable state machine. First, the wait: the task sleeps for a random interval, typically 0 to 20 seconds. Randomness matters here, since a fixed interval lets the system settle into a rhythm that masks race conditions, while a random one lands triggers at arbitrary points in the system's execution, sometimes idle, sometimes busy. Second, the actuate: the task fires a simulated proximity trigger event, a full emulation of the interrupt handler's downstream effect, not just a function call. Third, the observation: it monitors global system state, atomic flags and mutexes, to verify whether the system reacted correctly or incorrectly. Fourth, the signal: the task uses the device's physical status LEDs to report its virtual status, giving human observers a visual feedback loop.

This decoupling let us run the device headless. We could leave it on a desk overnight, powered but with no physical movement around it, and wake up to find it had processed thousands of capture cycles. That effectively compressed weeks of real-world usage into a single night.

Deep mocking: synthesizing the protocol, not just the signal

A fair criticism of mocking is that it often skips the complexity of the data packet itself. Real sensors don't just say "trigger," they send metadata, timestamps, validity flags, signal strength, checksums. A naive mock that just calls a start function misses the vulnerability in the payload parsing logic.

Our Ghost went deeper. It didn't just trigger the logic, it constructed a full command packet, identical to what a remote controller or a complex sensor hub would send, using the system's binary serialization library to generate valid, binary-compatible payloads.

Mock command IDs let us trace a specific trigger through the logs. If the Ghost fired trigger #5044, we could check the filesystem and confirm the corresponding image file existed, closing the loop on data integrity. Mock timestamps let us measure time-of-flight for the data processing pipeline, calculating exactly how long it took from the virtual hand wave to the file-closed event.

By injecting these full packets into the command queue, we tested the entire stack: the deserialization layer, the command dispatcher, the queue depth management, and the logic handler. We were fuzz-testing our own internal protocols.

Validating the happy path and the chaos

The real value of the Ghost showed up once we adjusted the aggression of the simulation.

The happy path. Initially we set the random interval to a comfortable duration, letting the pipeline fully flush between triggers. Capture restarted, buffers drained, the radio transmitted. The Ghost task had access to the pipeline's completion flags, so when the logic reported pipeline complete, it fired a success command. To an observer, the device sat on the desk, periodically blinking one color for the trigger and another for success, giving immediate, glanceable confidence in baseline functionality. It worked like a heartbeat monitor for the code.

The chaos mode. Then we cut the interval. We let the randomizer pick near-zero delays and trigger bursts of events in rapid succession. This is where the virtual world caught bugs the physical world had missed.

We found buffer exhaustion: if a trigger arrived exactly when the circular buffer was wrapping around, and the write operation lagged by a few milliseconds, the DMA controller would fail to allocate a slot. In the physical world this looked like a "glitch." In the Ghost world it was a reproducible buffer overflow log, and we tuned buffer sizes and watermarks based on that data.

We found bus contention: the thermal sensor needed a monopoly on the communication bus during readout, but external triggers were waking the proximity driver to poll. The Ghost let us identify exactly when the contention happened, and we implemented a "zone of silence," a software interlock where the mock task mimicked muting the proximity sensor during thermal acquisition.

We found logic fallthroughs, probably the most insidious bug: a success indication would occasionally get followed instantly by a busy indication, confusing users. The Ghost, firing completion events in a predictable, high-frequency stream, made the pattern obvious: the code was missing a conditional flow statement, letting execution fall through from success into busy checks. A human waving a hand every few minutes would never have noticed the correlation.

Visual feedback as a debug interface

One of the most effective parts of this implementation was repurposing the hardware status LEDs as a real-time debug interface. In a typical production environment, LEDs are for the end user. In our mock environment they became our console: a trigger color meaning "the Ghost pressed the button," a success color meaning "the logic layer reports success," an inhibited color meaning "the logic layer reports busy," and an error blink meaning "the logic layer rejected my stimulus."

This let developers debug complex state machine interactions without hooking up a hardware debugger. We could see the backpressure algorithm kicking in. If the LEDs started blinking the error color during a simulated storm, we knew our flow control logic was correctly protecting the memory heap.

Scaffolding code is production code

This approach highlights a broader principle often overlooked in embedded development: scaffolding code deserves the same rigor as production code. Test code often gets treated as second-class, scripts hacking together external tools, temporary files that get discarded. But the random trigger task and its helper functions followed the project's variable naming conventions, used the standard logging macros, and were integrated into the system's initialization sequence.

That rigor meant other engineers could read the test logic and understand exactly what was being tested, effectively documenting expected system behavior. It meant we could reconfigure burst size, random seed, or payload type via preprocessor directives, pivoting from stability testing to stress testing in seconds. And it meant that although the mock task is disabled in the final production binary, it stays in the codebase, effectively shipping with the source. If a field issue surfaces months later, we can re-enable the Ghost, configure it to reproduce that field condition, and debug it on a desk without flying to the customer site.

Where this leaves us

The Ghost in the Shell technique, mocking sensor inputs through internal software tasks, changed how we validate firmware. It moved us from physical, anecdotal testing, "it seems to work when I wave my hand," to rigorous, automated logical verification.

By injecting simulated events at the architectural seam between the driver and logic layers, we validated the entire downstream pipeline: buffering, storage management, filesystem integrity, and transmission protocols. We found and fixed race conditions that were statistically unlikely in a manual test session but statistically guaranteed across a fleet of thousands of devices.

The most effective way to validate hardware-dependent code turns out to be removing the hardware from the equation. Once the logic is verified in a pure, controlled simulation, the system is ready when the real world finally shows up.