Watchdogs and Livelock: Real Failure Patterns and Fixes

Many embedded systems behave flawlessly during development. They pass unit tests, survive stress testing, and run reliably in controlled environments. Once deployed in the field, though, they occasionally stop responding with no diagnostic trace, no crash logs, no core dumps, no kernel panic. The device simply appears frozen.

In real-world deployments, IoT products, medical wearables, automotive controllers, robotics, industrial gateways, most failures aren't crashes. They're livelock conditions: the CPU stays active, interrupts still fire, tasks keep running, yet the system stops making forward progress. Traditional watchdogs miss these conditions entirely, which leads to silent failures that often go unnoticed until users report problems.

When watchdogs provide a false sense of security

Watchdog timers get treated as a simple reliability solution: hardware watchdog enabled, yes; kick performed in the main loop, yes; deployment approved. That mindset creates a dangerous assumption. A watchdog only detects when code stops executing. Most field failures happen when code keeps executing but is trapped in a non-productive loop, the defining trait of a livelock.

while(network_retrying()) {
    retry_send();
    kick_watchdog();
}

From the system's perspective, the CPU is running, interrupts are active, and the watchdog is being serviced. From the user's perspective, the device is unresponsive. The watchdog reports a healthy system while the user observes a failure.

Why livelock matters

Livelocks lead to missed scheduled operations, frozen interfaces, loss of communication, stalled sensor acquisition, repeated retries that drain the battery, and blocked control loops. In consumer or safety-critical products, that translates into regulatory non-compliance, customer dissatisfaction, increased warranty claims, product returns, and long-term brand damage. At production scale, these silent failures get expensive fast.

How livelocks occur in real systems

The most common causes, based on extensive experience in embedded design: infinite retry logic, where a modem or radio becoming unresponsive traps the system in an endless loop. Mutex or lock contention with missing timeout logic, trapping a task indefinitely while it waits on a semaphore. Interrupt flooding, from noisy GPIO inputs, misconfigured SPI interrupt lines, CAN bus flooding, or wireless driver interrupt storms, starving the main loop. Busy waiting, polling without event signaling, wasting cycles in non-productive execution. And dead peripherals, frozen I2C slaves, misclocked SPI devices, unresponsive modems, causing software to wait forever for a state change that never comes.

Why traditional watchdogs fail

Traditional watchdogs detect halted execution and total software hangs. They don't detect infinite loops, repeated retries, stalled state machines, blocked queues, or starvation conditions. The system appears active and keeps resetting the watchdog, which prevents any corrective action from ever triggering.

Solution: progress-based watchdogs

The core principle: only kick the watchdog when measurable progress has actually happened. That means new data acquired, successful communication, a state transition, a processed queue element, or a completed control-loop iteration.

void watchdog_task() {
    if(progress_counter == last_counter) {
        reset_system();
    }
    last_counter = progress_counter;
    hw_wdt_kick();
}

This approach makes the watchdog monitor forward motion instead of raw CPU activity.

A production failure and how we fixed it

Hoomanely builds modular IoT pet-care systems where timely, reliable operation matters. One deployed EverBowl experienced intermittent connectivity from bad network conditions. During modem firmware freezes, the networking stack entered a loop resembling connect, fail, retry, fail, retry. The watchdog kept getting serviced through each retry, and the device stayed unresponsive for hours, missing feeding schedules.

The fix added progress counters for successful data transmission, retry limits with backoff, modem hard reset triggers, and progress-aware watchdog supervision. After deployment, the failure rate dropped significantly, no missed feeding cycles were reported, and watchdog resets fell sharply.

Modern recovery pipelines

A robust recovery system needs to detect livelock conditions, attempt automated recovery, and escalate in controlled stages: detect the stall, restart the affected subsystem, reset the relevant hardware peripheral, power-cycle the module if needed, reboot the system if that fails, fall back to safe mode, and log the failure for post-mortem analysis.

if(no_network_progress) {
    restart_network_stack();
    if(still_no_progress) {
        power_cycle_modem();
        if(still_no_progress) {
            system_reboot();
        }
    }
}

This structured recovery avoids unnecessary full resets and maximizes uptime.

Implementation errors to avoid

Kicking the watchdog in every loop iteration regardless of progress. Missing timeouts on semaphores or locks. Infinite retry loops with no ceiling. Ignoring hardware reset controls. Assuming peripherals self-recover. Servicing watchdogs inside interrupt handlers. And relying solely on hardware watchdogs without any software-level progress tracking. These patterns almost guarantee field failures eventually.

A blueprint for production-grade watchdog strategy

Detect progress by tracking counters or state transitions. Limit retries with exponential backoff and a maximum attempt count. Reset peripherals by power-cycling or reinitializing modules when required. Escalate recovery from soft resets up to full system recovery when needed. And log failures so post-mortem analysis is actually possible.

Key takeaways

Most embedded failures come from livelocks, not crashes. Traditional watchdogs can't detect livelocks on their own. Progress-aware watchdogs meaningfully improve reliability. Structured recovery pipelines increase system resilience. And reliability comes from design discipline, not luck. A missing timeout or a poorly placed watchdog kick can lead to widespread product failures, so investing in robust detection and recovery strategies protects both customers and engineering teams.