Auto-Recovery Pipelines

Networks misbehave in the real world. BLE links jitter, Wi-Fi bursts collide, buffers overflow, and low-power sensors occasionally flip a bit. The worst part isn't the packet loss, it's the visible glitch your user sees: a frozen chart, a jumpy graph, or a dropped audio blip. This post shows how to build an auto-recovery pipeline that quietly detects corruption, drops the bad data, resynchronizes in milliseconds, and conceals the failure so the experience stays smooth.

Problem

Corruption happens from RF noise, partial frames, buffer overruns, or misordered delivery. Late and duplicate packets arrive during roaming and power transitions. Naive retries create visible stalls and battery drain. UI and ML pipelines magnify tiny defects into obvious artifacts, spikes in time-series, audible clicks, stuck values. Background constraints like sleep states and throttled timers make "retry until success" unreliable and expensive. The challenge: guarantee smoothness without heavyweight handshakes or wasting energy, especially on mobile plus embedded links, while keeping downstream analytics trustworthy.

Approach

Design the runtime around four fast, local decisions. Detect: per-packet integrity via CRC, length, and schema checks, monotonic sequence checks, and timing sanity. Decide: classify as good, late, duplicate, corrupt, or missing. Conceal: for gaps, apply play-out smoothing, interpolation, hold-last, or PLC tuned to the modality. Re-sync: use selective retransmission (NACK/SACK) or lightweight FEC to repair without stalling the UI. Wrap this with a tiny state machine, a jitter buffer, and exponential backoff timers. Push just enough intelligence to the edge to hide transient problems, keeping cloud and app aggregation idempotent.

Packet format that makes recovery cheap

A compact header makes wrong data obvious and repair cheap:

// Minimal header (6 bytes)
typedef struct __attribute__((packed)) {
  uint16_t seq;       // increments per packet
  uint8_t  flags;     // bit0:keyframe, bit1:parity, bit2:resync
  uint8_t  schema;    // data layout version
  uint16_t crc16;     // header+payload CRC
} hdr_t;

The seq field detects loss or out-of-order arrival quickly. Flags mark keyframes, parity or FEC blocks, or resync beacons. Schema rejects incompatible layouts immediately. CRC16 gives fast validation on MCU or SoC without heavy CPU. Put the header up front, payload fixed-length where possible, and align to DMA-friendly boundaries so CRC happens zero-copy.

Fast integrity and ordering

Do O(1) checks per packet before touching shared buffers:

bool good = crc16_ok(pkt) && len_ok(pkt) && schema_ok(pkt.schema);
if (!good) { stats.bad_crc++; classify_and_drop(pkt); return; }
 
int16_t d = (int16_t)(pkt.seq - expect_seq);  // handles wrap
if (d == 0)          accept_in_order(pkt);
else if (d > 0)      enqueue_gap_and_pkt(d, pkt);   // missing (d-1) packets
else                 handle_late_or_duplicate(pkt); // negative delta

Rules of thumb: accept up to N late packets, say 3, if re-sequencing is cheap; drop duplicates and count them without punishing the link; and trigger a fast resync on a huge sequence jump.

Jitter buffer that hides human-visible stalls

A small time-aligned buffer smooths bursts without hurting latency:

// Pseudocode
const int DEPTH_MS = 80;   // tune via P95 inter-arrival jitter
JitterBuffer jb(DEPTH_MS);
 
on_packet(pkt) {
  jb.push(pkt.ts, pkt.payload);
  while (jb.ready(now_ms())) {
    emit_to_consumer(jb.pop());
  }
}

Size it starting simple: depth_ms roughly equals p95_jitter_ms plus decode_budget_ms plus a 10ms safety margin. If you can't measure yet, begin with 60-100ms, then track P95 and shrink as transport stabilizes.

Concealment that fits the signal

Corruption creates gaps, fill them differently per modality. For scalar time-series like weight, temperature, or heart rate, use linear interpolation for a gap of one sample, slope-limited interpolation for 2-3 samples, and hold-last plus a degraded flag beyond that:

# linear interpolate one missing scalar sample
def conceal(prev_val, next_val):
    return prev_val + 0.5*(next_val - prev_val)

For vector IMU streams, clamp velocity, interpolate per axis, and use median-of-3 to kill spikes. For audio or continuous signals, use packet-loss concealment with short time-stretch or noise fill, avoiding audible clicks. For categorical events, use last-valid-wins with decay, emitting "unknown" if decay exceeds a threshold. The golden rule: concealments are annotated with a flag bit so downstream analytics can ignore or weight them less.

Resync without freezing the UI

Prefer selective repair that doesn't block playout. NACK or SACK asks only for what's missing within a small window. Parity FEC, an XOR across a short block, recovers one lost packet per block without a round-trip. Keyframes or resync beacons let you jump to now when repairing the past is too costly.

A tiny FEC XOR example, a block of 4 plus 1 parity:

// Sender: build parity for packets k..k+3
uint8_t parity[PAYLOAD] = {0};
for (int i = 0; i < 4; i++) xor_bytes(parity, pkt[k+i].payload, PAYLOAD);
send_packet(PARITY_FLAG, parity);
 
// Receiver: if exactly one missing in the block, reconstruct
if (missing_count == 1) {
  uint8_t rec[PAYLOAD] = {0};
  xor_bytes(rec, parity, PAYLOAD);
  for (each received r in block) xor_bytes(rec, r.payload, PAYLOAD);
  deliver(rec);
}

When to NACK versus FEC: short RTT with light loss favors NACK for lower overhead, bursty loss or higher RTT favors small-block FEC to avoid a UI stall.

Timers and backoff that don't kill battery

Immediate local actions, detect, decide, conceal, are synchronous and cheap. Network repair uses exponential backoff with jitter to avoid thundering herds:

// After sending a NACK and not hearing back
retry_ms = min(retry_ms * 2, 1000);
retry_ms = jitter(retry_ms, +/-0.2); // 20% jitter
schedule(retry_ms);

Guardrails: cap retries per window, auto-promote to RESYNC after K attempts, and never block playout.

Observability: prove you're hiding the pain

Expose counters and percentiles: packets_total, bad_crc, late, duplicate, gaps_filled, fec_repairs, nacks_sent, resyncs, along with p95_interarrival_ms, p99_jitter_ms, and stall_time_ms. Track experience SLOs like glitch-free minutes per session, time-to-steady after a link flap (P95), and concealment ratio, how often you're patching.

Idempotent aggregation in the cloud or app core

When the same sequence shows up twice due to retransmit or late arrival, upserts on stream_id plus seq keep storage clean. Derived roll-ups like per-minute stats should be computed on confirmed packets only, or on values flagged as not concealed unless explicitly allowed:

INSERT INTO samples(stream_id, seq, payload, concealed)
VALUES(:s, :q, :p, :c)
ON CONFLICT(stream_id, seq) DO UPDATE
SET payload=excluded.payload,
    concealed=LEAST(samples.concealed, excluded.concealed);

Latency budget and configuration knobs

Define a hard budget from wire to UI: total budget in milliseconds equals transport jitter P95 plus decode plus jitter buffer plus UI queue, targeting under 150ms for streams that need to feel live. Pragmatic defaults: jitter_buffer_depth_ms starting at 80 and shrinking toward 40 as the link stabilizes; late_accept_window of 3 packets; fec_block of 4+1 parity for bursty links, off for very low loss; resync_after triggered by max(3 NACK retries, 250ms gap); and concealment_max_span of 3 samples, else mark degraded.

recovery:
  jitter_ms: 80
  late_window: 3
  fec:
    enabled: true
    block: 4
  nack:
    max_retries: 3
    backoff_ms: [50, 100, 200, 400]
  resync:
    keyframe_interval_ms: 1000
  conceal:
    max_span: 3
    policy: scalar_linear|vector_median|audio_plc

Threads, buffers, and zero-copy

Use single-producer single-consumer ring buffers to avoid locks between ISR/DMA producers and user-space consumers. Pre-allocate packet nodes to avoid heap churn, dropping oldest under pressure. And do zero-copy CRC by pointing the CRC engine at DMA'd regions, avoiding memcpy just to validate:

volatile uint32_t head, tail;
Packet slots[RING_SIZE];
 
bool push(Packet p) {
  uint32_t h = head, n = (h+1) & (RING_SIZE-1);
  if (n == tail) return false; // full
  slots[h] = p;
  head = n;
  return true;
}

Transport-specific notes

After a disconnect, wait for kernel or stack cleanup before reconnect attempts, using short randomized delays to reduce collisions. Schedule repair work in permitted background windows, never letting repair timers block playout. Treat negotiated PHY or MTU parameters as volatile, caching them short-term but handling downgrade without assuming stability.

Chaos testing and validation

Build a local fault injector covering random bit flips and truncations, burst loss (dropping K of N consecutive packets), reorder windows, and latency jitter shaped to match field P95/P99. Acceptance criteria: stall time per minute under a threshold like 200ms, glitch-free minutes per session trending up versus baseline, concealment ratio staying within guardrails like under 8% of samples, and no unbounded growth in retry counter or jitter depth.

Backpressure and flow control

When consumers fall behind, prefer dropping the oldest unconsumed data rather than the newest, degrading gracefully. Signal upstream with a watermark, when buffer exceeds 70% request a lower rate or more keyframes. And protect the UI thread, never blocking the render path, emitting a degraded state instead.

Security and robustness notes

Treat schema as a gate, quarantining unknown schemas rather than parsing them. On CRC failure, drop without side effects, don't store. Keep sequence math wrap-safe and resistant to malicious jumps, promoting to RESYNC on absurd deltas. And log the first occurrence of each error type per session to reduce noise.

Results

Teams implementing this pattern commonly observe near-zero visible stalls for short burst losses of 3 packets or fewer, thanks to jitter buffers plus concealment. Fast recovery after roaming or RF spikes, since selective repair avoids full pipeline pauses. Lower energy use than fixed-interval retries, because backoff plus jitter prevents retry storms. And cleaner analytics, since downstream models can ignore flagged concealments, reducing false spikes.

At Hoomanely, our mission is turning raw pet signals into trustworthy, real-time wellness insights. Auto-recovery pipelines make the data boringly reliable, even when radios aren't. Whether it's activity streams, bowl weight changes, or ambient audio cues, this layer ensures our app and models see clean, continuous signals, so pet parents get confident guidance instead of jittery charts.

Key takeaways

Put detect, decide, conceal, resync between transport and consumers. Sequence plus CRC make bad data cheap to spot, jitter buffers make bursts invisible. Use NACK/SACK for quick holes, small-block FEC for bursty links or higher RTT. Conceal by modality, scalar, vector, audio, always flagging patched samples. Keep retries bounded with exponential backoff plus jitter, never blocking playout. And instrument experience SLOs, glitch-free minutes, stall time, time-to-steady, optimizing for what users feel.