Edge Compression of Telemetry: Delta, Varint, LZ4 & MessagePack
A technical guide to shrinking IoT telemetry without breaking edge systems or cloud pipelines.
IoT and embedded systems generate telemetry that overwhelms traditional networks. Sensor readings, actuator states, health logs, diagnostics, and event traces strain wireless bandwidth, burn through MCU flash from local buffering, and stress cloud ingestion pipelines. At Hoomanely, telemetry is the backbone of our distributed pet-care IoT fleet, device health, feeding analytics, motor performance, battery behavior, and environmental context all depend on continuous data flow.
Raw telemetry is expensive to transmit and store. We built a compression strategy lightweight enough for MCUs, scalable enough for gateways, and simple enough for cloud recovery pipelines. This post breaks down four techniques, delta encoding, varint, MessagePack, and LZ4, how they complement each other, where each should run, and why they form a reliable, maintainable telemetry pipeline.
Why telemetry compression matters
Telemetry from IoT devices is usually predictable (temperature changes gradually), structured (key-value records, protobufs, TLV formats), repetitive (the same fields across messages), and numeric-heavy (integers, floats, timestamps). That makes it ideal for compression, but embedded systems come with strict limits: MCU RAM measured in kilobytes, limited flash for program and buffer space, CPU budgets shared with real-time tasks, and power constraints on battery-driven nodes. That rules out heavyweight codecs like gzip, zstd, or brotli, and points toward a layered, edge-suitable approach that's predictable, low-footprint, streaming-friendly, and deterministic.
Delta encoding: the first layer
Delta encoding replaces absolute values with the difference between consecutive samples. Instead of transmitting 24.2°C, 24.3°C, 24.4°C, you transmit 24.2°C (baseline), +0.1°C, +0.1°C.
Most telemetry evolves slowly: temperature moves gradually, battery voltage changes in small increments, motor current varies within predictable windows, and timestamps increase monotonically. Deltas fall into smaller numeric ranges, which compresses more efficiently in later layers. The cost is very low, simple subtraction, works with integers and floats, needs only the previous value as state, is fully lossless, and needs no dynamic memory allocation:
typedef struct {
float last_value;
bool initialized;
} DeltaEncoder;
float encode_delta(DeltaEncoder* enc, float current) {
if (!enc->initialized) {
enc->initialized = true;
enc->last_value = current;
return current;
}
float delta = current - enc->last_value;
enc->last_value = current;
return delta;
}The catch: delta encoding requires correct ordering. If packets arrive out of order, reconstruction fails, so you need sequence numbers in packet headers, delta encoding only over reliable transport layers, or monotonic timestamps as implicit sequence markers.
Varint encoding: squeezing numbers further
After delta encoding, you often end up with small integers, especially with quantized floats or integer sensors like ADC values and counters. Varint, variable-length integer encoding, packs small values into fewer bytes, using the high bit of each byte as a continuation flag: if bit 7 is set, more bytes follow, and if it's clear, this is the final byte, with the lower 7 bits carrying data.
size_t encode_varint(uint32_t value, uint8_t* buffer) {
size_t index = 0;
while (value >= 0x80) {
buffer[index++] = (value & 0x7F) | 0x80;
value >>= 7;
}
buffer[index++] = value & 0x7F;
return index;
}For values 0 to 127, varint uses 1 byte against 4 for a standard int32. For 128 to 16,383, it uses 2 bytes. For 16,384 to 2,097,151, it uses 3 bytes. Combined with delta-encoded telemetry, where most values are small, that's significant savings. It's also branch-light, needs no heap allocation, is streaming-friendly, and pairs naturally with protobuf or MessagePack formats.
MessagePack: binary serialization with built-in efficiency
Before applying delta, varint, or LZ4, you need to serialize structured data, and the format you choose fundamentally affects final payload size and processing efficiency. MessagePack is a binary serialization format significantly more compact than JSON while keeping similar expressiveness and schema flexibility.
A JSON payload like {"temp": 24.2, "humidity": 65, "battery": 3.7, "timestamp": 1732550400} runs about 70 bytes. The MessagePack equivalent runs about 35 bytes, roughly a 50 percent reduction before any additional compression, since integers get encoded as integers rather than ASCII digits, common types get single-byte tags, it stays self-describing like JSON, and encoding and decoding skip text parsing entirely.
MessagePack should be the first serialization step, applied before delta and varint, because it strips structural overhead like field names and delimiters, normalizes numeric types so integer versus float distinctions survive, and creates a denser input for downstream compression to work on. Against protobuf, MessagePack needs no schema and iterates faster, at the cost of slightly less density, a fine trade-off for telemetry where schema evolution is frequent and on-device debugging matters.
LZ4: dictionary-based compression for gateways
LZ4 is a high-speed, lossless compression algorithm optimized for cases where decompression speed matters as much as ratio. It isn't suitable for tiny MCUs, but it excels on edge gateways with ARM Cortex-A CPUs, Linux-class hubs like Raspberry Pi or industrial computers, and SoMs and modules like CM4, AM62, or the i.MX series.
LZ4 scans input for repeated sequences and encodes repetitions as offset-length pairs against a sliding window of recent data, using fast heuristics rather than exhaustive search. It shines on structured, repetitive data: repeated field names even inside MessagePack binary, similar message structures across time intervals, batch processing of multiple messages together, and delta-encoded values clustering around zero. That makes it the natural final compression layer before network transmission or cloud storage. It's extremely fast on GHz-class ARM cores, has low memory overhead since its dictionary fits in L1 or L2 cache, streams well for continuous log-like data, produces deterministic output, and its frame format includes checksums for integrity verification.
Don't run LZ4 directly on MCUs when the CPU runs below 100 MHz, RAM is under 64 KB, real-time guarantees are critical, or you're compressing single messages with a poor ratio. Use it at the gateway layer when batching multiple messages, when CPU cycles are available, when network bandwidth is constrained, or when cloud ingestion costs matter.
The complete pipeline
A production-ready telemetry pipeline layers all four techniques in order, and that order isn't arbitrary, each stage prepares data for the next. MessagePack goes first, eliminating ASCII overhead and creating binary structure. Delta comes after MessagePack, operating on compact binary representations and reducing numeric magnitude. Varint comes after delta, compressing the now-small delta values efficiently while keeping MCU CPU cost minimal. LZ4 runs last, at the gateway, exploiting repetition across entire batched messages.
Measured on typical IoT telemetry, temperature, humidity, battery, timestamps, state flags, MessagePack alone roughly halves the JSON baseline, delta adds another 2 to 3x, varint adds another 1.5 to 2x on top of that, and batched LZ4 adds another 2 to 4x. Combined, that's a 12 to 48x reduction versus raw JSON. Beyond size, this layering gives fault isolation, since corruption in one layer doesn't cascade, incremental optimization since each layer tunes independently, clear interfaces for testing, the ability to decode up to any stage for inspection, and known worst-case behavior at every stage.
Failure modes and recovery design
Compression adds complexity, and without careful design a single corrupt byte can break reconstruction of an entire stream. Without block boundaries, corruption in one message propagates forward because delta encoding maintains state, varint parsing depends on continuation bits, and MessagePack structure assumes valid type tags. The fix is a block-based architecture: divide the stream into self-contained, independently decodable blocks, so corruption in one block only loses that block.
Each block carries a header with a magic number for boundary detection and resync after corruption, a version field for pipeline evolution, flags indicating which compression layers are active, payload length for skipping corrupted blocks, a monotonically increasing sequence number for detecting gaps, a timestamp for temporal ordering, and a CRC32 for corruption detection.
Delta state management across blocks needs its own handling, since if a block is lost you need to reconstruct subsequent deltas. The combination that works best: periodic baseline resets, sending an absolute value every N samples, plus a baseline value included in the block header itself, giving immediate recovery with small overhead per block.
The design principle underneath all of this: error isolation matters more than maximum compression ratio. A pipeline that achieves 50x compression but fails catastrophically on single-bit errors is worse than one achieving 30x with graceful degradation. Keep blocks small, generally sub-1KB, always include integrity checks, design for missing data and not just corrupted data, and test recovery logic as thoroughly as compression logic.
Why this matters at Hoomanely
Our devices operate in challenging environments, intermittent connectivity, power constraints since they're battery-operated and need to last weeks, real-time requirements, deployment across homes, clinics, and shelters with different networks, and fleet scale with thousands of devices generating continuous telemetry. That telemetry covers device health (MCU temperature, voltage rails, current draw, flash wear, RAM usage, signal strength), behavioral analytics (feeding patterns, activity levels, environmental context), predictive maintenance (battery discharge curves, failure precursors), and operational intelligence (firmware version distribution, feature usage, performance benchmarking).
A well-engineered compression pipeline pays off economically through reduced cloud storage and network egress costs, technically through better battery life, less flash wear, and faster real-time performance, and operationally through faster analytics queries and easier troubleshooting.
Key takeaways
MessagePack replaces JSON with compact binary serialization, apply it first to strip text overhead and preserve type information. Delta encoding shrinks numeric variation, ideal for time-series telemetry with slowly changing values. Varint compresses integers efficiently without heavy CPU cost. LZ4 gives dictionary-based compression, best deployed at gateways with CPU and memory to spare. Layered compression outperforms any single technique, since each stage prepares data for the next. Recovery-friendly block design prevents cascading failures, self-contained blocks with headers, checksums, and sequence numbers enable graceful degradation. Design for failure isolation, not maximum compression, a slightly lower ratio with robust error handling beats maximum compression that fails catastrophically. And apply techniques at the right system tier: MCUs handle MessagePack, delta, and varint, gateways add LZ4 batching, cloud focuses on efficient decompression.
