Synchronizing Time Across Multiple SoMs: Building a Distributed Clock System

Picture debugging a critical production issue where the logs show a sensor event was processed before it was captured. Impossible? No, just unsynchronized clocks. That was our reality building distributed embedded systems, since our platforms rely on multiple Systems-on-Module working in concert, each handling different sensors and processing tasks. Without synchronized time, we were essentially flying blind.

Why multiple SoMs

Modern embedded systems increasingly use multiple SoMs rather than one monolithic computer. Our EverHub for edge computing and EverBowl for intelligent multi-sensor processing both lean on this. Dedicating separate SoMs to different high-bandwidth sensors, cameras, thermal imaging, audio, prevents resource contention. Distributing computational load across physical modules avoids the thermal throttling that plagues single-board solutions under sustained load. If one SoM hits an issue or needs to reboot, the others keep operating, so critical communication stays available even if sensor processing temporarily fails. Each SoM can specialize, some for low-latency sensor capture, others for compute-intensive processing, one for coordination and connectivity. Scaling to more sensors or algorithms just means adding or upgrading the relevant SoM. And you get true hardware parallelism, multiple sensors captured and processed simultaneously on different physical modules, not just different threads.

But this architecture introduces a fundamental challenge: time synchronization.

When every SoM lives in its own time

In a multi-SoM system, events happen across modules constantly, and without synchronized clocks you simply can't know which event actually happened first, since each SoM's clock drifts independently. Without synchronized time, you can't correlate multi-sensor data across modules, can't calculate true pipeline latencies, can't reconstruct event sequences during debugging (logs from different SoMs with inconsistent timestamps can show effects before causes), can't perform accurate sensor fusion (temporal misalignment corrupts fusion algorithms), and can't meet hard timing requirements that real-time, safety-critical, or auditable systems demand.

Clock drift is inevitable

Every SoM's crystal oscillator is an imperfect physical device, not a perfect timekeeper. Environmental factors drive drift: temperature changes the crystal's oscillation frequency, supply voltage fluctuations shift frequency, mechanical stress and vibration affect resonance, and electromagnetic interference can modulate the oscillator. Manufacturing variability adds more: every crystal has inherent tolerance from its nominal spec, frequency drifts with age over months and years, and no two crystals behave identically even from the same batch.

Even with high-quality crystals, two SoMs powered on at the same instant will show different clock readings within minutes, and the gap keeps growing. A SoM running heavy computation and warmer will drift at a different rate than one running lighter loads and staying cooler. One-time calibration doesn't work either, because drift accumulates continuously, temperature changes during operation shift frequency, and a calibration done at room-temperature startup becomes invalid within minutes once the system heats up. You need continuous, active synchronization.

A master clock and synchronization protocol

Our solution designates one SoM as the master clock, the time authority, and implements a protocol that continuously corrects the others. The master should have external connectivity (useful for syncing to NTP or GPS, though optional), a stable thermal environment (lower processing load means less thermal variation), reliable power, and a central coordination role. All other SoMs become time clients, periodically syncing their local clocks against the master over whatever bus connects them, CAN, Ethernet, SPI, or anything else, with the algorithm adapting to the medium's characteristics.

Measuring and correcting clock offset

The synchronization process has to answer one question: how different is my clock from the master's? That's trickier than it sounds because the act of asking takes time.

Using Cristian's Algorithm: the client sends a sync request, recording local time t0. The master receives it and timestamps it t1 on its own clock. The master replies with t1. The client receives the reply, recording local time t3. From there:

Round-Trip Time (RTT) = t3 - t0
Estimated One-Way Delay = RTT / 2
Clock Offset = t1 - (t0 + RTT/2)
Error Bound = RTT / 2

Assuming the message takes roughly the same time in both directions, the difference between what the master's clock said and what the client's clock should have said reveals the offset.

Not every measurement is equally trustworthy, though. Communication delay varies from message queuing, CPU scheduling, interrupt latency, and bus contention, and a sync request delayed 5ms in a queue throws off the offset calculation by roughly half that. The fix is weighting measurements by quality: track the minimum observed RTT as the best-case delay, trust new measurements close to that minimum, and discard or downweight ones with suspiciously high RTT that likely hit queuing or contention.

Simple offset correction isn't enough either, since clock drift has two independent components: phase error (the current offset) and frequency error (the rate of drift). Correct only phase and the underlying frequency error means the clocks drift apart again within seconds. Correct only frequency and the initial offset never goes away. You need both, corrected independently, using a two-stage phase-locked loop. Frequency correction runs slow, over many seconds, adjusting a virtual clock's tick rate based on observed long-term drift, filtering out short-term noise. Phase correction runs fast, every second or few, making fine adjustments on top of that stable frequency base. Because they operate on different time scales and different aspects of the clock, they don't fight each other, and the system converges quickly and stably.

Tracking accuracy over time

Between sync measurements, error grows as time passes:

Current_Error = Last_Measurement_Error + (Time_Since_Last_Sync × Residual_Uncertainty)

If the last sync had an RTT of 1000 microseconds (giving 500us of measurement error), and residual frequency uncertainty is 10 parts per million (10us per second), then 5 seconds after sync, current error is 500 + 50 = 550 microseconds.

If current error exceeds your accuracy requirement, the system enters an out-of-sync state: it stops providing timestamps, logs a warning, attempts more frequent synchronization, and resumes normal operation once error drops back below threshold. Rather than delivering timestamps with unknown accuracy, the system fails explicitly and loudly, since silent failures with bad timestamps corrupt data in ways that are much harder to debug.

The transform function approach

We deliberately don't adjust the operating system's clock. Instead we maintain a mathematical transform function converting local monotonic time into synchronized time. Adjusting the system clock typically needs elevated privileges, conflicts with other time services like NTP daemons, can break monotonic-time assumptions in databases and loggers if adjusted backward, and locks you into one shared time reference across all processes.

The transform function keeps a phase offset (a constant to add) and a frequency multiplier (a scaling factor for elapsed time). To get the current synchronized timestamp: read the local monotonic clock, calculate elapsed time since a reference point, scale that elapsed time by the frequency multiplier, add the phase offset, and return the result. This works as an unprivileged process, doesn't interfere with the system clock or other time services, and stays easy to test and validate. As sync measurements arrive, the frequency multiplier and phase offset update through the two-stage correction process described above, with gains tuned to balance rapid convergence against stability.

Practical considerations

Thread safety matters since multiple application threads might request timestamps while a background thread updates transform parameters, so mutexes protect reads and writes, with a fast atomic check on sync status before the slower locked read. For API design, a simple get_timestamp() and is_synchronized() cover most needs, while get_timestamp_with_error() gives applications with strict requirements the error bound so they can decide whether current accuracy is good enough.

Each SoM needs configuration specifying its role: masters decide whether to sync to an external time source, clients specify the master's address, sync interval, maximum acceptable error, and tuning gains. Adaptive sync intervals help too, frequent syncs during initial lock acquisition, gradually lengthening once stable, tightening back up if error starts growing.

The master SoM itself runs a simple service: read its current time, send it back to the requesting client, done, no per-client state needed. It can optionally sync to an external absolute source like NTP or GPS for absolute UTC time, though that's not required for relative synchronization between SoMs.

Performance expectations

Achievable accuracy depends on communication latency and jitter (consistent delay matters more than low delay), reliability (dropped messages degrade sync frequency), thermal stability (transients force more frequent syncing), processing load (heavy load delays message handling), and clock quality (better crystals drift less). With well-designed systems, low-latency communication, moderate thermal stability, reasonable load, sub-millisecond synchronization is routinely achievable, often in the hundreds of microseconds. Under challenging conditions, accuracy degrades but typically stays within a few milliseconds, and critically, the system continuously measures and reports its own accuracy rather than silently delivering bad data.

Common pitfalls

Don't assume one-time calibration suffices, thermal drift accumulates within minutes. Don't ignore communication variability, weight or discard corrupted measurements. Don't silently deliver bad timestamps, fail explicitly instead. Don't forget thermal management, since clock drift is largely temperature-driven. And don't skip validation, subtle bugs hide easily in sync code that appears to work.

Conclusion

Synchronizing time across multiple SoMs is fundamental to building reliable distributed embedded systems. Our approach combines one SoM as master clock, Cristian's algorithm for measuring offset despite communication delay, a two-stage phase-locked loop for independently correcting frequency and phase, a transform function that avoids touching the system clock, continuous accuracy tracking with explicit error bounds, and graceful, loud degradation when requirements can't be met. With proper implementation, sub-millisecond synchronization is achievable across commodity hardware with no specialized timing equipment.

Don't assume your SoMs' clocks agree. They don't, and they never will without active synchronization. Build it in from the beginning, track accuracy explicitly, and fail loudly when precision degrades.