Device Commands with Durable Acks over MQTT

When a backend calls another service, the answer comes back on the same connection. When it tells a connected device to do something over MQTT, three separate things have to happen: the message has to reach the broker, the device has to be awake to hear it, and the device has to publish an answer on a different topic that comes back through a different pipe, maybe twice, maybe late, maybe never.

We recently built the first version of this command path for our connected devices. This post is about three design choices from that work that generalise to any cloud-to-device command path: how to handle timeouts without a timer, how a single conditional write makes redelivered messages harmless, and one thing about managed IoT brokers that surprised us.

A few constraints shaped everything. The backend runs as several identical, horizontally scaled instances with no instance-local state allowed. There is one scheduler in the system and it is not built for 30-second deadlines. Every write is tenant-scoped. And tests run against real local versions of the database and cloud services rather than mocks.

The shape of a command

A command is a single record, keyed by device and a time-ordered id, living in the tenant of whoever owns the device. It carries a status, a deadline, and the device's last answer.

The path out is boring on purpose. An admin issues a command, the backend writes the record as pending together with a "command sent" event in one transaction, then publishes the encoded message to the device's command topic at QoS 1 using the broker's stateless publish API. We deliberately did not run an MQTT client inside the backend. A long-lived connection with a unique client id per instance is exactly the kind of per-instance state the architecture forbids, and for publish-only traffic a stateless call buys the same delivery.

The path back is where the interesting part lives. The device publishes a response to an ack topic. A broker rule forwards it to a queue, and a worker consumes it and applies the ack to the record. Connectivity rides the same pipe: the broker's own connect and disconnect lifecycle events go through a second rule into the same queue.

Statuses only move forward: pending → received → ok | error. The device reports received when the command lands and ok or error when it finishes. Two more statuses never come from the device: not_sent when the publish itself failed, and timed_out, which is the subject of the next section, because it is never written at all.

Lesson 1: derive the timeout on read, never run a timer

The obvious design for a 30-second deadline is a timer: schedule something that fires at the deadline and flips the record to timed_out. We did not build that, and the reason is more than taste.

A timer is state. On a horizontally scaled service it has to live somewhere shared, be leader-gated so two instances do not both fire it, and survive a redeploy. Scheduling a durable job for every short-lived command would be heavy, and adding a second in-process timer loop was something our own guardrails ruled out. That forced a better question: do we need to store timed_out at all?

We do not. The record already carries the deadline, written when the command was sent. The reader computes the status:

def effective_status(stored, deadline, now):
    if stored in {PENDING, RECEIVED} and now > deadline:
        return TIMED_OUT
    return stored

That is the whole timeout implementation. No worker, no lease, no retry. And it has a property a timer cannot give you for free: the timeout is not terminal. A device that was asleep and answers late still moves the record from pending to received, because nothing ever wrote timed_out over it. A timer-based design has to decide what to do with a late ack against a terminal record; usually it drops it, and now the record says the command failed when the device actually did the thing.

Major cloud IoT platforms make the same choice for cloud-side command timeouts, which was reassuring to find after the fact. The cost is that anything that wants to react to a timeout, such as an offline alert, has to poll or be triggered by something else. Nothing reacts to timeouts yet, so we have paid nothing for a problem we do not have.

Lesson 2: a conditional write is your idempotency primitive

Everything on the return path is at-least-once. MQTT QoS 1 can duplicate. The rule-to-queue hop can duplicate. The worker can crash after writing and before acknowledging the message. And the device itself can retry an ack. So the handler will see the same received twice, and sometimes ok before received.

The tempting fix is a dedupe table, or a check-then-write in the handler. Both have a race between the read and the write across two workers. What we do instead is make the state transition itself the guard. Each ack maps to a small table of allowed predecessors:

ALLOWED_FROM = {
    RECEIVED: {PENDING},
    OK:       {PENDING, RECEIVED},
    ERROR:    {PENDING, RECEIVED},
}

The handler issues one update whose condition says "the stored status is one of these predecessors". If the condition holds, the status advances. If it does not, the database refuses the write, and that refusal is the idempotency signal.

What made this clean was writing domain events in the same transaction as the state change. When the conditional update fails, the whole transaction fails, and the event is never appended. The handler treats that refusal as a normal outcome and returns. The result is exactly what you want from a duplicate: no state change, and crucially no second "command acked" event for subscribers to double-process.

try:
    emit("command_acked", payload, with_writes=[guarded_update])
except WriteRefused:
    return  # duplicate or out of order: nothing happened, nothing emitted

Connectivity uses the same primitive with a different guard. Broker lifecycle events can arrive duplicated and out of order, and the documentation says so. Each event carries enough information to order it: a timestamp and a per-connection version. We build a sortable key from those, with disconnect ordering after connect for the same version, store it on the connectivity record, and the condition is simply "incoming key is greater than stored key". A stale disconnect that arrives after a newer connect is refused by the database, not by application logic that could race.

One write, one condition, and the database is the arbiter. We test this against a real local database instance, because the whole point is the database's behaviour, which a mock would have to pretend to know.

Lesson 3: broker lifecycle events are account-wide

This was the surprise. We run more than one environment side by side in the same cloud account, separated by naming prefixes on every resource. We assumed a broker rule created for one environment would only see that environment's devices.

It does not. Lifecycle events are published for every client in the account, and a plain topic subscription matches any device publishing on that topic, whichever environment it belongs to. The first time one environment's presence rule ran, its queue received connect and disconnect events for devices it had never heard of. Only rule-scoped ingest topics are limited to the rule you name.

We considered three fixes:

1. Separate accounts per environment. Correct, and a much larger change than this feature.
2. Move acks to rule-scoped ingest topics. Fixes the ack path but needs a firmware change, and does nothing for lifecycle events, which have no scoped form.
3. Resolve every message through the environment's own ownership records and drop anything it does not hold.

We chose the third. The transport layer never trusts a device's claimed identity. For every ack and every lifecycle event it asks "who owns this device, here?" against this environment's own records. No owner means the message is logged and dropped, and nothing is written. The same rule protects against the more serious case: a misrouted message from one tenant's device landing in another tenant's record.

The lesson underneath is that the ownership lookup was already required for tenant isolation. The account-wide fan-out just made it load-bearing for environment isolation too. Two problems, one check, which is usually a sign the check is in the right place. Moving acks onto scoped topics is still on the backlog, because it also cuts down the dropped-message noise.

What we would keep, and what is still open

Keep: the forward-only state table, the conditional write as the only dedupe mechanism, and the timeout as a function of the record rather than a stored fact. Together they make the handler a pure function plus one guarded write, which means the state rules can live in a module with no I/O and be unit-tested directly.

Keep also: publishing from the backend through a stateless API instead of holding an MQTT connection per instance. It removed a whole class of per-instance state and a certificate-management story along with it.

Still open:

• Devices connect with a clean session, so a command sent while a device is offline is lost and will read as timed out. Persistent sessions need a firmware change.
• There is no disconnect debounce yet, which is recommended before acting on a disconnect. Nothing acts on disconnects today, so it has not mattered.
• The full loop still needs to be confirmed end to end on real hardware, including the device's ack and a complete connect and disconnect cycle.

If you are building a command path to devices, the short version is: let the database decide what counts as a duplicate, let the reader decide what counts as late, and never trust the broker to tell you which environment a message belongs to.