Signed IoT commands over MQTT: why the broker's ACL is not enough
The first time we moved a set of EverBowl smart bowls over to a new signing key, telemetry kept flowing as normal. Every command we sent, including a plain ping, came back with the same answer: rejected, signature. Nothing was broken. The bowls were doing exactly what we had designed them to do, which was to refuse any command that the key they trusted had not signed, even though it arrived over a mutually authenticated TLS session on a topic only our backend may publish to.
That incident is the clearest argument I have for the design in this post. The MQTT broker proves who delivered a message. It does not prove who wrote it. For a fleet that can reboot, restart services, and drive hardware in someone's kitchen, that gap matters.
The short version
- Every cloud command is serialized into one canonical form covering what to do, its arguments, when it expires, who asked, and which device it is for.
- The backend signs those bytes with an asymmetric key held in a managed KMS. The private key never leaves it. The device holds only the public half.
- The device inserts its own identity before verifying, so a command signed for one device fails on every other device.
- Expired commands are refused, and a short memory of recent request ids blocks replays.
- On the cloud side, an idempotency key maps to exactly one command, and exclusive commands hold a per-device slot through a conditional write.
What the topic ACL actually proves
The broker authenticates each device with a certificate and authorizes each publish and subscribe against a policy. Our devices can subscribe only to their own command topic, and only our backend may publish to it. That is real security and we rely on it.
But look at what it guarantees. A message on a device's command topic was accepted by the broker from some principal allowed to publish there. That set is wider than "our backend wrote this on purpose":
- Other services that share the broker. A service with a misapplied policy can publish into a topic it was never meant to touch.
- Humans with publish rights. Anyone with permission on the topic, including an engineer in a console, can send a command.
- Our own bugs. A retry loop that re-sends a reboot is indistinguishable from an intended reboot.
The device cannot tell these apart from the transport alone. It needs a proof that depends on something only our backend has, and that the message itself carries. That is a signature.
The same reasoning applied before MQTT. Our earlier polling transport already demanded a signature on every action, so when we moved commands onto MQTT (described in device commands with durable acks over MQTT) we kept the check and moved the executor behind the new transport, rather than trusting the new transport to replace it.
The canonical form
Signing JSON is a classic trap. Two serializers that disagree on key order, whitespace, or escaping produce different bytes from the same object, and the signature fails for reasons nobody can see in a log. So both sides sign and verify an exact byte string produced by fixed, shared rules: deterministic key order, no optional whitespace, one encoding. The device rebuilds the same object from the message it received and serializes it the same way.
Each field in the signed body does a job:
| Field | What it defends against |
|---|---|
| Request id | Replay. The device remembers recent ids and refuses a repeat. |
| Command and arguments | Tampering. Change one argument and the bytes change. |
| Expiry | Delayed delivery. A command queued while the device was offline dies on its own. |
| Issuer | Audit. Who or what asked, carried into the device's own log. |
| Device identity | Cross-device misuse. See below. |
The signature covers the whole set, so there is no field an attacker can edit without the check failing.
Binding a command to one device
This is the part I like most. The backend knows which device it is addressing and includes that identity in the signed body. The device does not read the identity from the message when it verifies. It writes its own into the canonical form.
If the signed identity and the real identity match, the bytes match and the signature verifies. If someone captures a valid command for one device and republishes it to another, the second device reconstructs the body with its own identity, the bytes differ, and the signature fails. No per-device key, no list of trusted peers, just one field and one rule about where its value comes from.
We proved each guard on the bench before the first live round trip: a command for another device, tampered arguments, an expired command, an unsigned command, and a command outside the allow list were each rejected with their own reason, and a repeated command was accepted once and refused as a replay.
Why a managed KMS and not a key in the environment
The simplest implementation is a private key in the backend's secrets store, loaded at boot. It works, and it is how many fleets start. We chose an asymmetric key in a managed KMS instead, for three reasons.
The private key is never in process memory. The backend sends the bytes to the KMS and gets a signature back. Nobody can dump the key from a container, a core file, or a log.
Access is an IAM statement. Only the backend's signing role may use the key, and that permission is something we can read, review and audit.
The public half is a simple export. The device verifies with standard tooling it already ships. It needs no cloud SDK and no credentials to verify.
The cost is one KMS call per command. Commands are human-paced events, so this is noise. We would not sign telemetry this way, and we do not need to, because telemetry flows the other direction and is bound to its owner on the way in, as described in DynamoDB multi-tenant isolation with LeadingKeys.
Here is how the options compare:
| Approach | Private key exposure | Scoped to one device | Rotation cost | Stops a leaked broker credential |
|---|---|---|---|---|
| Topic ACL only | None (no key) | Yes, by topic | None | No |
| Shared HMAC secret | On every device and in the backend | Only with per-device secrets | Touch every device | Only if the secret is also safe |
| Asymmetric key in backend secrets | In process memory | Yes, via signed identity | Update the public key on devices | Yes |
| Asymmetric key in a managed KMS | Never leaves the KMS | Yes, via signed identity | Update the public key on devices | Yes |
Expiry, replay, and the executor we already trusted
On the device, a small module receives each command from the MQTT client and hands it to the same executor the old transport used. That executor already had the allow list, the signature check, the expiry check, the replay memory, and an audit log of every decision. Writing a second executor would have meant two allow lists that drift apart. So the executor became a library and the MQTT client became its caller.
The allow list is short on purpose: health checks, diagnostics, log collection, a few restarts. Nothing that opens a shell or wipes a device. Commands that end the device's session acknowledge first so the cloud sees the answer, and every command carries its own timeout sized to the work it does.
Idempotent on the way out
The device side stops bad commands. The cloud side has to stop sending duplicates in the first place, because an operator clicking twice and a worker retrying after a timeout both look like two legitimate requests.
Two guards handle it. First, a caller may pass an idempotency key, which the service claims for a fresh command with a put-if-absent write. A second request with the same key gets the command the first one created. Second, exclusive commands claim a slot named for the device and the command, held until the command's deadline. A second one while the first can still run is refused as busy, not queued.
Command state only moves forward: received, progress, then ok or error. A late progress message cannot reopen a finished command. This is the same forward-only, conditional-write discipline we use for exactly-once events on DynamoDB.
The key switch, and what the rejections told us
Back to the key switch. Moving each device over was a scripted sequence: update its registration, attach its policy, repoint its routing, restart its client, then confirm telemetry on the new path and silence on the old one. Telemetry passed on every device.
Then a ping: rejected, signature. The backend was signing with the new key. The device was still verifying with the old public key. Nothing in the transport, the policies, or the routing could have caught that, because none of them know anything about the signing key.
The fix became a permanent runbook step: install the new public key on the device before the backend switches, keep the previous key for rollback, and restart. The next ping answered pong, and every device after that got the key before its first command.
The lesson is not "we forgot a step". It is that the device refused in the safe direction with a reason we could read, and a key mismatch that no part of the transport could see was caught by design rather than by luck. If we had trusted the topic ACL alone, the first ping would have succeeded and we would have learned nothing.
How EverBowl™ uses this
EverBowl™ is the smart bowl at the other end of these commands. Most of the time it needs none of them. It records how a dog approaches, eats, and leaves, and the cloud learns the pattern from the record. Commands exist for the moments a human needs the bowl to do something now, such as running a diagnostic after a reported fault or collecting logs for a technician.
Those are also the moments when a mistaken or malicious command does the most harm, which is why every one of them is signed, scoped to one device, and allowed to run once. EverBowl does not diagnose; the value is the record, and the signing layer is there so the record cannot be disturbed by anyone except the people responsible for it.
Conclusion
Mutual TLS and topic policies answer the question "who delivered this?" A device that can act on a command needs the answer to a different question: "did my backend write this, for me, recently, and only once?" A canonical body, a KMS-held signature, the device's own identity folded into the signed bytes, an expiry, and a short replay memory answer it with very little code on each side. The reward arrived on the day we switched keys, as a wall of signature rejections, which is exactly the failure you want.
See the rest of our backend posts in the software hub, or learn about the bowl itself at hoomanely.com.
FAQs
Doesn't mutual TLS already authenticate the command?
It authenticates the connection between the broker and each party. The device learns that the broker sent the bytes, and the broker learns which principal published them. The device never sees the publisher's identity, and the broker cannot know the publisher's intent. A signature travels with the message and survives any number of hops.
Why let the KMS decide the algorithm?
Keeping the private key inside the KMS mattered more than any particular curve. We picked an algorithm the KMS supports and the device can already verify with its existing tooling, rather than one that would have forced the key out of the KMS.
What happens when the signing key rotates?
Create a new key, distribute its public half to devices, then switch the backend to sign with it. Devices that have not yet received the new key reject commands with a clear reason until they do, which is the safe direction.
Why does the device insert its own identity instead of checking the one in the message?
Checking a field is a comparison someone could forget or get wrong. Folding the device's real identity into the signed bytes makes a cross-device command fail cryptographically rather than logically, with no extra code path to test.
How large should the replay window be?
Large enough to cover every command that could arrive within the expiry window, with margin. The expiry does most of the work; the replay memory covers the gap between issue and expiry.
Does signing slow commands down?
One KMS call is small next to a command that then crosses the internet to a device on home Wi-Fi. For human-paced commands it does not register.
