From Reconnection to Recovery: Making MQTT Clients Truly Resilient

Few of our smart bowls went deaf for hours while every dashboard stayed
green. The cause was one assumption almost every MQTT client invites you to make.

They kept sending weight readings every few seconds. The fleet view showed them
online. The cloud could see every one of them. But nothing we sent the other
way arrived: no commands, and no go-ahead to upload a file it was holding. Each
bowl stayed like that for hours, until its process was restarted, and then it
was fine again.

Nothing crashed. Nothing logged anything that looked like an error. When we
found the cause, it was a single assumption that is very easy to make with
almost any MQTT client library: that when the library reconnects for you, you
are back where you were.

By default, you are not. This post covers why, why the symptoms pointed us the
wrong way, and how to build a client that can't fall into this gap.

THE PROBLEM: HALF A CONNECTION

MQTT has an asymmetry that is easy to forget.

• To PUBLISH, a client needs a connection. That's all.
• To RECEIVE, a client needs a connection AND subscriptions and the
  broker keeps those subscriptions as part of the client's SESSION.

When a client connects with a clean start (the usual default), the broker gives
it a brand-new, empty session. No subscriptions. Anything published to the
topics it used to listen on has nobody to go to, so the broker drops it. This
is correct behaviour, so the broker logs nothing about it. And no quality of
service level helps you , QoS controls how a message is delivered to a
subscriber, and there is no subscriber.

Now add a helpful feature. Modern MQTT client libraries reconnect for you. If
the broker or the network drops the connection, the library quietly dials back
in, and your code does nothing. That is a good feature. But the new connection
is a new, clean session.

Our device code tracked "am I connected?" with its own flag. It set the flag
when it called connect itself, and it had no hook for disconnections. So when
the broker dropped a bowl and the library reconnected it, the device code saw no
change. It was connected, it believed it was subscribed, and it kept
publishing. Publishing worked, because publishing only needs a connection.
Every inbound message went nowhere.

WHY IT HIDE FOR SO LONG

Looking back, every symptom fit. At the time, each one pointed somewhere else.

1. OUR HEALTH SIGNAL ONLY MEASURED THE HALF THAT WORKED.
   "Readings are arriving" was our proof that a bowl was healthy. That proves
   the outbound path, and only the outbound path. A device can be perfectly
   healthy at sending and completely unreachable at the same time, and an
   outbound-only check will call it healthy.
2. A RESTART FIXED IT, WHICH MADE IT LOOK LIKE FLAKINESS.
   Restarting the process runs the startup path, and the startup path
   subscribes. "Works after a restart" sends you looking for leaks and bad
   state, not for a missing step after a reconnect.
3. THE LOGS SAID SOMEONE ELSE WAS USING THE BOWL'S IDENTITY.
   The broker's connection log showed the bowls being disconnected with the
   MQTT 5 reason "session taken over" (code 142, 0x8E): another connection had
   arrived with the same client ID. The obvious reading was a second tool or
   leaked credentials acting as the bowl.It was the bowl itself, for two reasons:The duplicate-ID messages were real. The "someone else" was us.
   • After a network blip, the bowl's reconnect could arrive before the broker
     had noticed the old connection was dead. The new connection took the old
     session over a takeover with a connect in the same second.
   • A connect attempt that timed out was left retrying in the background
     while the code started a fresh one. So a single process could hold two
     clients with the same ID, each knocking the other off the broker, over
     and over.

There was one more trap. When code throws away a client object, that client's
callbacks can still fire later. A late "connection failed" from an abandoned
client can then clear the state of the live one. Every callback needs to know
which client it belongs to, and ignore itself if that client is gone. A simple
generation number does the job.

THE APPROACH COUNT SESSIONS, NOT CONNECTIONS

The fix is less about adding a reconnect handler than about who owns which
fact.

LET THE LIBRARY OWN CONNECTION STATE. If the library reconnects on its own, only
the library knows when the connection changed. So the library's callbacks are
the only thing that may change "connected": success sets it, and disconnection
or failure clears it. Calling connect yourself doesn't get a vote.

COUNT SESSIONS INSTEAD OF HOLDING A BOOLEAN. A true/false flag can go
true -> false -> true between two times you look at it, and you'd never know.
A number that goes up by one every time the broker accepts a session can't
hide a reconnect: if it changed, you're in a new session, whatever happened in
between.

TIE SUBSCRIPTIONS TO A SESSION NUMBER. Remember which session your
subscriptions were made in. If the current session is a different one,
subscribe to everything again. In MQTT, subscribing to a topic filter you
already hold just replaces it, so re-subscribing is always safe.

RE-ASK ANYTHING STILL IN FLIGHT. If a request was waiting for a reply when the
session dropped, the reply (if there was one) went to a session that no longer
exists. Don't wait out a back-off that could last many minutes. Send it again
the moment the new session is up.

MAKE THE RECEIVING SIDE IDEMPOTENT. Re-asking means you will sometimes get two
answers. The second answer for work that is already done should be a no-op,
not a second upload.

ONE CLIENT PER PROCESS. Starting a connection stops any previous client. A
failed or timed-out attempt cleans up after itself instead of retrying next to
its replacement. And while the library is reconnecting, the application waits;
it never starts a second client of its own.

The core of it is small enough to fit in a few lines. Here it is as generic
pseudocode:

on_connection_success():     session += 1;  connected = true
on_disconnection(reason):    connected = false;  log(reason)

loop:
    if not connected:
        wait()                       # the library is reconnecting
    elif session != subscribed_in:
        subscribe_all()              # every new session starts empty
        resend_unanswered_requests()
        subscribed_in = session

"WHY NOT JUST USE A PERSISTENT SESSION?" It's a fair question. MQTT 5 lets a client connect without a clean start and ask the broker to keep its session for
a while, and the broker's reply says whether a stored session was found. That
is a useful optimisation. But it isn't a guarantee: brokers limit how long they
keep a session, long outages outlast that limit, and takeovers and broker-side
resets can still give you a clean session. Re-subscribing on every new session
makes the client correct. A persistent session only makes it cheaper.

BOUND EVERYTHING THAT GROWS WHILE YOU CAN'T HEAR. A deaf device keeps working,
which means it keeps queueing. Files waiting for a go-ahead that never comes,
items that failed, local logs of what was sent all of these grew without a
limit during the deaf hours. Each now has a cap. When it is full, the oldest
item goes first, and every drop is logged. An unbounded local queue on a small
device turns a few hours of deafness into a full disk.

TESTING BREAK IT ON PURPOSE, AGAINST A REAL BROKER

A mocked MQTT client could never have caught this. The bug lives in the meeting
point between the library's automatic reconnect and the broker's session rules,
and a mock only does what you told it to. It would have reconnected straight
back into a session with every subscription intact, because that's what we
assumed.

So the test makes the real failure happen, on demand, against a real broker:

Two details matter. First, the test refuses to run while the real device
process is running, because forcing a takeover would knock a live device off
the broker. Second, when no broker is configured, the test SKIPS -- it never
passes. A reconnect test that goes green without a broker to reconnect to is
worse than no test.

RESULTS

The diagnosis explains every symptom we saw: sending worked, receiving was dead,
a restart cured it, and the duplicate-ID noise came from the bowl itself. More
usefully, the failure can now be reproduced in seconds instead of waiting hours
for a network blip. The fix is going through validation on a bench device with
that test before it reaches the fleet.

TAKEAWAYS

01 An automatic reconnect restores the connection, not the session. If the
library reconnects for you, assume it reconnected you into an empty one.

02 Count sessions; don't just hold a connected flag. A number that changed
can't hide a reconnect that a flag already forgot.

03 A device that can send is not a device that can hear. Check health on the
path you need to work, both ways, end to end.

04 "Someone else is using this identity" is sometimes you, twice. Before you
hunt for an impostor, count how many clients your own process holds.

05 Test failure modes against the real thing and make the test skip, not
pass, when the real thing isn't there.

WHERE THIS FITS AT HOOMANELY

Hoomanely builds connected pet-health products a smart bowl, a daily health
companion, a timeline of an animal's life for people who can't simply ask
their pet how it's feeling. The bowl is where much of that care starts , it sees
how much and how often a pet eats, and it picks up the small changes that are
easy to miss. That only works if the bowl can be reached as well as heard , told
to start a capture, given the go-ahead to upload, updated when it needs it. A
device that talks but can't listen quietly erodes the trust our product is
built on. Making the connection honest in both directions is part of making the
bowl something a family can rely on without thinking about it.

AUTHOR'S NOTE
Our first theory was an impostor, and the evidence for it was technically true:
the broker really was seeing two connections with one identity. Real evidence
can still point you at the wrong suspect. What finally cracked it wasn't another
read of the logs. It was asking the plainest question about the device: after
the library reconnects, who subscribes again? The answer was nobody.