Building Stateful Continuity in Stateless LLM Services: A Multi-Tier Session Architecture

Modern AI products feel personal only when they remember, that bug you were debugging yesterday, the pet profile you set up last month, the long-running coaching thread you keep coming back to. Under the hood, though, LLM inference APIs don't remember anything. Every call is a clean slate, you send a prompt, you get tokens back. All the continuity users feel is an illusion built by your backend.

This post walks through a multi-tier session architecture that makes that illusion reliable: sessions survive app restarts, reconnects, and server restarts; recovery happens in under a few seconds; and token overhead for replaying context stays predictable instead of exploding over time. We'll layer state across device, edge, and cloud, use checkpointed snapshots instead of raw logs, and design idempotent replay paths that are safe even when things crash halfway through.

The problem: stateful expectations on stateless LLMs

Users don't think in "requests," they think in sessions, "continue that architecture review from yesterday," "use the same pet profile as last time," "pick up this debugging thread on my laptop instead of the phone."

From a backend perspective, three hard problems show up. Context bloat and token cost: the naive approach of resending the entire conversation history on every call means token counts grow linearly with time, latency creeps up, and your bill spikes. Fragile recovery after crashes or reconnects: mobile app restarts, WebSocket reconnects, pods getting cycled, if you keep state only in memory you either lose context entirely or replay from a stale snapshot and risk confusing the user. Multi-device and long-lived sessions: a single logical session might span multiple devices, multiple backends, and hours or days of elapsed time, needing a consistent session ID and shared state model rather than whatever one pod happens to remember. The goal is clear: stateful continuity that doesn't depend on any single process and doesn't explode your token budget.

The approach: layered session state across device, edge, cloud

Instead of a single "session store," treat state as layered. The device tier gives instant, local continuity, holding a small recent window of the conversation plus last-known summaries, used for quick UI restore and short reconnections, and working with temporary offline actions or queued requests. The edge tier is a fast, hot session cache, living close to your inference gateway (Redis, regional store, or in-memory service with replication), storing hot sessions, last few checkpoints, recent requests, and replay metadata, optimized for sub-10ms access and sub-3s full recovery. The cloud tier is durable, canonical history, storing authoritative session history and long-lived summaries, optimized for durability and analytics rather than ultra-low latency, used to rebuild edge or device state when needed.

The key idea: each tier knows enough to recover the next tier down, but only one tier is source of truth for any given artifact. The cloud is canonical truth for history, the edge is truth for hot checkpoints and replay metadata, and the device is truth for what the user last saw.

Modeling session state explicitly

First, stop thinking in terms of "we store the raw messages" and start thinking in terms of artifacts. The transcript log is the full list of turns (user, assistant, tool). The rolling summary is a compressed representation of the session so far. The active context is the messages you actually send to the LLM for the next request. The replay cursor is a pointer saying "we've applied events up to sequence N into the LLM-facing context." And per-session metadata covers user ID, device IDs, and flags like safe mode or expert mode.

A typical checkpoint payload might look like:

{
  "session_id": "sess_123",
  "checkpoint_version": 12,
  "summary": "Short summary of the entire conversation so far...",
  "recent_messages": [
    {"role": "user", "content": "Let's refine the architecture."},
    {"role": "assistant", "content": "Here are three options..."}
  ],
  "replay_cursor": 58
}

This is what you store and move between tiers, not the entire raw log.

Defining clear responsibilities per tier

The device tier caches the last rendered assistant response, the current draft user message, and the most recent checkpoint from edge or cloud. On restart it reconstructs the UI from local cache immediately, then in the background verifies with edge what the latest checkpoint version is.

The edge tier stores N recent checkpoints per session (say the last 5) plus the replay cursor and small metadata for each. It responds quickly to "give me the latest checkpoint for session X," maintains per-session invariants like monotonic checkpoint_version and idempotent updates, and optionally holds a short-lived fast transcript buffer of the last 20 events.

The cloud tier stores the full transcript log (append-only), all checkpoints for analytics or backfill, and long-term per-session metrics like duration, cost, and device mix. It's reconstructible into a new edge cache when needed, and supports offline analytics and reprocessing. Writing down these responsibilities avoids the classic trap where "some of the truth" lives in every tier and nothing can be safely rebuilt.

Checkpointing: when and what to snapshot

A good checkpoint strategy balances freshness with cost. Snapshot after each assistant turn that completes successfully, after any tool-call boundary like a retrieval step or multi-step tool chain, and on significant state milestones like a new profile being created, a goal updated, or a session escalated to expert mode. You don't need to snapshot after every keystroke or streaming token, think in semantic turns, points where you'd be comfortable resuming from if a crash happened.

What to snapshot: a rolling summary, one or two paragraphs capturing the entire session; tail messages, the last 3-8 user/assistant turns most relevant to the next step; a replay cursor ensuring you know which events are already applied; and feature flags or mode indicating anything that changes what the model should do next. In many production systems this keeps LLM context per request under a few hundred tokens, even for very long sessions.

Idempotent replay: never double-apply the same event

Replay is where sessions often go wrong. Consider this scenario: the app sends a request with session state up to event seq=60, the backend processes it and sends it to the LLM, but the response never reaches the device due to a network blip, and the app retries the same request. If your system isn't idempotent you might apply some events twice to the summary, advance the replay cursor inconsistently, or confuse downstream logs.

A robust pattern: every user or tool event gets a monotonically increasing seq number. Checkpoints record replay_cursor as the max seq that's been applied into the LLM-facing summary. On any replay, the edge tier loads the latest checkpoint, replays only events with seq greater than replay_cursor, re-derives the new summary and tail messages, and produces a new checkpoint with an incremented checkpoint_version. Because the replay is purely functional, given checkpoint plus new events equals new checkpoint, retries are safe. If the same batch gets applied twice, you can detect and ignore duplicates based on seq. This also makes it straightforward to rebuild edge caches from cloud history by re-running the same replay function over the stored log.

Differential context: reducing token cost

Even with checkpoints, you need to be smart about what you send to the LLM. Think of context as two parts: stable context (the rolling summary and any static instructions or profile) and delta context (the last few messages and any new events since the last checkpoint). On each request your LLM input looks like global instructions, the rolled-up session summary, key session metadata, recent tail messages, and the new user message. The only part that grows is the recent tail plus new message, which you periodically reset by creating a new checkpoint and trimming the tail.

Compared against a naive full-transcript-every-time approach, you'll typically see token replay reduction of 70-90% over long sessions, more predictable latency because context size has a soft upper bound, and easier cost modeling for finance.

Degradation modes: failing gracefully, not catastrophically

Even with good architecture, things break, edge cache clusters go down or get evicted, cloud history becomes temporarily unavailable, a device has an extremely old cached checkpoint. You want predictable, user-safe degradation. A simple ladder of fallbacks: happy path, the device has a fresh checkpoint version, edge confirms, you resume within milliseconds. Edge miss but cloud hit, edge can't find the checkpoint but cloud has full history, rebuild the latest checkpoint by replaying from cloud, cache it at edge, and from the user's perspective it's a small delay but no loss in context. Cloud degradation with partial history, resume from the last known good checkpoint, inform the user gently some fine-grained context may be missing, and keep the high-level summary intact so conversation still feels coherent. Total failure, if everything fails, be explicit, start a new session and show a brief, honest notice. The important part: never pretend you remember when you don't, that's worse than admitting a reset.

Observability: measure recovery, not just requests

Once you have this architecture, observability becomes richer than simple request counts. Track per-session metrics like recovery latency (time from reconnect to first usable assistant response), token replay overhead (tokens spent on summary/context per call versus tokens in the final answer), checkpoint coverage (fraction of sessions with a recent checkpoint, under 3 turns old), and degradation rate (percentage of sessions falling back from edge-to-cloud or cloud-to-reset). These metrics tell you whether your multi-tier design is doing its job, especially as you scale to tens of thousands of sessions.

Results: what changes when you ship this

User experience stabilizes, reconnects feel more like resume than restart, multi-device workflows become natural, and crashes or app upgrades don't blow away context. Inference cost becomes predictable, you can bound the context size instead of letting it grow unbounded, finance teams can estimate cost per active session, and you avoid pathological long-lived sessions dominating your LLM bill. The backend becomes teardown-safe, individual pods can restart freely, edge caches can get replaced without permanent data loss, and you can run A/B tests on different LLMs or prompts without coupling them to in-memory state.

In dogfooding and production-style workloads with 15k-plus sessions, teams typically see recovery latencies under 3 seconds even after full reconnects, token replay overhead reduced by roughly 80-90% compared to naive full transcript replay, and more consistent behavior in stress tests simulating mobile network flakiness. Exact numbers vary by product, but the directional improvements are consistent.

At Hoomanely, our mission is to help pet parents keep their companions healthier and happier using data and AI. That means our assistants and dashboards can't just answer one-off questions, they need to remember a pet's history across multiple devices and surfaces, take into account long-running context like feeding patterns, weight trends, and behavior notes, and remain stable even when users open and close the app frequently or move between networks. A multi-tier session architecture is what lets an AI assistant understand that today's feeding question relates to the pattern seen last week, survive a temporary connectivity drop without losing track of the conversation, and scale as we add more AI-driven features without each new workflow reinventing its own session memory.

Key takeaways

Don't rely on the model for memory, LLM APIs are stateless, your backend owns continuity. Layer your state, device for UI continuity, edge for hot cache, cloud for canonical history. Use checkpoints, not raw logs, snapshot summary plus tail messages plus replay cursor. Make replay idempotent and deterministic with sequence numbers and pure checkpoint-plus-events logic. Send differential context to the model, stable summary plus small deltas instead of full transcripts. Define clear degradation modes, edge miss, cloud rebuild, and explicit reset when all else fails. And instrument recovery and token overhead, treating session continuity as a first-class SLO. Get these pieces right and you end up with resilient, token-efficient, reconnect-friendly LLM services that behave like they were stateful all along, without sacrificing the scalability and simplicity of stateless inference APIs.