Intelligent Conversation Memory: A 2-Level Summary System

Imagine this: you're chatting with an AI vet about your anxious rescue dog, Luna. Over hours, you've discussed her separation anxiety, dietary restrictions, medication schedules, and behavioural patterns. Then the next day, you ask a follow-up question about her progress, and the AI responds as if you're strangers meeting for the first time.

This isn't a hypothetical scenario. It's the fundamental limitation of conversational AI today. Most chat systems face a brutal tradeoff: either maintain full conversation history, hitting token limits and degrading performance, or discard older context, losing critical continuity. For pet health conversations, this isn't just frustrating, it's potentially dangerous. A forgotten allergy mention or overlooked symptom pattern could lead to harmful advice.

At Hoomanely, where we're building an AI-powered pet healthcare assistant, this problem is existential. Our conversations aren't casual chitchat, they're longitudinal information. Pet parents return days or weeks later expecting the AI to remember Luna's anxiety triggers, her medication changes, and that time she ate something she shouldn't have. We needed a system that could maintain conversation continuity across days or weeks without hitting LLM context limits. The solution: a hierarchical memory architecture that mimics how human memory works, compressing details while preserving meaning.

Why traditional approaches fall short

Full history retrieval sounds ideal but crashes into hard limits. LLMs can have a large token context window, but in practice you don't want to fill it entirely with conversation transcripts, you need space for other context too, like pet details and relevant information about the question, and hitting the context window limit increases latency.

Sliding window approaches, keeping only the last N messages, work for customer support but fail catastrophically for healthcare. If someone mentions a critical detail in message 3, and you're now at message 12, that information is gone. In veterinary contexts, early symptom mentions are often crucial for later diagnosis.

Simple summarisation, periodically compressing history into a single summary, creates its own problems. As conversations grow, that one summary becomes a bottleneck, either too vague to be useful or too detailed to fit in the context window. You end up summarising the summary, losing fidelity with each compression. The real challenge isn't just remembering, it's remembering hierarchically, the way humans do.

The landscape of memory solutions

Vector embeddings and semantic search, the trendy approach, embed all messages as vectors and retrieve semantically similar context for each new query. The appeal is obvious, smart retrieval that finds relevant past conversations even with different wording. The reality is messier, semantic similarity doesn't capture temporal causality, a symptom mentioned last week is more relevant than a similar symptom from three months ago that was resolved. Vector search treats time as just another dimension, not as the organising principle. For healthcare, temporal coherence matters more than semantic similarity.

Fixed-window with keyword extraction, keep a sliding window of recent messages but extract keywords or facts from older conversations and store them separately, Luna is allergic to chicken becomes a stored fact. This scales better than full history and preserves some long-term context, but it reduces rich conversations to bullet points, nuance dies in extraction.

What all these approaches miss is that human memory doesn't work through retrieval, it works through layered abstraction. You don't remember every word of a conversation from last month, you remember themes, outcomes, emotional beats. This is the insight we built around.

Two-level memory architecture

We built a system inspired by human memory formation: immediate recall for recent events, consolidated summaries for older context, and high-level patterns for long-term understanding.

The L1 layer, tactical summaries, captures conversation chunks, typically 5-10 message exchanges grouped by natural conversation boundaries. When a conversation crosses the threshold, we trigger L1 generation, extracting core themes discussed, user concerns and emotional state, and actionable items or recommendations given. Each L1 summary includes metadata: which messages it covers, how many exchanges it represents, and its creation time. The beauty of L1 summaries is specificity, they preserve details because they're covering a small window.

The L2 layer, strategic memory, consolidates multiple L1 summaries into higher-order insights. When 3-plus L1 summaries accumulate, we trigger L2 generation, extracting recurring themes across conversations, temporal progressions, cross-topic connections, and long-term user concerns. This layer includes its own metadata, which L1 summaries were processed, a block index for tracking consolidation state, and version information. The L2 layer is where we capture the story arc of a pet parent's journey.

Tradeoffs and design decisions

What we gained: temporal coherence, unlike vector search, our system preserves the timeline. Graceful degradation, as conversations grow longer, context quality doesn't cliff, it gradually compresses. Cross-session persistence, when a pet parent returns after days or weeks, the system rebuilds context seamlessly. Predictable performance, token usage is bounded and predictable.

What we sacrificed: perfect recall, we can't retrieve every specific detail from 50 messages ago, if exact recall matters we store that separately as structured data. Semantic flexibility, unlike embedding-based systems, we can't magically retrieve context based on conceptual similarity, our system excels at what happened over time but doesn't do find all mentions of X across history. Summarisation latency, generating summaries costs time, a few hundred milliseconds per operation, but this latency is backloaded and the tradeoff for continuity is worth it.

What we learned building this

Compression is an art, early iterations compressed too aggressively, we learned that tactical summaries need to be lossy but faithful, they simplify wording but preserve meaning and specifics. State management is critical, one of the hardest problems was ensuring we never double-summarise content and always maintain consistent state across concurrent operations, we learned to treat summarisation as a state machine problem. Pet-specific isolation is non-negotiable, in multi-pet households, context bleeding is catastrophic, partition design and query filtering aren't just performance optimisations, they're correctness requirements. Users don't notice good memory but they notice bad memory, when memory fails, trust breaks immediately.

Key takeaways

• The infinite context problem isn't really about context windows, it's about how we organise and compress memory.
• Vector embeddings, RAG, and sliding windows each solve specific problems well.
• But for longitudinal, high-stakes conversations like healthcare, hierarchical summarisation aligns with how humans naturally compress and recall information over time.
• Our two-level architecture preserves what matters: temporal progression, narrative coherence, and graceful scaling.
• Building effective AI memory isn't about cramming more tokens into prompts.
• It's about architecting systems that mirror how humans actually remember, with layers, compression, and context-aware detail.