Observability for AI Systems: How We Solved LLM Performance in Production

Pet parents face a unique challenge: their companions can't tell them what's wrong. Is that whimper a sign of anxiety or something more serious? Can Bella eat blueberries? Why is Max suddenly refusing his favourite food? These questions keep pet parents awake at night, scrolling through conflicting advice on forums and second-guessing every decision.

This is where Hoomanely's AI chat assistant steps in, combining veterinary knowledge, nutritional databases, and behavioural insights to provide instant, personalised guidance at 2 AM when the vet's office is closed. But here's the catch, when you deploy an LLM to answer critical questions about pet health, you've created a new problem: you have no idea what it's actually saying to your users.

The black box problem

Large Language Models are powerful, but they're also unpredictable. Ask the same question twice, and you might get two different answers. Change the temperature parameter slightly, and the tone shifts from cautious to overly confident. Feed it incomplete context, and it might hallucinate facts that sound authoritative but are dangerously wrong.

For a pet care AI, the stakes are real. An incorrect answer about toxic foods could lead to an emergency vet visit. Overly confident advice might delay treatment for a serious condition. Inconsistent responses erode trust, if your AI says one thing today and something else tomorrow, users will stop relying on it. Traditional monitoring doesn't help here, you can track API uptime, response times, and error rates, but none of that tells you whether your LLM just told someone that chocolate is safe for dogs. You need visibility into what the model is actually doing, the prompts it receives, the responses it generates, the context it uses, and the resources it consumes.

Approaches to LLM observability

Manual spot-checking is cheap to implement, you just need a database query and someone's time, but it's slow, doesn't scale, and you'll miss most issues. User feedback systems, thumbs up or down, catch egregious errors but suffer from selection bias, most users don't leave feedback unless something goes really wrong, so you'll never know about subtle quality degradations or users who silently left.

Third-party observability platforms like LangSmith, Weights & Biases, or Helicone offer sophisticated LLM monitoring out of the box, powerful but adding external dependencies, costs, and sometimes latency. For teams with strict data privacy requirements or cost constraints, it's not feasible. Custom logging infrastructure gives complete control over what's tracked, where data lives, and how it's analysed, but you're building and maintaining infrastructure instead of buying it.

Building our log dashboard

We chose to build a custom observability system, not because we thought we'd build something better than dedicated platforms, but because we needed something that fit our specific workflow, integrated with our existing tools, and gave us flexibility to track metrics unique to pet care interactions. The core insight was simple: treat every LLM call as a structured event you can query, analyse, and learn from.

Every time an LLM gets invoked, answering a pet parent's question, analysing a food label photo, generating a pet insight, we log the event with key metadata. Model configuration tracks which model was used, temperature, max tokens, and any special parameters, helping correlate quality issues with specific configurations. Prompt engineering tags each call with its use case type and logs what context was passed, did we include the pet's medical history, breed information, previous conversation turns, knowing what the model had access to is crucial for debugging hallucinations.

Resource consumption tracks token counts and their associated costs, since LLM usage can get expensive fast, single conversations have consumed thousands of tokens when users ask follow-up questions. Performance metrics measure end-to-end latency in milliseconds, both time to first token and total completion time, since pet parents won't wait 10 seconds for an answer. Response metadata captures the actual completion text, any tool calls made, and whether the response used retrieval, the ground truth of what we actually told the user.

Having this data centralised changes how we work. Instead of guessing why users report inconsistent answers, we can query logs by use case and see exactly what prompts were used and what context was available. When a pet parent says the AI gave bad advice, we can pull up that exact conversation, see the prompt, examine the context, and understand what went wrong. This helps with hallucination detection, if the LLM mentions something we've never documented or contradicts established guidelines, we know it needs checking, and prompt optimisation, tracking performance by prompt type to detect anomalies and optimise for that specific use case.

Key takeaways

• If you're building AI-powered products, especially in domains where accuracy matters, invest in observability early.
• You don't need a perfect system from day one, but you need something that lets you see what's happening in production.
• Start with structured logging, making every LLM call generate a queryable event with relevant metadata costs almost nothing to implement.
• Track what matters to your use case, generic metrics like average response time are fine, but you need domain-specific signals too, for us that's retrieval quality and context completeness.
• Remember that observability is a means to an end, the goal isn't to collect data, it's to build better, more reliable AI systems that users can trust.