Gateways, routing, and backpressure — engineering notes on running LLM systems in production.
Structured outputs (constrained decoding) guarantee that an LLM’s response is valid against your JSON Schema. They do not guarantee that the response is correct, because providers implement only the subset of JSON Schema that compiles to a grammar: no string length limits, no numeric bounds, no minItems above 1
As AI systems move from experimental features to core product capabilities, the definition of "good intelligence" changes. In early stages, intelligence is judged by fluency, creativity, or raw problem-solving ability. In production systems, those qualities matter far less than consistency, predictability, and control. A production AI system doesn&
Client applications don't usually fail because of one big bug. They fail because a dozen tiny unbounded behaviors accidentally combine: a user tapping too fast, a scroll listener firing dozens of times per second, a reconnect event triggering a sync job, or a background worker replaying offline writes
Retrieval-Augmented Generation systems rarely fail loudly. They fail in subtler, stranger ways: a support answer sounds different today than yesterday, a troubleshooting step the assistant always got right suddenly drifts, or the bot becomes strangely confident about something no one remembers teaching it. When you dig deeper, the pattern becomes
If you've ever shipped a RAG system into production, you already know the uncomfortable truth: these systems don't fail dramatically. They fail quietly. No alarms. No error stacks. No catastrophic crashes. One day the answers simply aren't as sharp as they used to be.
AI assistants are becoming deeply embedded across apps, devices, and internal tools. They read context, recall previous turns, guide users through troubleshooting, analyze data, and invoke tools on behalf of the user. Yet the systems beneath them, models, prompts, routing layers, retrieval indexes, formatting logic, safety filters, are in constant
Introduction Choosing the right language model architecture for production is one of the most underrated engineering decisions teams make. Behind every AI feature - from a chatbot response to a search ranking - sits a strategic decision: Should we use an encoder-only model? A decoder-only model? Or a full encoder–
The first LLM integration is usually simple: you pick one provider, one model, one API key, and ship. The second and third are where things start to hurt. Suddenly you have a "cheap" model for bulk tasks, a "smart" model for critical flows, maybe a vision
Why we needed a question classifier At Hoomanely, pet parents send us thousands of questions covering health, behavior, grooming, food, and training. Those questions tell us what people actually care about and where the product falls short, but reading every one by hand is slow, inconsistent, and doesn't
LLM features usually start life in a happy place: a handful of users, low traffic, and plenty of quota. Latency looks fine, tokens are cheap, logs are quiet. Then a product launch, a marketing campaign, or a new "Ask AI" button lands, and suddenly your once-stable service is