The Search Index That Was Always Empty: Debugging Silent Failures in OpenSearch Serverless

Amazon OpenSearch Serverless (AOSS) presents the OpenSearch REST API. You point the standard client at it, sign your requests, and everything you already
know appears to apply. That appearance is the whole problem. It is not a
managed OpenSearch cluster, it is a different product that accepts a subset of
the same requests, and the boundary of that subset is nowhere near where we
guessed it would be.

We found out slowly. A text search feature went live and ran for two weeks
against an index holding zero documents. Every query returned 200 and an empty
list. Every write had been rejected since the day it deployed. Nobody saw an
error, because from the outside there wasn't one.

This is what we learned about that boundary, told in the order we hit it.

THE WRITE THAT WAS NEVER GOING TO WORK

We assumed we owned our document IDs. Every search engine we had used lets you
choose them, and the API accepts them in the URL, so the assumption never
surfaced as an assumption.

AOSS mints every document's internal ID itself, and rejects a client-supplied
one on every collection type but one. Ours was not that one. What comes back is
not ambiguous:

400 Document ID is not supported in create/index operation request

Every document we had ever tried to write hit that response. The index sat
there with its mapping intact and nothing in it, while queries against it went
on answering cheerfully with no results. The feature looked deployed. It was
inert.

The thing worth dwelling on is not the 400 it is what the 400 removes. If
you cannot choose an ID, you cannot address a document by one either, which
means your ID has to live inside the document as an ordinary field. Every
operation that used to be a single addressed call becomes a lookup first.

Two things let this ship. The constraint was already recorded in our codebase
discovered a year earlier on a different write path, noted, and then not
found again by the person who added the second one. And our tests ran against a
local stand-in for the engine that was more permissive than the engine, so it
accepted the exact write production refuses. Every test passed. We have written
about that second failure mode separately,it is the more general one.

THEN THE WORKAROUND NEEDED ITS OWN WORKAROUND

Find-then-replace assumes the find is accurate. It is not, because AOSS
refreshes an index on its own schedule and a document written a moment ago is
not necessarily searchable yet.

That is unremarkable for a search feature and quietly serious when a search is
how you locate the thing you are about to replace. A write that arrives twice
inside the refresh window leaves two copies carrying the same logical ID, and
the second attempt cannot see the first to clean it up.

We stopped trying to make the lookup exact. It returns however many copies it
can currently see, and the replace sweeps all of them, so each subsequent write
clears more than it creates and the state converges. A search running inside
that window can still rank the same item twice. That is a bounded, self-healing
wrong, and we took it over anything involving a lock the alternative to
converging here is coordinating, which is a much larger thing to own than the
problem justifies.

THE SECOND TIME WE MADE THE SAME MISTAKE

Querying an index that nothing has created yet returns a 404, which the client
library raises as an exception nothing above it was expecting. So in any
environment where the endpoint was configured before the first document was
written, the search route answered a bare 500. Fair enough a small gap, an
easy fix.

The easy fix was a query flag that tells the engine to treat a missing index as
an empty one. Cleaner than catching an exception, expressed in the right place,
and used everywhere in the OpenSearch ecosystem.

It is not in AOSS's supported operations.

That is the sentence worth sitting with, because reaching for it would have
been the identical mistake that caused the first bug taking an operation
from the upstream project's API surface and assuming the serverless subset
carries it. We had been burned by exactly this a fortnight earlier and walked
straight back toward it. The version that shipped catches the library's own
exception in one place instead, and reads a missing index the same way it reads
an empty one: no results. A corpus that does not exist has no hits in it
either.

We also considered creating the index on the read path, and rejected it for a
reason that has nothing to do with AOSS. It would let a read establish
infrastructure so the shape of the index would be whatever the first reader
happened to ask for, rather than what the writer declares.

THE BUG THAT WOULD NEVER HAVE ANNOUNCED ITSELF

This one we caught before it shipped, and only because by then we were
suspicious of everything.

OpenSearch will create an index from the first document you hand it. Dynamic
mapping turns every string into an analysed field, which means an ID gets
broken into tokens. A filter that is supposed to match one exact identifier
then matches on fragments of it.

So the mapping is declared before anything is written, and the shape of it is
almost boring , one field analysed the prose a person is actually searching
and every identifier and category stored whole.

"text":   {"type": "text", "analyzer": "english"},
"doc_id": {"type": "keyword"}

The choice of analyser is not cosmetic. Without stemming, a lexical search
answers only for the exact inflection whoever wrote the document happened to
use: someone searching "training" finds nothing written about "trainings".

WHAT THE CONSTRAINTS DID TO THE DESIGN

By this point we were making design decisions on the basis of what the
platform would actually accept rather than what the API suggested, and the
biggest of those was about retrieval.

Semantic search and lexical search answer different questions. They also cost
very differently, and that turned out to be the deciding factor rather than a
footnote to it.

Someone typing words into a search box is asking a lexical question. Answering
it through embeddings buys some extra recall for a per-item cost on content
that is written continuously, which is the wrong trade for that surface. We
kept both retrieval models available and made them genuinely separate rather
than one path with a switch partly because they share no argument that means
the same thing in both, and partly because a switch is something a caller
eventually flips without meaning to.

Four assumptions, and what each one actually cost

WE ASSUMED We choose our document IDs.
WHAT WAS TRUE The engine mints them and rejects ours.
IT COST US Two weeks of an empty index and a search that looked fine.

WE ASSUMED A written document is immediately findable.
WHAT WAS TRUE Refresh happens on the engine's schedule.
IT COST US A workaround that had to converge rather than be correct.

WE ASSUMED Query flags from the upstream docs are available.
WHAT WAS TRUE The serverless subset does not carry all of them.
IT COST US Nearly a repeat of the first bug, two weeks later.

WE ASSUMED Letting the index infer its own shape is harmless.
WHAT WAS TRUE Inferred shapes break exact match filtering silently.
IT COST US Nothing, this time we were finally looking.

WHERE IT LANDED

Search works now. Writes land, an edit replaces rather than duplicates, and an
index nobody has created yet returns no results instead of an error.

The costs we knowingly carry are worth naming, because a post that ends at
"and then it worked" is not much use to anyone. One logical write is three
round trips. A duplicate can exist briefly inside the refresh window. Deep
paging is bounded by policy rather than by capability, because asking for a
distant page makes the engine rank and discard everything ahead of it.

None of that is a complaint about AOSS. The service is doing what it says; we
were reading a different document. The generalisable part is smaller and
duller than a list of gotchas: API compatibility is a claim about the shape of
requests, not about their semantics. When a managed service publishes a table
of the operations it supports, that table is the specification, and the
upstream project's documentation is background reading.

WHAT WE THINK NOW

01 We read the supported-operations table before writing the call, not after
reading the 400. It is a short document and we have now paid for it twice.

02 We treat "compatible API" as a warning rather than a reassurance. The
closer a service looks to something familiar, the less our familiarity is
worth.

03 We do not let anything infer the shape of a store from the first thing
written into it. An inferred shape fails by returning plausible answers.

04 We choose between retrieval models by cost shape first. Capability
comparisons between semantic and lexical search are interesting; the
billing difference is decisive.

05 We assume nothing reads back what it has just written. Anything that has
to is designed to converge, not to be right the first time.

WHERE THIS FITS AT HOOMANELY

Hoomanely builds connected pet-health products , a smart bowl, a daily health
companion, a shared record of an animal's life. A great deal of what people
know about caring for their pets lives in what other pet parents have already
written down, so being able to find that reliably is not a side feature of the
product; it is a good part of the point. Getting retrieval honest the right
model for the right question, at a cost that scales with a growing community
rather than against it is what lets everything above it lean on the answer.