Streaming Tokens From a Blocking SDK
Our pet-health assistant streams its answers token by token. Under load, something odd showed up: response times for completely unrelated endpoints climbed in lockstep with how many people were mid-conversation. History reads, a settings fetch, a suggestion request. CPU sat close to idle. Nothing was throttled, nothing was retrying, no