AI
Model serving, inference runtimes, adapters, reasoning parsers and model evaluation. 19 posts, newest first.
Posts
A start-tag search over the whole reply: non-stream reasoning in Xinference
A non-stream chat reply that had no <think> before </think> but mentioned one after it either failed with an HTTP 500 or came back with the wrong reasoning_content.
Checking the new chunk first: a repeated end tag in streamed replies in Xinference
Once a streamed reply's reasoning had ended, a later </think> in the answer was read as the end of the reasoning again.
A start tag inserted ahead of a newline: streamed reasoning in Xinference
A streamed reply that starts with a lone newline chunk, with reasoning_content on, put the model's own <think> inside reasoning_content.
Terminated pods that hold their names: PodSet replacement in AIBrix
A PodSet in AIBrix, a control plane for serving LLMs on Kubernetes, never replaced a pod that ended Failed or Succeeded.
An unset strategy that nothing defaulted: RayClusterFleet in AIBrix
A RayClusterFleet in AIBrix, a control plane for serving LLMs on Kubernetes, never created a RayClusterReplicaSet when its strategy was unset.
An end index counted from zero: content after a leading newline in Xinference
A non-stream chat reply with a newline before the <think> tag came back with a stray > at the start of content.
A parser branch that assumed no start tag: unfinished replies in Xinference
A non-stream chat reply that stopped inside the thinking block, for example at max_tokens, came back with <think> at the front of reasoning_content.
Counter rules keyed with a suffix the parser drops: runtime metrics in AIBrix
The runtime sidecar in AIBrix, a control plane for serving LLMs on Kubernetes, copies engine metrics to aibrix: names.
Four early exits dropped the chunk: empty reasoning streams in Xinference
A streamed chat reply with no text at all, from a reasoning model that needs the <think> tag added, reached the client as data: [DONE] and nothing else.
Strategic merge kept both halves of an env entry: a merge fix in OME
An InferenceService could override a runtime's environment variable and still inherit half of the old one.
An environment override kept the source it was meant to replace in OME
OME could replace an environment variable's text and leave its old source attached.
A refactor dropped the webhook and controller: BenchmarkJob restored in OME
On OME's main branch, creating a BenchmarkJob failed at admission. If it had passed, nothing would have reconciled it.
Two route lists, one missing path: image generations in AIBrix
AIBrix's gateway accepted POST /v1/images/generations, but with no routing strategy set, the route it handed the request to had no match for that path, for every engine…
A hardcoded IPv4 address type: ModelAdapter EndpointSlices in AIBrix
AIBrix built every ModelAdapter EndpointSlice with addressType: IPv4, even when the Pods had IPv6 addresses.
The duplicate guard that never recorded a path
I found a set under // avoid duplicates that nothing wrote to.
CreateUser echoed the request, not the record: a stored-limits fix in AIBrix
Calling POST /CreateUser again for an existing user returned the limits from the request, not the ones in the store.
Two URL lists: the adapter webhook and the runtime sidecar
I found that AIBrix could admit an adapter URL and then reject it when the runtime sidecar tried to load it.
Two directories for one adapter: a Hugging Face download fix in AIBrix
AIBrix's runtime handed the inference engine a different directory for the same LoRA adapter, depending on whether the adapter had been downloaded before.
A colour convention that looked like a sim-to-real gap
I built a render generator and a 20,000-image benchmark to ask one question.