hf:// alias in PR #3130, merged 2026-10-16 about 2 hours 27 minutes after I opened it, and an absolute path starting with / in PR #3131, merged 2026-10-17 about 6 hours 5 minutes after opening. The webhook and the sidecar kept separate lists of acceptable inputs.
Both failures depended on the controller using the runtime sidecar. With that path enabled and an aibrix-runtime container present, the controller sent the adapter URL to the sidecar. The direct path handled the same inputs differently: it resolved the alias to a repository identifier and passed an absolute path through to the engine.
The alias failure was narrower than a broken downloader. The longer huggingface:// spelling already worked. The docs and samples used that spelling, while the webhook’s supported-input error advertised hf:// too. For absolute paths, the loading guide explicitly told readers to mount shared storage into the pod and use a value starting with /. I was following an accepted input to its consumer, rather than adding a new URL form.
In pkg/webhook/modeladapter_webhook.go, the webhook parses artifactURL and calls ValidateArtifactURL. Its list lives in pkg/utils/modeladapter.go: AllowedSchemas includes both hf:// and /.
The controller’s pkg/controller/modeladapter/lora_client.go then chooses the load payload. For the sidecar, loadAdapterCall puts the original value into artifact_url. It does not turn the alias into the longer spelling or change a mounted path into a remote URL.
Next comes load_adapter_with_delegation in python/aibrix/aibrix/runtime/artifact_service.py. Before my changes, it sent every value through download_artifact. That reached get_downloader in python/aibrix/aibrix/runtime/downloaders.py, which selected a downloader by the parsed scheme. There was an entry for huggingface, none for hf, and none for the empty scheme of an absolute path. The webhook’s acceptance did not carry across that boundary.
#556 added absolute-path support to the controller and its validation. Later, #1680 introduced downloading through the runtime. Its downloader map had the longer spelling but no alias, and its delegation method sent all inputs to the downloader.
For the alias, I added the missing dispatch entry:
"huggingface": HuggingFaceArtifactDownloader(),
+ "hf": HuggingFaceArtifactDownloader(),
The downloader already built the repository identifier from the URL’s netloc and path. It did not need a different download implementation for the alias.
For a mounted path, there is nothing to download. The merged branch is:
if artifact_url.startswith("/"):
self._validate_lora_name(lora_name)
local_path = artifact_url
else:
local_path = await self.download_artifact(
artifact_url, lora_name, credentials
)
The engine receives that value as lora_path. It resolves the path in its own container. The runtime does not copy the mounted directory or check that it exists there.
My PR descriptions record the local before-and-after runs. For the alias, the focused file had 1 failed, 2 passed before the fix and 3 passed after it. The runtime suite reported 178 passed. That test uses a stand-in for the download call; it checks dispatch and the returned directory, without a real Hub download.
For the local-path change, the final focused file had 2 failed, 2 passed before and 4 passed after. The failures were the path-load cases. The other cases checked invalid names. The full runtime directory went from 2 failed, 179 passed to 181 passed in the runs recorded for that PR.
The path tests use a recording engine. They check the forwarded path and that the runtime writes nothing into its download directory. They do not prove a real engine loaded files from a mounted volume.
On the alias PR, the review reproduced the failure and suggested sharing a downloader instance and listing the alias in docstrings. Those were optional suggestions; the merged change contains the dispatch entry and its test.
The local-path review found a more useful mistake in my first version. A review bot pointed out that bypassing download_artifact also bypassed name validation. The review confirmed it: a name containing a slash could load through the new branch, then produce a confusing cleanup error after the engine had already unloaded it.
I added the explicit validation call in c8ce9b77, plus a test requiring local and remote inputs to reject the same invalid name before reaching the engine. The follow-up review removed that call and saw the local case fail. @varungup90 merged both PRs.
If you have found an input accepted at the front door and rejected later, reply on X.
]]>// avoid duplicates that nothing wrote to. AIBrix’s appendCustomModelRouterPaths checked it before appending a path, so repeated annotation entries became repeated HTTPRoute matches. I fixed it in PR #3133, merged 2026-10-17 about 1 hour 57 minutes after I opened it. The impact was narrow: identical matches did not change routing, but they used space near the 64-match limit.
The input is the custom-path annotation defined by ModelAnnoRouterCustomPath. A list such as /version,/score,/version added the first path again at the end. The helper also removed spaces before checking for duplicates, so /f oo and /foo became the same path and still both got appended. A custom entry equal to a default path could repeat a match too.
I do not have an outage report. My evidence is the code, the failing table-test row, and the local count-and-schema check recorded in the PR.
In pkg/controller/modelrouter/modelrouter_controller.go, workload annotations reach createHTTPRoute. buildHTTPRoute first makes the default matches from modelPaths, then calls appendCustomModelRouterPaths to extend the route.
The helper reads the comma-separated annotation, removes spaces, and skips empty entries. Each accepted custom path becomes a prefix match with the same model-header condition as the default matches. All of these matches go into the same rule. Repeating an identical path here adds no new routing choice.
There was already a pathSet, and the loop checked whether it contained each normalized path. But the set started empty and stayed empty. The exists check could never find an earlier entry. Seeing a set and a duplicate-check comment was not enough; I had to look for the write.
I traced the helper to #1841, which added custom paths through an annotation. Its first commit used a pathMap and recorded accepted paths. A follow-up in that PR replaced it with pathSet and dropped the write. The empty-set guard was already present when #1841 merged.
My first fix filled that gap inside the loop. It kept the first occurrence and recorded each path before appending it. That solved repeated entries within the annotation. Review made me look at the matches already on the route as well.
This is the relevant part of the merged change, with the existing skip check kept as context:
pathSlice := strings.Split(paths, ",")
// avoid duplicates
pathSet := make(map[string]struct{})
+for _, match := range httpRoute.Spec.Rules[0].Matches {
+ if match.Path != nil && match.Path.Value != nil {
+ pathSet[*match.Path.Value] = struct{}{}
+ }
+}
for _, path := range pathSlice {
// remove illegal space in path
path = strings.ReplaceAll(path, " ", "")
if _, exists := pathSet[path]; path == "" || exists {
continue
}
+ pathSet[path] = struct{}{}
Seeding the set means a custom path already covered by a default match is skipped. Recording each new entry means later repeats in the annotation are skipped too. The first occurrence keeps its place; the change does not reorder the remaining paths.
At this change, modelPaths had 22 entries. The HTTPRoute schema used by the pinned Envoy Gateway dependency allowed 64 matches per rule. That left room for 42 distinct custom paths outside the default set.
The PR records a local check against that schema: 42 such paths plus a repeat made 65 matches before the fix, so the route exceeded the limit. After deduplication, it had 64. Repeating a default path had the same effect. A list of 43 distinct non-default paths still made 65 matches after my change; removing duplicates cannot make that list fit.
On initial creation, a rejected route makes createHTTPRoute log Failed to create httproute and return. That is a concrete consequence, but it requires a long list near the cap. I would not describe this as a general routing fix. Most lists only lose useless repeated matches.
I added a row to TestAppendCustomModelRouterPaths in pkg/controller/modelrouter/modelrouter_controller_test.go. It starts with an /origin match and supplies /foo,/bar,/foo, /f oo ,/bar,/origin. The expected paths are /origin, /foo, /bar. The existing table checks also verify the model header on added matches.
The final PR body records the failure before the fix as expected 3 matches, got 7. Across the package, the verbose run showed 22 top-level tests passing and 1 failing. With the fix, all 23 passed. Those are the recorded local results, rather than a new run for this post.
A review bot caught the limit of my first version: adding to an initially empty set only deduplicated the annotation itself. A custom entry could still duplicate a default match.
I added the initial scan in f06d171f and extended the repeated-path test with /origin. That existing match is why the row now checks both kinds of repetition. @varungup90 merged the PR.
If you have found a guard whose state never changed, reply on X.
]]>POST /v1/images/generations, but with no routing strategy set, the route it handed the request to had no match for that path, for every engine except xdit. I fixed it by adding one line to a list (PR #3132, merged 2026-10-17, about 55 minutes after I opened it). There was no existing issue for it.
The image guide in the repo (docs/source/features/gateway-routing/omni-image.rst) shows a curl call with a model field and no routing-strategy header. On a setup where nothing else sets a strategy, that call reached the gateway, and then the model’s own route had nothing that matched the path. PR #3017 says that entries added to modelPaths “404 for models deployed before an upgrade”. That has a different cause, a stale route, but the same shape: the model’s route lacks the path. My evidence is reading the code and the unit test below, not a request through a running gateway, so I give no status code of my own.
Without a routing strategy, a request passes through two routes: a shared one, then one per model. The shared route, aibrix-reserved-router, lists the API paths the gateway serves and sends them to the gateway plugin. That list lives in two files, config/gateway/gateway-plugin/gateway-plugin.yaml and the Helm chart’s httproute.yaml. Both already had /v1/images/generations.
The plugin reads model from the body and looks for a routing strategy, in this order: one locked in the model’s config, the routing-strategy request header, the config profile’s strategy, then the ROUTING_ALGORITHM environment variable. If none is set, the strategy is RouterNotSet. The plugin then picks no pod. It sets a model header, and a code comment says RouterNotSet “delegates routing to the HTTPRoute/k8s Service”.
That second route exists once per model. buildHTTPRoute in pkg/controller/modelrouter/modelrouter_controller.go builds it with one match per entry of modelPaths, and each match also requires the model header to equal the model’s name. A path that is not in modelPaths has no match.
One engine was unaffected. For pods whose engine is xdit, getEngineBasedPathRewrite turns /v1/images/generations into /generate, and /generate was already in modelPaths. An engine that keeps the OpenAI path, such as vLLM-Omni, got no match.
/v1/image/generations to the OpenAI spelling /v1/images/generations. Before it, the plugin rewrote the old path to /generate for every engine, and /generate was in modelPaths; it limited that rewrite to xdit pods. It changed nothing under pkg/controller/modelrouter. At that commit modelPaths had eight entries and none was an OpenAI-style image path./v1/images/edits to both shared-route files and to modelPaths, “so it is served both with and without a routing strategy”. It opened a // Image group in modelPaths holding only that path.So before this PR, /v1/images/edits was in the shared route (both files) and in modelPaths, and /v1/images/generations was only in the shared route.
// Image
+ "/v1/images/generations",
"/v1/images/edits",
A comment above modelPaths says it also sets the order of each route’s matches and that new paths go into their group, so the line sits next to the edits path. Routes of models that already exist are brought up to date by the resync from #3017, so on a build that has both changes, existing models should pick up the new path after an upgrade.
func TestBuildHTTPRouteMatchesImageGenerations(t *testing.T) {
route := buildHTTPRoute("default", "demo", nil, nil)
paths := routeMatchPaths(route)
if !slices.Contains(paths, "/v1/images/generations") {
t.Errorf("per-model HTTPRoute does not match /v1/images/generations, got %v", paths)
}
}
The package already had tests that build a route and compare its match paths to modelPaths, among them TestInformerEventsCreateHTTPRoute and TestEnsureHTTPRoutesUpdatesRouteMissingNewPaths. The route and the expected value come from the same list, so a path missing from the list is missing from both sides and the comparison passes. The new test writes the path as a literal.
I checked that locally on the merge commit. With my line removed, 22 of the package’s 23 tests passed, the existing comparison tests among them, and the new one failed with per-model HTTPRoute does not match /v1/images/generations, got [...]. With the line back, all 23 passed. I counted top-level tests in go test -v output, since plain go test prints no counts.
A review bot posted a summary 30 seconds after I opened the PR, with nothing to change. @varungup90, who had also merged #3086 and #3017, approved with no review text and merged 13 seconds later. The PR has no other comments and stayed one commit.
If a list in your own code has two homes and one of them drifted, reply on X.
]]>valueFrom, making the rendered workload invalid. I fixed the matching-entry assignment in PR #1231, merged 2026-10-17, about 59 minutes after I opened it.
An environment entry can provide a literal through value, or obtain its value through valueFrom, such as a ConfigMap or Secret reference. I used a ConfigMap reference in the regression test. The troublesome transition was from a sourced entry to a computed literal with the same name.
For the source trace below, I use spec.deploymentMode: RawDeployment, a runner with a recognized accelerator resource count of one, and no leader or worker blocks. The input is PARALLELISM_SIZE using valueFrom.configMapKeyRef in spec.engine.runner.env. With no tensorParallelismOverride in the runtime’s matched accelerator configuration, OME computes the literal "1".
That overwrite was intentional. PR #1168 documented that the computation replaces any existing PARALLELISM_SIZE in the runner environment. I was fixing how that promise was implemented, rather than adding a new precedence rule.
The service schema in config/crd/full/ome.io_inferenceservices.yaml accepts the runner’s ConfigMap reference. In pkg/controller/v1beta1/inferenceservice/utils/merging.go, runtime engine defaults are merged with the service’s engine overrides. The service’s values take precedence.
Rendering then reaches EnginePodSpec in pkg/render/engine.go. With no leader block, it selects the top-level engine runner. It applies the environment updates, merges resources, and applies accelerator argument overrides. When no accelerator parallelism override owns the setting, it calls SetParallelismEnvVar.
That function lives in pkg/render/accelerator.go. It computes the parallelism size from the recognized resource count and the primary and worker counts. It builds a literal EnvVar named PARALLELISM_SIZE and passes it to UpdateEnvVars in pkg/controller/v1beta1/inferenceservice/utils/container.go.
The old helper searched the container’s environment by name. On a match, it assigned only container.Env[i].Value. The ConfigMap reference lived in a different field, ValueFrom, so it survived. The entry now said to use "1" and to read a source. A Secret reference in the same field would survive that assignment too; the added regression test specifically covers the ConfigMap case.
In pkg/controller/v1beta1/inferenceservice/reconcilers/deployment/deployment_reconciler.go, the rendered pod spec becomes the Deployment’s pod template. For a new Deployment, the reconciler submits a create request. For an existing one, it first attempts a dry-run update, where validation can reject the template before the normal update.
The validation failure documented in my PR is “may not be specified when value is not empty”. It rejects valueFrom beside a nonempty literal at the pod template. In this RawDeployment path, the Deployment itself fails admission; I would look there before looking for a container startup failure.
The reconciler returns the error. In pkg/controller/v1beta1/inferenceservice/controller.go, a non-conflict component reconciliation error becomes a ComponentReconcileError warning event on the InferenceService. That connects the small helper assignment to the service-level symptom. My evidence here is the source trace and the PR’s tests; I did not validate a live deployment.
The existing update test started with a literal and replaced it with another literal. There was no old source to clear. The helper’s append branch already copied the whole incoming entry, so adding a name and replacing a name did not use the same rule.
-// If it exists, update its value.
-container.Env[i].Value = envVar.Value
+// If it exists, replace it.
+container.Env[i] = *envVar
The replacement copies Name, Value, and ValueFrom together. A literal override has no source pointer, so replacing the entry clears the old one. The search still stops at the first matching name, and the append branch is unchanged.
I added a row to TestUpdateEnvVars in pkg/controller/v1beta1/inferenceservice/utils/utils_test.go. It starts with TEST_VAR reading key value from ConfigMap settings, then overrides it with new-value. The expected entry contains the literal and no source.
The results recorded in my PR isolate the change: with only the helper reverted, 1 row failed and 4 passed. With the fix, all 5 passed. The scoped suites passed with 240 leaf cases in utils and 71 in render, excluding parent test groups. Those are the PR’s local results, not fresh runs for this post.
I also recorded a validation limit. The pinned full linter could not finish type checking because of incompatible dependency export data. Its fast mode passed. I did not claim a full lint pass or a full-suite run.
A review bot reported no actionable comments, but separately warned about docstring coverage and flagged redundant comments. GitHub records no inline review comments or approval reviews. @pallasathena92 merged the one-commit PR.
If an override in your own code left an old source attached, reply on X.
]]>A job groups database targets and the collectors to run against them. The jobs[] query parameter selects which configured jobs to scrape. It can appear more than once. With jobs named a and b, /metrics?jobs[]=a asks for a; omitting the filter selects all targets.
The failure needs overlapping requests. One scrape asks for a while another asks for b on the same exporter. This can happen when separate Prometheus scrape jobs select different jobs from that exporter. A response for b could include scrape_errors_total series labelled job="a". Less often, the wrong job’s up and scrape_duration_seconds series appeared too. My previous sql_exporter post covered the warning fixed in #1115. Here, the problem reached the response’s metrics.
In cmd/sql_exporter/main.go, the exporter is constructed and passed into ExporterHandlerFor. The handler lives in cmd/sql_exporter/promhttp.go. Before my change, it read req.URL.Query()["jobs[]"] and called exporter.SetJobFilters(jobFilters) on that shared exporter.
The next step looked reassuring: exporter.WithContext(ctx) supplied a copy for gathering. In exporter.go, the interface describes that copy as single use. But its constructor copies jobFilters from the original. Another request could change that field after the setter and before the copy. The copy then carried the wrong selection into filteredTargets, which chooses the database targets to collect.
There was another read later. The handler gathered metrics from the copy and the registry, then called exporter.FilterScrapeErrorsTotal(mfs) on the original exporter. That method filters the error counter’s series by their job labels. Another request could replace the filter during gathering, so even a request that collected the right targets could select the wrong error series at the end.
I found two chances to use another request’s input: before copying, and before filtering the error counter. The setter also wrote jobFilters without a lock while other requests read it. The PR’s race-detector run reported a data race.
PR #433 added jobs[] selection and the setter-before-copy order. #915 added the later error-counter filter. #916 made target selection return a filtered slice instead of replacing the exporter’s target slice.
Returning a filtered target slice did not make the handler’s jobFilters request-local. The handler still wrote that field on the shared object and read it again later.
I took the copy before setting the filter, then used that same copy at both consumers. These are the changed calls, with intervening code omitted:
+exp := exporter.WithContext(ctx)
-if err := exporter.SetJobFilters(jobFilters); err != nil {
+if err := exp.SetJobFilters(jobFilters); err != nil {
-gatherer := prometheus.Gatherers{exporter.WithContext(ctx), registry}
+gatherer := prometheus.Gatherers{exp, registry}
-mfs = exporter.FilterScrapeErrorsTotal(mfs)
+mfs = exp.FilterScrapeErrorsTotal(mfs)
The handler now sets the request’s selection on exp, gathers from exp, and filters the error counter with exp. In multi-job mode, an unknown job name still returns a 400 response; single target mode still ignores job filters. WithContext makes a shallow copy, but SetJobFilters assigns the filter field on that copy rather than changing a shared slice’s contents.
I added TestExporterHandlerJobFilters in cmd/sql_exporter/promhttp_test.go. It runs two goroutines, each making up to 10 requests through the same handler and stopping early if a check fails. One selects a, the other b. The database targets are deliberately unreachable: up and scrape_errors_total still carry job labels, so the test can check selection without a working database.
Each response must contain its own job and contain none of the other job. Checking only for foreign series would let an empty error response pass.
My PR records the local before/after results: before the fix, 20 of 20 runs failed; after it, 20 of 20 passed. The race-detector run with -count=5 failed before and passed after. The full suite, its race-detector run, formatting, and vet also passed after the fix. These are the checks recorded with the PR, rather than fresh runs for this post.
@burningalchemist approved with “Great catch! Thanks a lot! 👏” and merged 1 minute and 36 seconds later. There were no inline review comments or issue-thread comments. The PR was one commit.
If request input escaped into another response in your own code, reply on X.
]]>download_artifact in python/aibrix/aibrix/runtime/artifact_service.py has two ways to return a path. On the first call it runs the downloader and returns the path the downloader reports. Once a .aibrix_download_complete marker exists, it returns local_path, the directory it gave the downloader. For S3, GCS, TOS and HTTP those are the same directory. For Hugging Face they were not.
The Hugging Face downloader called snapshot_download(repo_id=..., cache_dir=local_path). With cache_dir, huggingface_hub writes the files under local_path/models--<org>--<name>/snapshots/<revision>/ and returns that snapshot directory. So the first load gave the engine the snapshot directory, and every later load gave it local_path, which has no adapter_config.json at its root. Later loads happen: a retry after an engine error, a repeated load request, a restart that keeps the adapter volume, an unload with cleanup_local=false.
cache_dir means “keep a cache here”. The option for “put the files in this directory” is local_dir, and the repo’s model downloader in aibrix/downloader/huggingface.py already used it. The whole change to downloaders.py:
- # Download model/adapter
+ # Download model/adapter. Use local_dir so the files land directly
+ # in local_path, like the other downloaders. With cache_dir they
+ # would sit under models--<org>--<name>/snapshots/<revision>/, a
+ # path the artifact service does not return once the completion
+ # marker exists.
download_path = snapshot_download(
repo_id=repo_id,
- cache_dir=local_path,
+ local_dir=local_path,
token=token,
)
huggingface_hub 0.24.x with a small public repo. cache_dir=/x/a returned /x/a/models--<org>--<name>/snapshots/<sha>, and local_dir=/x/b returned /x/b with config.json directly inside.download_artifact twice against the real Hub. Both calls returned the same directory containing the file.snapshot_download that follows both layouts. One test checks that the files land in the adapter’s directory. The other calls download_artifact twice and compares the results. Before the change both failed: one got the snapshot path where it expected .../adapter, the other got a different path on the reload. After it, 2 passed. The review also ran all of tests/runtime with no failures.None of my checks involved an inference engine. That an engine would reject a directory without adapter_config.json at its root is an inference from the layout. The review listed a real engine loading an adapter among the things it had not run either.
A flag to add. A review bot suggested adding local_dir_use_symlinks=False, on the worry that local_dir leaves symlinks into a cache that breaks in containers. I replied that in the version poetry.lock pins, 0.24.7, the parameter is deprecated and ignored: hf_hub_download only warns, and the files are written into local_path itself. Adding the flag would add the warning and nothing else. The review, posted from the varungup90 account, called that reply accurate and said it had read the 0.24.7 wheel to check. The repo’s model downloader does pass that flag, so the suggestion matched what the code next door does. The review agreed that leaving it out was right.
Two strings for one directory. The review’s first optional follow-up: snapshot_download(local_dir=...) returns the real path of local_dir, while the marker path returns local_path as given. They are the same directory, but if the adapters directory is reached through a symlink they are different strings. The review suggested returning local_path from both branches. I agreed and offered to add it to the PR or leave it for later, as the review had proposed. The merged PR has its one commit and no realpath change.
What the fix does not repair. The second follow-up: a directory downloaded under the old layout, with its marker, keeps returning local_path. If a deployment keeps adapters on a persistent volume, the review wrote, this “will bite after the upgrade”, and a release note would help. My description already had a Compatibility section saying this. In my reply I added that nothing regresses, because the marker path returned local_path before the change too, so such an adapter was already broken on its first reload. It needs cleanup_local, or removing the directory, once.
The review also reproduced the failure with the base file and my tests. It checked that the resume files a download leaves under .cache/huggingface/ cannot make a failed download look complete, because the content check runs only after the downloader returns successfully.
When the review looked, the Python test jobs and image builds were still queued. By the time I commented, they had passed. The review recommended approval but was posted as a comment, and @varungup90 merged the PR 30 minutes after my CI comment.
cache_dir is not an output directory. Check what the call returns and where the files land, and read the source of the version your lockfile pins, not the latest documentation.If a cached path in your own code came back different the second time, reply on X.
]]>make verify-docs and make verify-go-modules in cri-tools rewrote files in your working tree, then failed if git status showed anything dirty. I changed them so they only check (PR #2212, merged 2026-10-16). The PR fixes issue #2197, which @SergeyKanzhelev filed, and it follows the plan @saschagrunert wrote in a comment there. @saschagrunert then reviewed the PR and raised four points. All four changed the merged scripts.
The old hack/verify-go-modules.sh ran the commands that fix the tree, then asked git whether anything was left over:
go mod tidy && go mod vendor && go mod verify
# Test if we can resolve all go modules
go list -mod=readonly -m all
STATUS=$(git status --porcelain)
if [[ -z $STATUS ]]; then
echo "tree is clean"
else
echo "tree is dirty, please commit all changes"
echo ""
echo "$STATUS"
exit 1
fi
verify-docs.sh had the same shape with go-md2man and docs/crictl.1. So a verify run edited your files, and it could not pass while you had uncommitted work. The issue puts it this way: “verify commands are both disruptive (override in-place) and cannot run on not-committed files.”
@saschagrunert’s comment on the issue chose a narrow version of the third option in the issue. Build into a temp location and diff against the tree, without copying the tree. For modules that means go mod tidy -diff, go mod vendor -o "$tmp" plus a diff against vendor, and go mod verify. For docs it means running the doc tool into a temp file. Then add a make docs target next to make vendor, so that “verify-* only checks and docs/vendor write files”.
The comment also warned that a bare entry in .gitignore matched a vendored file, so that file was not in git and a plain diff -r against vendor/ would report it. The PR anchors that entry with a leading slash. The comment suggested committing the vendored file as well. At the version in go.mod the dependency ships no such file, so there was nothing to commit, and the anchor only stops a later bump from dropping one silently.
I wrote the PR to that plan. The tree stayed untouched, and in the PR description I wrote that I had checked a clean tree passes, that an edited docs/crictl.md, an untidy go.mod and an edited vendor/modules.txt each fail with a hint, and that the tree is byte-identical after every run. The description lists no run from another directory and none with an ignored file in the tree.
The review was on commit b8c6f16. The PR head that was merged is 9d7078f; the PR is still one commit.
Run from anywhere. My verify-go-modules.sh ran diff -ru vendor "$TMP_DIR". @saschagrunert pointed out that vendor is relative to the current directory, so cd hack && ./verify-go-modules.sh fails with diff: vendor: No such file or directory and a misleading “run make vendor” hint. The old git status check worked from any subdirectory, so I had made it worse. My verify-docs.sh already had the fix, and I had not copied it into this script. The review comment carried a suggestion block, and the merged script has exactly those lines:
# Root of the repository
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "${REPO_ROOT}"
Compare what git will commit. This comment is the one I learned most from. A diff against vendor/ compares the files on disk, including files git ignores. @saschagrunert listed where that bites: a dependency bump can vendor a file matched by a root rule (*.iml, *~, .*/, release-notes.md) or by one of the 28 vendored .gitignore files. Their examples were otel ignoring gen/, wazero ignoring build and dist, and logrus ignoring vendor. Then make vendor writes the file and it never gets committed. In the reviewer’s words: “The check then passes locally and fails in CI with a hint that make vendor can’t fix.” Anchoring the one rule the plan comment named covers one instance. A root !/vendor/** negation would not cover the rest, because the vendored .gitignore files take precedence. The reviewer’s answer was a check after the diff, and the merged script contains it as written:
ignored="$(cd "$TMP_DIR" && find . -type f | sed 's|^\./|vendor/|' | git -C "$REPO_ROOT" check-ignore --no-index --stdin || true)"
if [[ -n $ignored ]]; then
echo "vendored files are gitignored and would not be committed:" >&2
echo "$ignored" >&2
exit 1
fi
I did not have this idea. It is @saschagrunert’s.
Do not bury the hint. On vendor drift, diff -ru prints a unified diff of every changed vendored file. @saschagrunert measured it: bumping golang.org/x/net without re-vendoring gives about 1000 lines with diff -ru and 9 with diff -rq. “Since the fix is always make vendor, file names should be enough.” The merged script uses diff -rq. verify-docs.sh still uses diff -u, because it compares one file.
One copy of the doc-tool call. In my first version the Makefile docs target and verify-docs.sh each had their own go run -C hack/tools github.com/cpuguy83/go-md2man/v2 ... line. The review said defining it once keeps the two from drifting, and offered two ways: a variable, or “having the script take the output path and make docs call it”. I took the second. The merged Makefile has CRICTL_MAN ?= $(CURDIR)/docs/crictl.1, and the script runs:
make -s docs CRICTL_MAN="$TMP_DIR/crictl.1"
@saschagrunert approved with “Thanks!” and the merge bot merged the PR the same day.
None of the four points is covered by the checks the description lists. Each of those asks “does this fail when it should”. The review asked other questions. Where is the script run from? What will CI see that my disk does not? How much output does a failure produce? Is the doc-tool call written down once?
The second question is the hard one. A verify script that reads the working tree answers “is this tree consistent”, and CI needs the answer “is this commit consistent”. Those differ whenever the working tree holds anything git ignores. The same split shows up in tests. A test that reads the machine it runs on, such as a global git config, passes on the machine where it was written and fails on another. I met that in a git-stats change for oh-my-openagent, where the CI run on Windows failed on a config path.
git status the tree before and after a run. A verify script that changes anything has failed.mktemp -d and remove it in a trap ... EXIT. Never write into the tree and diff afterwards.git check-ignore --no-index on the list of files the script wrote.git status check worked from any directory, and the replacement lost that.If a verify script of yours passed on your machine and failed in CI, reply on X.
]]>Records() call, only to read the name of the result. I sent the fix (PR #6805, merged 2026-10-16). In the PR’s benchmark, with encryption on and an in-memory provider, one Records() call over 1,000 endpoints went from 512.59 ms to 23.47 ms and allocated 96.06% less. Then @mloiseleur found a case that my version, and the code it replaced, both got wrong, and sent a follow-up commit. That commit is part of the merged PR.
The TXT registry keeps an ownership record for each DNS record it manages: a TXT record whose value holds labels such as the owner ID. TXTRegistry.Records() reads the provider’s records and attaches those labels to the endpoints. It also has a migration block, marked TODO: remove this migration logic in some future release, for TXT records created before the current naming format. For every record this instance owns, the block checks that the current-format ownership TXT exists. If it does not, it sets txt/force-update on the endpoint, so the next apply creates it.
To find the missing ones, the old code called generateTXTRecord for every owned endpoint. That builds the whole TXT endpoint: it serializes the labels and, with --txt-encrypt-enabled, encrypts them with AES-GCM. In endpoint/crypto.go, EncryptText gzips the text before it seals it. The caller then read DNSName from the result and dropped the rest.
Records() runs on every reconcile loop, so a deployment with encryption on pays this on every loop, for every owned record. The --txt-cache-interval flag makes Records() return a cached result inside the interval, but it is off by default. I measured with it off.
I added BenchmarkTXTRegistryRecords. One iteration is one Records() call in the steady state of a loop. An in-memory provider holds N endpoints, alternating A and CNAME. Every one is owned by the registry and already has its ownership TXT. I ran 1,000 and 10,000 endpoints, with encryption on and off, and compared master with the branch using benchstat:
go test ./registry/txt -run '^$' -bench BenchmarkTXTRegistryRecords -benchmem -count=10
Time and allocation per Records() call, from the PR description (n=10, p=0.000 for every difference):
| Input | Time, before to after | Allocated, before to after |
|---|---|---|
| encryption on, 1,000 endpoints | 512.59 ms to 23.47 ms (-95.42%) | 1137.14 MiB to 44.81 MiB (-96.06%) |
| encryption on, 10,000 endpoints | 3076.0 ms to 244.3 ms (-92.06%) | 11369.5 MiB to 447.3 MiB (-96.07%) |
| encryption off, 1,000 endpoints | 8.356 ms to 5.552 ms (-33.56%) | 3.071 MiB to 2.590 MiB (-15.68%) |
| encryption off, 10,000 endpoints | 90.86 ms to 64.48 ms (-29.03%) | 29.86 MiB to 25.05 MiB (-16.11%) |
The benchmark checks its own fixture before timing. It reads Records() once and fails if the count is wrong, a record is not owned by the registry, or any record already carries the force-update flag.
Two limits. The provider is in memory, so there is no API latency. And the ratios are for this one step, not for a whole reconcile loop.
Compute the name of the ownership record and skip the value. This is the helper on my branch after the first review round, and the call site:
// txtRecordName returns the name generateTXTRecord gives the ownership TXT record of r, without
// rendering its value. It returns "" when no such record can be generated.
func (im *TXTRegistry) txtRecordName(r *endpoint.Endpoint) string {
txt := endpoint.NewEndpoint(im.mapper.ToTXTName(r.DNSName, r.RecordType), endpoint.RecordTypeTXT)
if txt == nil {
return ""
}
return txt.DNSName
}
if txtName := im.txtRecordName(ep); txtName != "" && !txtRecordsSet.Has(txtName) {
ep.WithProviderSpecific(providerSpecificForceUpdate, "true")
}
endpoint.NewEndpoint trims the trailing dot and returns nil for a label over 63 characters, so the name matches what generateTXTRecord gives. I added two tests. One checks that the computed name equals the generated record’s name for prefix, suffix, wildcard and record type variants. The other checks that the force-update flag is still set for a missing TXT record with encryption on.
A return value. @ivankatliarchuk’s review on 2026-10-14 was marked as changes requested and had one inline comment, which began “Just a nit pick.” It said: “An empty string is already a reliable sentinel, there is no other logic empty - false, non-empty - true.” My helper had returned (string, bool). I pushed the change about an hour later; that is the form above. That reviewer commented /lgtm on 2026-10-15.
A case I missed. On 2026-10-16, @mloiseleur wrote: “Thanks, the speedup looks good. I opened spicysharkco/external-dns#1 against your branch with a fix for routed records and a few cleanups. Wdyt?” Their pull request, in my fork, had one commit, 5dfe52a. It made two changes.
The first is a bug fix. My check looked names up in txtRecordsSet, a set of TXT names. The old code had the same set. Take an A record that exists twice under one name, as two routed records with set identifiers set-1 and set-2. With the default naming, each has a legacy ownership TXT named foo.example.org. Only set-1 also has the current-format TXT, a-foo.example.org. The lookup “is there a TXT named a-foo.example.org?” answers yes because of set-1’s record. So set-2 is never flagged, and this migration does not create its current-format TXT.
The fix looks the name up in existingTXTs, a map the registry already filled in the same loop. It keys on name and set identifier, and the commit deletes my duplicate set. The lookup became:
if txtName := im.txtRecordName(ep); txtName != "" && !im.existingTXTs.has(txtName, ep.SetIdentifier) {
The second change makes generateTXTRecordWithFilter build its name with txtRecordName. Before, both functions had their own ToTXTName call, and only my test said they agreed. txtRecordName also checks label lengths directly instead of allocating an Endpoint. The commit message adds that this “stops Records() from logging over-long names on every call”.
The commit also grew the force-update test from two cases to five, including routed records and an alias A record with a legacy cname- TXT. I fast-forwarded my branch to 5dfe52a at 21:19 UTC. @mloiseleur approved the PR at 21:22, and the merge bot merged it at 21:30.
My tests could not see this. Both of my force-update cases used one A record with no set identifier, so a lookup keyed on name alone passed them. My PR description says the set of flagged records is “unchanged”. That was true of my first version, which kept master’s blind spot. It is not true of the merged code: for routed records it flags a record that master did not. The description still says “unchanged”.
If a lookup of yours turned out to be keyed on less than it should have been, reply on X.
]]>[FILE CHANGES SUMMARY] block after a task finishes. It lists modified, created and deleted files with their line counts. The block was wrong for renamed files, for non-ASCII names and for some untracked files. I fixed it by reading git’s output NUL-separated (PR #10133, merged 2026-10-15 as 6c161a1).
The parsing was not what review changed. The first Windows CI run failed on the test setup I had written, and the committed bundle went stale. @code-yeongyu diagnosed both. This post is about those two.
collectGitDiffStats ran three commands without -z: git diff --numstat, git status --porcelain and git ls-files --others. It then joined the three lists by path. Git’s line output did not match how the code joined the three lists. The PR description lists the results:
old_name.txt => new_name.txt, or d/{sub => sub2}/x.txt when only a directory changed, instead of its new name."caf\303\251.ts". An untracked one also lost its line count (+0), because the quoted name could not be read from disk.?? newdir/ entry, so its status entry did not match the per-file path.An excerpt of the before and after output from the PR description:
Before (dev):
Modified files:
d/{sub => sub2}/x.txt (+0, -0)
old_name.txt => new_name.txt (+1, -0)
newdir/one.txt (+1, -0)
newdir/two.txt (+2, -0)
with space.ts (+2, -0)
Created files:
"caf\303\251.ts" (+0)
After (this branch):
Modified files:
d/sub2/x.txt (+0, -0)
new_name.txt (+1, -0)
Created files:
café.ts (+3)
newdir/one.txt (+1)
newdir/two.txt (+2)
with space.ts (+2)
The PR follows #5690, which mapped R status lines to the destination path in the line parser. That fixed the status side. The numstat side still printed old => new.
All three commands now read NUL-separated, with rename detection pinned on so diff.renames and status.renames in the repository config cannot change the result:
git diff --numstat -z -M HEAD
git status --porcelain -z --find-renames --untracked-files=all
git ls-files -z --others --exclude-standard
--untracked-files=all gives every untracked file its own status entry, which fixes the new-directory case. The untracked line counts the collector builds by hand are NUL-terminated too.
A rename is the awkward record. With -z, numstat prints added\tremoved\t\0old\0new\0: the path field is empty and the two names follow as separate fields. The new parser handles that with one extra branch:
let path = field.slice(secondTab + 1)
if (path === "") {
index += 2
path = fields[index]
}
A rename is reported once, under the destination. A binary file shows - for both counts, and toCount turns that into 0. The status parser does the same job for porcelain records: for a rename or copy it skips the source field, and it maps a copy destination to “added”, because git diff -M without -C reports a copy as a new file.
One consequence I had to handle. -z output is never quoted, so a file name that holds a newline would print as extra lines inside the summary. So formatFileChanges now prints a path with a control character, U+2028 or U+2029 JSON-quoted with escapes, and leaves every other path unchanged. café.ts, 한글.md and with space.ts stay as they are.
The test file, collect-git-diff-stats.real-git.test.ts, has 12 cases that run against throwaway repositories with real git. They cover untracked names with spaces and non-ASCII characters, a staged rename with edits, a path with a tab, untracked files in a new directory, an intent-to-add rename, a copy with status.renames=copies, a binary rename, and diff.renames and status.renames set to false.
I ran the new tests against the unmodified dev sources and against the branch. On dev: 2 pass, 18 fail, and all 12 real-git tests fail. On the branch: 20 pass, 0 fail. The 20 are the 12 real-git tests and 8 formatter tests, and the 2 that pass on dev are formatter tests that check plain paths stay unchanged.
The setup keeps each test away from the machine’s git configuration. I set GIT_CONFIG_NOSYSTEM, author and committer names, and I pointed GIT_CONFIG_GLOBAL at os.devNull:
import { devNull, tmpdir } from "node:os"
const HERMETIC_GIT_ENV: Record<string, string> = {
GIT_CONFIG_GLOBAL: devNull,
GIT_CONFIG_NOSYSTEM: "1",
// author and committer names ...
}
CI reported two failing jobs, and only the Windows test job failed on this setup.
@code-yeongyu approved the workflow runs, and wrote that they “read the diff before approving CI”. They called the change “a careful fix” and said the real-git test “makes it believable”. Then CI ran, and they reported two failing jobs, “both small and both in this PR”.
The Windows git init failure. Every case in the real-git test file failed at git init on windows-latest with fatal: unable to access '\\.\nul': Invalid argument. In their words, GIT_CONFIG_GLOBAL: devNull “resolves to \\.\nul on Windows, and git can’t open that as a config file”. They proposed “Pointing GIT_CONFIG_GLOBAL at an empty file inside the temp dir (written in beforeEach)”, which “works on every OS and keeps the test hermetic”. That was their idea. I implemented it in 4d1980e:
root = mkdtempSync(join(tmpdir(), "omo-diff-stats-"))
writeFileSync(join(root, "gitconfig"), "")
process.env.GIT_CONFIG_GLOBAL = join(root, "gitconfig")
repo = join(root, "repo")
mkdirSync(repo)
afterEach now removes root, which holds both the config file and the repository.
The stale bundle. The second failing job was senpi-compatibility (ubuntu-latest): omo-senpi extension build is not current: stale-output for packages/omo-senpi/plugin/extensions/omo-task.js. @code-yeongyu explained that packages/utils is bundled into the committed extensions, so the bundle needs rebuilding, and that CI checks it “on Linux with node 24 and bun 1.4.2”, so a bundle built on another OS may not match byte for byte. They also noted the branch was about 100 commits behind dev, and that merging dev before rebuilding “avoids a second round”. My change added exports to packages/utils/src/git-worktree/index.ts, and the bundle carries a digest of every module the bundler read, so the digest changed. The bundled code did not.
An offer I did not need. They wrote that if I would rather not deal with the Linux bundle step, they could push the merge, the test fix and the rebuilt bundle on top of my branch, “keeping your commits and credit”. I did the work myself and reported it in a comment on the PR.
I did the steps in the opposite order to the advice. I rebuilt the bundle in the same commit as the Windows fix, before merging dev. The merge then conflicted in the bundle and in CHANGELOG.md. I resolved it and rebuilt the bundle on Linux in 6b67ea7.
dev kept moving while the PR waited. I merged it into the branch three more times before the PR was merged, in 6b9b5b9, d547fbf and 78ca1f0. Two of the four merges conflicted in CHANGELOG.md and three in omo-task.js, and the bundle was rebuilt on Linux after three of them. The final diff of omo-task.js is one line added and one removed.
The review result @code-yeongyu posted for e16a360 was “approved, with no blocking findings”. It said the -z parsing “handles renames (the destination path is kept), binary - counts and paths with tabs”. It said the real-git test “is hermetic: it uses a per-test empty GIT_CONFIG_GLOBAL and GIT_CONFIG_NOSYSTEM”. And it said that in the rebuilt omo-task.js “only the header digest changed; the bundled code is byte-identical”. The PR was merged on 2026-10-15, after the last dev merge and rebuild.
git status --find-renames needs git 2.18, the release that added status.renames. On older git the status call fails, the existing catch returns an empty list, and the summary reads “No file changes detected.” I wrote that in the PR description and added that I had not tested it on old git.
Two checks to make before the first push.
Anything the platform supplies is not a constant. os.devNull is a path, and it is a different path on Windows. A test that needs “an empty file” should make one in its own temp directory. This costs three lines, and it removes a platform branch.
CI is more than the test suite. I ran the tests I had written and the type check. The senpi-compatibility job checks a built file that lives in the repository and depends on files I had touched. Read the workflow list before pushing, find every job that compares a committed artifact with a rebuild, and run its check locally.
For committed build output, merge the target branch first, rebuild last, and make the rebuild a single command, because the target branch moves while the PR waits.
If you have been bitten by \\.\nul or a stale bundle, reply on X.
bool and a method, and gives a channel a buffer of 2. @bheemareddy reported the bug in issue #14787, with the cause, a standalone reproducer and a sync.Once fix.
It is Proxy(lconn, rconn io.ReadWriteCloser) error in pkg/daemons/control/proxy/proxy.go. It copies bytes both ways between two connections, one goroutine and one 32 KiB buffer per direction. The caller is TunnelServer.serveConnect in pkg/daemons/control/tunnel.go. It answers HTTP CONNECT: dial the backend, hijack the client connection, call proxy.Proxy. It serves the apiserver egress selector, the tunnel the apiserver uses to reach kubelets (k3s docs) and, in the reporter’s setup, aggregated APIServices.
The issue has a goroutine dump from that server. Every leaked goroutine had the same stack, parked in chan send; one showed as goroutine 17335244 [chan send, 467 minutes]:. The count grew by roughly 11 per hour. The server log showed lines like Proxy error: write failed: ... write: connection reset by peer. I did not find the leak; these numbers are the report’s.
The code before the fix (lines 26-40):
go p.pipe(p.lconn, p.rconn)
go p.pipe(p.rconn, p.lconn)
return <-p.errc
}
func (p *proxy) err(err error) {
if p.done {
return
}
if !errors.Is(err, io.EOF) {
logrus.Warnf("Proxy error: %v", err)
}
p.done = true
p.errc <- err
}
errc was unbuffered and Proxy receives from it once. done was meant to keep a second goroutine from sending, but it is a plain bool written by both goroutines. Both can pass if p.done before either sets it. The Warnf call sits in that gap.
The goroutine that leaks is not stuck in Read or Write. Those already failed. It is parked on p.errc <- err. The Go spec says “A send on an unbuffered channel can proceed if a receiver is ready.” Proxy has returned, so none ever is. The deferred Close calls wake a goroutine blocked in Read or Write, not one blocked on a channel. Per the PR description, it holds its buffer and both connections.
From the merged diff of pkg/daemons/control/proxy/proxy.go (excerpt):
- errc: make(chan error),
+ errc: make(chan error, 2), // one send per pipe goroutine, so neither can block
}
defer p.rconn.Close()
defer p.lconn.Close()
go p.pipe(p.lconn, p.rconn)
go p.pipe(p.rconn, p.lconn)
- return <-p.errc
-}
-
-func (p *proxy) err(err error) {
- if p.done {
- return
- }
- if !errors.Is(err, io.EOF) {
+ err := <-p.errc
+ if !isConnTeardown(err) {
logrus.Warnf("Proxy error: %v", err)
}
- p.done = true
- p.errc <- err
+ return err
+}
+
+// isConnTeardown reports errors that are expected when one side closes or resets the connection.
+func isConnTeardown(err error) bool {
+ return errors.Is(err, io.EOF) ||
+ errors.Is(err, net.ErrClosed) ||
+ errors.Is(err, syscall.ECONNRESET) ||
+ errors.Is(err, syscall.EPIPE)
}
In pipe, p.err(errors.WithMessage(err, "read failed")) became p.errc <- err, and the same for the write. Two slots for two senders means no send can block, whatever Proxy does. The guard is gone, so the race is gone. I kept “return on the first error” instead of waiting for both directions as remotedialer does. Per the PR description, that way Proxy cannot hang on a connection whose Close does not unblock a pending Read.
isConnTeardown. Their own PR, #14810, took the remotedialer approach. They called this one “smaller fix and easily back portable”, and @brandond went with it.ECONNRESET and EPIPE and suggested dropping the errors.WithMessage wrap. They gave no reason. Their isConnTeardown used a type assertion, err.(syscall.Errno).errors.As, because a TCP error is not a bare syscall.Errno. That part was my suggestion, not the reviewer’s. On a loopback reset I got a *net.OpError, the assertion gave false, and errors.Is(err, syscall.ECONNRESET) gave true.errors.Is, and it also keeps net.ErrClosed, which I had dropped.Test_UnitProxyBothDirectionsFail uses a failConn whose Read and Write always fail. Both directions fail together:
conn := failConn{errors.New("i/o error")}
before := runtime.NumGoroutine()
for range 10000 {
_ = Proxy(conn, conn)
}
g.Eventually(runtime.NumGoroutine, 5*time.Second).Should(gomega.BeNumerically("<=", before))
Eventually is there because the second goroutine exits just after Proxy returns. I ran this test against the old proxy.go. It failed five runs in five, leaving roughly 900 to 1,900 goroutines against a start of 2, depending on the machine, and -race printed WARNING: DATA RACE. Against the merged code, go test -race -count=20 passed. With GOMAXPROCS=1 the old code passed, as the PR warns, so run leak tests on more than one core.
The old and new shape in miniature. It makes 10,000 calls on failing connections and counts goroutines:
package main
import (
"errors"
"fmt"
"io"
"os"
"runtime"
"time"
)
// failConn fails every Read and Write, like a connection that was reset.
type failConn struct{}
func (failConn) Read([]byte) (int, error) { return 0, errors.New("reset") }
func (failConn) Write([]byte) (int, error) { return 0, errors.New("reset") }
func pipe(src, dst io.ReadWriter, report func(error)) {
buf := make([]byte, 1<<15)
for {
n, err := src.Read(buf)
if err == nil {
_, err = dst.Write(buf[:n])
}
if err != nil {
report(err)
return
}
}
}
// proxy returns the first error from either direction. With old set it has
// the pre-fix shape: an unbuffered channel and a plain bool as the guard.
func proxy(old bool) error {
a, b := failConn{}, failConn{}
errc := make(chan error, 2)
report := func(err error) { errc <- err }
if old {
errc = make(chan error)
done := false
report = func(err error) {
if done {
return
}
done = true
errc <- err
}
}
go pipe(a, b, report)
go pipe(b, a, report)
return <-errc
}
func main() {
old := len(os.Args) > 1 && os.Args[1] == "old"
before := runtime.NumGoroutine()
for range 10000 {
proxy(old)
}
time.Sleep(500 * time.Millisecond) // let goroutines that can exit, exit
fmt.Printf("before=%d after=%d\n", before, runtime.NumGoroutine())
}
On Go 1.24.5, go run . new printed before=1 after=1 in 15 of 15 runs. go run . old left a different count every run: 83 to 117 in one batch of 15, 12 to 64 in another. Any count above 1 is the leak. My copy leaks less than the real test. I did not test why; it omits the logging and error wrapping. go run -race . old reports the race.
sync.Once.runtime.NumGoroutine. Use -race and several cores.net/http/httputil.ReverseProxy handles protocol upgrades with the same shape (source): errc := make(chan error, 1), two copier goroutines, each sending exactly once. The parent reads one value, and a second only if the first was nil. A buffer of 1 is enough because the parent always reads at least once. The k3s buffer of 2 does not depend on that. The copiers also pass on a half-close with CloseWrite when the connection supports it.
rancher/remotedialer’s pipe uses no channel: a sync.WaitGroup, a closePipe that closes both ends, then wg.Wait() (source). It waits for both directions. #14810 followed it.
If you have chased a leak like this one, reply on X.
]]>