The context engine is the product

We are publishing the retrieval results behind Delphi, the corpora they were measured on, and the per-query traces they came from. On Agent Retrieval Bench v2, the shipped build returns more of the answer set than the leading hosted context engine and does it 5.7 times faster on a laptop. It does not lead on every metric, and this piece is specific about which.

Results

Delphi is an open-source context engine: it indexes repositories, documentation, papers, and datasets, and answers an agent's question with the files that answer it. The number that matters for an agent is whether the file it needs is in the window it gets, which is what these measure.

Recall@20 is the difference that survives: 0.107 in Delphi's favour, 95% interval [0.008, 0.207], winning 16 cases to 6. Latency is the other: 5.6s against 31.8s.

MRR reads 0.241 against 0.301, which looks like a loss and is not one we can claim: the interval runs from -0.142 to 0.020 and the cases split 18 to 24 with 18 ties. At sixty cases that is a coin flip. We say so rather than reporting it either way.¹

The two engines also answer differently. The comparator returns about 8 files per query and Delphi returns 20. Short lists concentrate on rank one, long lists cover more of the answer. Which you want depends on whether your agent gets one shot or can keep reading.

Against the benchmark's own published baselines on the development split, scored by the same code:

Delphi does not lead Recall@20 outright either. Plain grep gets 0.578. We would rather print that than drop the column. If your engine finds the right file somewhere in twenty results, you have not beaten grep -r. The argument has to be won at the top of the list, and that is where we still have work to do.

A corpus orthogonal to itself

The symptom was that semantic queries returned nonsense. We asked a 68-repository corpus about TLS configuration in a gRPC proxy and got back an interval tree, a systemd journal wrapper, and some file-locking utilities. Exact symbol lookups worked perfectly. Anything that went through the embedding did not.

Two embedding models can produce vectors of the same width. When they do, pgvector will compute a cosine between them without complaining. The query succeeds, results come back ranked, and the ranking is noise, because the two spaces have nothing to do with each other. There is no error to catch. The only thing wrong is the answer.

The check that found it

The check is embarrassingly simple. Take a chunk out of the index. Embed its own content with whatever model answers queries today. Compare that vector against the one already stored. If the two sides agree, a chunk has to be nearly identical to itself.

It scored 0.0101. That is orthogonal, the number you get from two random vectors. Pointing the query path at text-embedding-3-small, the model that had actually built the index, moved the same comparison to 0.943. The vector branch was weighted 0.5, the largest weight in the pipeline. Half the ranking signal had been random for the entire evaluation.

The database had known all along. repositories.embedding_model records which model indexed each repository; it simply was never compared against the model answering queries. Delphi now makes that comparison on every search and reports it on /backend-health, because an engine that returns confident nonsense is worse than one that returns an error.

Comparing incomparable numbers

With the embedding fixed, a second problem surfaced. Delphi fans a query out across vector, BM25, symbol, path, and trigram branches, then fuses the results. Each branch normalised its own scores by dividing by that branch's top score. So the best hit of every branch got pinned to exactly 1.0, however bad it was.

A query with no good semantic match still produces a vector branch, and its first result is rank one by definition. Rescaled, that least-bad hit became a perfect 1.0, and at weight 0.5 it outranked chunks that three branches independently agreed on. The fingerprint was a suspiciously round 0.5000 at the top of result lists: weight times a manufactured perfect score.

Reciprocal rank fusion exists for this reason. Ranks compare across branches. Raw scores do not. A cosine and a ts_rank_cd were never the same unit. Fusion now scores position instead of magnitude.

The path nobody indexed

The third problem was the most mundane. BM25 indexes chunk content. Nothing indexed file_path. A query naming a file could not retrieve that file's neighbours lexically at all. Searching for etcd_grpcproxy_test returned fileutil.go, because the filename existed nowhere in the searchable text.

This matters because agents anchor on paths constantly: “what tests cover grpc_proxy.go”, “why did tokens.py change”. Delphi now has a path-affinity branch that matches on separator-stripped lowercase, so an anchor of grpc_proxy reaches etcd_grpcproxy_test.go, which underscore-sensitive comparison misses. Results are capped per directory, because a stem like grpc_proxy matches thirty sibling files at a perfect score and would otherwise fill the branch before the one test in tests/e2e/ ever appeared.

The reranker that never ran

Delphi had a cross-encoder reranker. It had never once executed during the evaluation. The model loaded lazily on first query, the load was a multi-hundred-megabyte download, and the call site wrapped it in a try/except that fell back silently to fused ranking. Every query took the fallback.

Warming it at startup and reporting readiness on the health endpoint fixed the availability problem and produced a genuinely surprising result once we could measure it.

The 22M-parameter ms-marco-MiniLM model beats 278M-parameter bge-reranker-base on every metric while running about seven times faster. Preferring the larger, code-aware model, which is what the default did, cost latency and quality at the same time. Reranking depth behaves the same way: at a window of 100 with the cross-encoder deciding the order outright, Recall@20 collapses to 0.444, because the model confidently promotes plausible-looking files from the tail. Blending it over the fused score, shallowly, is what makes it useful.

What each fix was worth

The first row is what the previous evaluation actually measured. Almost the entire improvement comes from the embedding fix; rank fusion and the path branch add a little on top; the cross-encoder buys the top of the list. It would be more flattering to present this as four clever retrieval improvements. It was one configuration bug and three modest engineering fixes, and the honest version is more useful to anyone running a similar stack.

What we can claim

On Agent Retrieval Bench v2, 75 development cases: Delphi leads every published baseline on MRR and Recall@5, and trails grep on Recall@20. On the held-out split, all 220 positive cases with every corpus provisioned and 0 failed queries, it scores 0.309 MRR, 0.442 Recall@5 and 0.582 Recall@20 at 5.57s mean latency. Those come out slightly ahead of the split the pipeline was tuned on, which is the direction you want: no sign of having fit the tuning set.

The per-workflow spread says more than the average does. Retrieval is close to solved when the query names things that exist in the code, and barely works when it does not.

A failure trace hands the retriever real symbols, real file names, real stack frames, and Delphi finds the root-cause file in the top five 76% of the time. A review comment hands it English, something like “this should probably be extracted”, and the same engine manages 17%. The gap between those two rows is not a ranking problem. It is the difference between a query that contains evidence and one that does not, and no amount of fusion tuning closes it.

What does close some of it is giving the embedding something in its own vocabulary to match. A code index is written in code; the question is written in English. Asking a small model to draft the code it thinks the answer looks like, and embedding that alongside the question, moves every metric on the held-out split:

The snippet does not have to be right. It has to be written the way the corpus is written, which is enough to land the query vector in the right neighbourhood. Only the vector branch sees it. BM25, symbol, and path still get the caller's literal words, because inventing terms for an exact-match branch manufactures precision that is not there. Queries already full of identifiers are skipped entirely, and their scores are unchanged, which is the shape you would expect if the mechanism works for the reason claimed.

What we cannot claim is that Delphi is state of the art at the top of the list. On this benchmark it is not: the hosted comparator ranks better at MRR and Recall@5, and we publish that alongside the metrics we do lead. The claim we can defend is narrower. Delphi finds more of the answer set than the comparator, beats every published baseline on early precision, and does it much faster, on your own hardware.

The broader lesson is not about any one engine. A retrieval system can be completely broken and still return ranked, plausible, confident results, and every metric downstream of it will move smoothly and mean nothing. If you run a vector index, embed a chunk's own content and check it against its stored vector. It takes one query and it is the cheapest assertion in the stack.

Open the evidence

Every number above comes from queries that were recorded in full. Below are real traces from the benchmarked build, two per workflow, taken by position in the split rather than picked for how they turned out. Expand one to see the exact query the engine received, the ranked files it returned, which retrieval branch found each one, and the raw response record.

They are worth reading for the failures as much as the hits. Three of the eight miss the gold file entirely. The first code2test trace puts five changelog files above the regression test it was asked for: the query mentions a version bump, changelogs are dense with version strings, and BM25 has no way to know that a changelog can never be an answer to “which test covers this?”. The gold file appears at rank 9, found by the path branch. That is a live weakness, not a rounding error.

What comes next

The comparison above rests on 60 cases because that is how many the hosted comparator completed. The run against all 220 failed every query on an HTTP error, so the larger paired sample does not exist yet. Re-indexing those corpora into the comparator and running it again is the single thing that would settle MRR, and until it happens we are not going to describe that metric as won or lost.

On our own side the open problem is rank one. Across the held-out split the gold file is inside the top twenty far more often than it is inside the top five, so the candidate is usually retrieved and then not promoted. That is a judgement problem in the reranking stage rather than a coverage problem in retrieval, and widening the pool makes it worse rather than better. Two directions we have not tried: fitting a reranker to this task instead of using an off-the-shelf one, and routing by query shape so that a question asking “which test covers this” is ranked by a different rule than one asking where something is implemented.

Every artifact behind these numbers is in the repository: the corpus lock file, the per-query records, the summaries, and the measured-and-rejected list of everything that did not work.