Where Do Context-Engine Gains Come From?

1What the protocol does

A margin over BM25 does not say whether an engine's design or a learned reranker that would help any candidate pool produced it, nor how much of it was selected on the evaluation data. The protocol separates these by

• recording for every evaluation case whether development could have seen it, in an exposure ledger with three classes, and pooling only cases scored once and never before;
• building a ladder of conventional retrievers from Delphi's own embedding model, attaching Delphi's rerankers and query expansion one rung at a time with the same prompts and parsers;
• varying candidate generation and learned reranking factorially, so the two factors the ladder confounds can be told apart; and
• ablating Delphi's six candidate branches one at a time on a fresh draw that no development decision could see.

Every number is generated from persisted run artifacts by scripts in the released repository, and failed or invalidated runs are kept for the audit trail.

2Findings

Measured on three SWE-bench Verified draws that development never saw (62, 98, and 60 instances; sizes fixed before scoring) and one repository-disjoint commit-to-files set.

• The advantage is the candidate pool, not the reranker. Before any reranking, Delphi's candidates lead the full conventional stack by +0.19 MRR and +0.07 Recall@20. The shared rerankers add +0.34 MRR to conventional candidates but +0.19 to Delphi's, so after reranking MRR ties while the recall gap (+0.09) persists.
• The pool advantage is chunking, index, and weighting. The branch ablation attributes +0.15 MRR to Delphi's chunking and index (same two-branch fusion) and +0.06 to its vector-heavy weighting; the four structural branches add +0.06 jointly, none more than 0.03.
• It depends on query style. On repository-disjoint commit-to-files queries the pattern inverts: Delphi's pool has lower recall than two-branch fusion and the rerankers add nothing to either pool.
• Exposure changes the evidence class. A case-level ledger reclassifies Delphi's 220-case ARB partition from held-out to validation evidence. Confirmatory results rest on 238 cases scored once and never before, with intervals adjusted for three looks.
• Matched contracts remove a hosted lead. Routing every engine's evidence through one frozen synthesis stage turns an apparent hosted-engine lead on documentation into a four-way tie. Re-indexing a corpus from scratch reproduced Delphi's ordered top-20 on 1 of 18 cases.
• No seed effect in the executable pilot. 62 instances, four seed conditions, two repeats, one fixed mini-SWE-agent: a retrieval seed resolves no more verified repairs than a random-file seed, and repeats disagree by more than the effects under study. The pilot audits its own seed and yields a precision target for a decisive study.

The paper claims no state of the art; several findings are null or negative for the engine we built, and they are reported with the same prominence as the positive ones.

3Artifacts

Repository corpora (58 repositories, 425 snapshots) are re-cloned from public GitHub at the recorded commits by a script in the repository rather than redistributed. The trajectories, results, and analyses are released under CC BY 4.0.

Citation

@article{bansal2026contextengine,
  title   = {Where Do Context-Engine Gains Come From? A Component-Level Decomposition of
             Repository Retrieval Under Matched Baselines and Audited Exposure},
  author  = {Bansal, Aayam and Gangwani, Ishaan},
  year    = {2026},
  url     = {https://trydelphi.ai/research},
  note    = {Preprint}
}