benchmark seems incorrect

[arita37]

Thanks for the work.

Benchmark are comparing full files load vs partial results from cce
but Claude code and codex, uses grep and partial load of files...

---> Actually, it should be compared with partial load with grep.

[rajkumarsakthivel]

Fair point. The benchmark methodology is documented in the README:

"without CCE" means reading the full content of every file the query touches. This is conservative (agents often read more files than needed).

You're right that Claude Code uses grep, Read with line ranges, and other targeted tools rather than always reading full files. Our baseline overstates the "without" case for queries where the agent would have grepped instead of reading whole files.

We chose full-file as the baseline because:

1. It's reproducible and deterministic (no agent behavior variability)
2. It represents the worst case that agents frequently hit (especially on first exploration of unfamiliar code)
3. Measuring "what Claude Code would actually read" requires running the agent on each query, which introduces non-determinism

That said, the real-world savings are likely lower than 94% for users whose agents already grep efficiently. The 94% number is retrieval savings vs full-file baseline, not vs optimal agent behavior.

We should make this clearer in the benchmark docs. Adding a note about what the baseline represents and doesn't represent.

Thanks for raising this.