Benchmarks

Measured on public code. Checked against the code.

100 questions drawn at random in five public repositories, an agent with and without Wingbox, every edge compared with the code. Intervals are 95 % confidence intervals. The scripts, the reference lists and every result file are public, and one container reproduces the run.

Scripts and results on GitHubRead the full report
100 of 100

questions where wingbox impact names the complete list of dependent files (96 to 100 %).

-35 %

tokens an agent reads per answer with Wingbox, against its own search alone (-47 % to -20 %). Cost: -51 %.

55 of 55

complete answers from the agent with Wingbox. With its own search alone: 48 of 55.

3.2 s

to build the graph of FastAPI, 1,227 files, from a fresh checkout: graph, report and map.

Test 1 · agent

An agent with and without Wingbox

55 questions of the kind "which files depend on this file, directly or through one other file", in five public repositories (TypeScript, Rust, Python). Claude Code answers each one twice: with its built-in search only (Read, Grep, Glob), and with the wingbox command added. One trial per question and setup.

Tokens per answer
Mean of 55 questions: everything the model read and wrote.
With Wingbox32,183
Built-in search only49,292
Data table
SetupTokens per answer
With Wingbox32,183
Built-in search only49,292
Cost per answer
Mean of 55 questions, US dollars.
With Wingbox$0.031
Built-in search only$0.063
Data table
SetupUSD per answer
With Wingbox$0.031
Built-in search only$0.063
SetupComplete answersF1, meanTokens, meanUSDTurns
With Wingbox55 of 55 (93 to 100 %)1.000 (1.000 to 1.000)32,183 (28,229 to 36,523)0.0313.0
Built-in search only48 of 55 (76 to 94 %)0.979 (0.955 to 0.996)49,292 (42,863 to 56,703)0.0635.0
  • Tokens -35 % with Wingbox (-47 % to -20 %), cost -51 % (-58 % to -43 %). Fewer tokens on 39 of the 55 questions, more on the other 16.
  • The complete list: with Wingbox on all 55 questions; with search alone on 48. On 7 questions only the agent with Wingbox was complete, on 0 only the agent without it (exact sign test, p = 0.016).
Tokens per answer, by repository
Mean over the questions of each repository. The saving is large on the Rust and Python repositories and absent on the two TypeScript ones.
  • With Wingbox
  • Built-in search only
chrome-devtools-mcp
12 questions
39,146
42,406
immich-server
12 questions
35,867
31,950
agent-browser
12 questions
35,158
78,529
claudes-c-compiler
12 questions
24,270
50,428
fastapi
7 questions
22,396
38,753
Data table
RepositoryWith WingboxBuilt-in search only
chrome-devtools-mcp39,14642,406
immich-server35,86731,950
agent-browser35,15878,529
claudes-c-compiler24,27050,428
fastapi22,39638,753
Example

One question, with and without

Which files depend on src/backend/riscv/mod.rs (claudes-c-compiler), directly or through one other file? The reference list holds 32 files.

$ wingbox impact src/backend/riscv/mod.rs 2
Impact of changing src/backend/riscv/mod.rs
  direct dependents   2
  within 2 hops       32

Depth 1
  src/backend/mod.rs      imports
  src/backend/common.rs   imports

Depth 2
  src/backend/arm/codegen/emit.rs           imports
  src/backend/generation.rs                 imports
  src/backend/i686/codegen/alu.rs           imports
  src/backend/i686/codegen/asm_emitter.rs   imports
  src/backend/i686/codegen/atomics.rs       imports
  src/backend/i686/codegen/calls.rs         imports
  src/backend/i686/codegen/casts.rs         imports
  src/backend/i686/codegen/comparison.rs    imports
  src/backend/i686/codegen/emit.rs          imports
  src/backend/i686/codegen/float_ops.rs     imports
  src/backend/i686/codegen/globals.rs       imports
  src/backend/i686/codegen/i128_ops.rs      imports
  src/backend/i686/codegen/intrinsics.rs    imports
44 lines in all: the 32 files, nothing else
The agent with Wingbox

2 turns, 16,482 tokens. All 32 files named, none missing.

The agent with search alone

13 turns, 147,818 tokens. 18 files named, 14 missing.

What the agent is asked

Before changing `src/backend/riscv/mod.rs`, I need a complete review list: every file of this repository, test files included, that depends on `src/backend/riscv/mod.rs` directly, or through exactly one other file. [...] Do not list `src/backend/riscv/mod.rs` itself. Answer with only a JSON array of repository-relative file paths.

This is the question where Wingbox saved the most. The chart above shows the mean and the repositories where it saved nothing.

Test 2 · command

Is the answer complete? 100 questions

No agent here. Per repository, 20 files are drawn at random (fixed seed) among the files that at least one other file imports; nothing else is filtered. wingbox impact is run on each, and the files it names are compared with a reference list built by resolvers that share no code with Wingbox: Python's own parser (equal to grimp on FastAPI), a TypeScript resolver and a Rust resolver. The lists hold 1 to 503 files, median 9.

RepositoryLanguageQuestionsComplete and nothing extra
chrome-devtools-mcpTypeScript2020 of 20
immich-serverTypeScript2020 of 20
agent-browserRust2020 of 20
claudes-c-compilerRust2020 of 20
fastapiPython2020 of 20
All100100 of 100 (96 to 100 %)

No required file is missing in any of the 100 answers. In 90 the answer equals the reference list file for file. In the other 10 Wingbox also names files the reference accepts without requiring them: scripts that import the compiled output of the file, which tsconfig maps back to the source, and a Python package file executed on the way to the import. The median answer is 166 tokens long.

Test 3 · graph

Are the edges in the code?

Every file-to-file dependency of the Wingbox graph against every import statement of the code, read by a resolver that shares no code with Wingbox. Wrong: an edge the code does not have. Missing: a dependency of the code the graph does not have.

CorpusLanguageDependencies in the codeWrongMissing
immich, serverTypeScript3,79500
chrome-devtools-mcpTypeScript90101
agent-browser, Rust sourcesRust39100
claudes-c-compilerRust1,46200
FastAPI, whole treePython1,80100
FastAPI, the fastapi/ packagePython12400
nanoGPT, minGPT, microgradPython1500

The missing dependency in chrome-devtools-mcp: src/third_party/index.ts imports src/third_party/lighthouse-devtools-mcp-bundle.d.ts, the type declarations of a vendored bundle. Wingbox leaves vendored and minified bundles out of the graph.

Test 4 · build time

How long the graph takes to build

Seconds from a fresh checkout to the graph, the report and the map, with no earlier output and no cache.

Build time, seconds
Median of 3 runs. GitHub-hosted runner, Linux, 4 logical CPUs.
fastapi (1,227 files)3.23 s
immich-server (707 files)1.84 s
agent-browser (363 files)2.06 s
claudes-c-compiler (351 files)2.23 s
chrome-devtools-mcp (296 files)0.79 s
karpathy (29 files)0.28 s
Data table
CorpusBuild, seconds
fastapi (1,227 files)3.23 s
immich-server (707 files)1.84 s
agent-browser (363 files)2.06 s
claudes-c-compiler (351 files)2.23 s
chrome-devtools-mcp (296 files)0.79 s
karpathy (29 files)0.28 s

The size of the graph each build wrote is recorded with the time (1,227 modules and 1,818 import edges for FastAPI), so a build that did less cannot pass as a fast one.

Scope

What this does not show

  • One kind of question: which files import this file, directly or through one other file. It is the question asked before a change; other questions are not measured here.
  • The questions are drawn inside five repositories, so they are not independent in the strict sense; the per-repository rows show where a result comes from.
  • The agent test uses one agent and one model, one trial per question. A second run will differ in the details, which is what the intervals are for.
Reproduce in a clean container
$ git clone https://github.com/aileron-tech/wingbox && cd wingbox$ docker build -f Dockerfile.benchmark -t wingbox-benchmark .$ docker run --rm wingbox-benchmark

The container installs Wingbox 0.2.2 from PyPI, fetches the repositories at pinned commits, draws the same questions (seed 20261116), runs the tests that need no model and stops with an error when a result differs from the committed one: the log of that run ends with REPRODUCED, and a workflow repeats it on a hosted runner at every change of the benchmark. The agent test needs paid model sessions and is run apart (Wingbox 0.2.1, whose answers on the 100 questions are identical; model claude-sonnet-5-5); its trials are in the result files. Run of 2026-11-16.