Benchmarks · not cherry-picked

CodeMesh vs. reading the files yourself — a 100-question benchmark

The same 100 real developer questions, answered two ways: an agent reading the source files it would normally reach for, versus one query against the live CodeMesh graph over the same path the MCP connector uses. Every number on this page is measured from a real run, not modeled.

DATE 2026-07-20CORPUS 74,680 graph nodes · 7 languages · a real multi-service monorepoPRICED AT Claude Sonnet 5

91%

Median token reduction

11.6× fewer tokens, typical case · even the worst quartile (p25) is 6.5× (85% fewer) across 93 real questions

381,992

Tokens saved, this run

408,163 → 26,171 tokens across the 93-question core set

$0.76

Cost saved, this run

At the Claude Sonnet 5 intro rate, $2/MTok input · projected $82/month saved at 10,000 queries

1.0 ms

Graph query, median

p90 73 ms · max 641 ms · zero errors across all 100 queries

2.8–10.8 ms

Reading files instead (shell)

cat one file: 2.8 ms · repo-wide grep: 10.8 ms — the graph is now faster than both

98/100

Answers under 500 ms

The remaining 2 wait on typed call-graph edges this project hasn't materialized yet (629–641 ms) — see §Engineering

How to read this

Nothing flattering was quietly dropped

A benchmark is easy to make look better than reality by discarding the questions that don't flatter it. Four categories showed up here that a cherry-picked benchmark would have dropped — they're kept in the full table below, and explained here instead.

93 CORE COMPARISONSOrdinary “what does this do / where is this / who calls this” questions. CodeMesh wins on tokens in essentially every one — median 11.6×, worst case still 6.5× at the 25th percentile.

1 OUTLIER, KEPT“Which issues mention sync?” has no file to grep — that data only exists via a connector. The file-reading method degenerates to scanning ~1M tokens of unrelated docs. Real result, not a representative one.

4: NO GRAPH ADVANTAGEPure counts (“how many files exist”) are already near-free with one shell command. The graph's JSON envelope made these slightly more expensive, not less. The graph wins on navigation, not on trivial counting.

2: SERVER-SIDE TIMEOUTQueries with an unbound relationship target initially timed out entirely (>562 s measured). Root-caused to an O(degree) read pattern and fixed in the engine — they now run in 218–447 ms.

Distribution

How often is it 100×, not just 11×?

The median undersells the tail. 93 core comparisons, bucketed by how many times cheaper the graph answer was than reading the source file(s).

Token-reduction multiple, by question

93 core comparisons

<5x

15 questions

5–10x

25 questions

10–25x

32 questions

25–100x

13 questions

100–500x

7 questions

500x+

1 question

Speed

The graph is now faster than the method it replaces

Measured across the full 100-query battery: median 1.0 ms, p90 73 ms, 98/100 under 500 ms, zero errors. Compared the way tools are actually invoked, a graph round trip (including the HTTP hop) now beats the practical fallback: spawning cat on one file costs 2.8 ms and a repo-wide grep costs 10.8 ms (medians of 20+ runs each). Only a bare in-process file read (0.02–0.2 ms — a raw syscall no agent performs in isolation) beats it.

Median latency by method

Lower is better · log-scaled bar widths

CodeMesh graph query

1 ms

cat (one file)

2.8 ms

grep (whole repo)

10.8 ms

Engineering

How the engine got this fast

This benchmark surfaced three real bottlenecks in the graph engine mid-run. They were root-caused and fixed in the same session, then the full battery was re-run for the numbers above. These are before/after measurements of our own engine, not a comparison against any other product — shown here for the same reason the outlier and no-advantage questions are: nothing flattering gets quietly dropped.

1 · Planner anchor orientation

10,235 ms → 79 ms

The query planner always anchored on the first pattern node, scanning the entire graph to answer “what's in this file.” It now scores both ends of a query pattern and anchors on whichever is more selective — cost becomes proportional to the answer, not the size of the graph.

2 · Ranked seek keys + indexes

79 ms → 1 ms

Seek-key selection took the first filter condition instead of the most selective one. Every condition is now ranked by index selectivity before the engine picks which one to seek on.

3 · O(degree) point reads

562 s → 410 ms

Reading one node was decoding its entire adjacency list every time — tens of thousands of entries per read against a well-connected node. A stub-free read path restores the invariant that reading a node never costs more as the graph grows around it.

Two questions still land over 500 ms: “who calls X” today runs a scan over an array property across ~32,000 function nodes (629–641 ms) — inherently proportional to that array, no index helps it. The durable fix is materializing typed call edges at ingest time, which turns this into a ~1 ms edge walk; that work is scheduled, not yet shipped for this project.

Verified with new engine regression tests plus the full existing test suite — all green, and every benchmark answer is byte-identical to the pre-fix run.

All 100 questions

The complete, unfiltered result set

Click a column header to sort. “Reading files” tokens are the actual byte count of the file(s) an agent would read, ÷4. “CodeMesh” tokens are the actual JSON response from the live graph query, ÷4.

#↑	Question	Reading files	CodeMesh	Reduction	Latency (pre-fix)	Latency (now)
1	What does the OAuth server in the gateway do?	2,010	410	4.9×	10.5 s	14.8 ms
2	What's implemented in the JWT signing code?	984	190	5.2×	10.4 s	1.4 ms
3	What does the code-sync bootstrap module implement?	2,696	228	11.8×	10.4 s	1.4 ms
4	What does the query guard do?	1,987	186	10.7×	10.4 s	1.2 ms
5	What's in the policy engine?	1,308	299	4.4×	10.4 s	1.4 ms
6	What does the quota module do?	557	186	3×	10.7 s	1.0 ms
7	What does the audit module do?	980	203	4.8×	10.7 s	1.0 ms
8	How does project mode (auto vs manual) work?	534	183	2.9×	10.7 s	0.9 ms
9	What does the gateway's main.go set up at startup?	3,973	224	17.7×	10.6 s	1.0 ms
10	How does the OpenMetadata connector sync work?	5,963	431	13.8×	9.8 s	1.4 ms
11	How does the extraction engine parse a repo?	4,164	607	6.9×	10.3 s	1.7 ms
12	How does per-file extraction work?	3,308	300	11×	10.2 s	1.1 ms
13	How are languages mapped to parser grammars?	1,179	192	6.1×	10.2 s	0.9 ms
14	How are graph elements emitted for ingestion?	2,315	249	9.3×	10.2 s	1.2 ms
15	What's the manifest format used for skip detection?	549	167	3.3×	10.6 s	0.8 ms
16	What does the sync daemon's CLI entry point do?	4,655	418	11.1×	10.6 s	1.1 ms
17	How does the sync daemon work?	2,880	305	9.4×	10.7 s	1.0 ms
18	How does a sync job run end to end?	1,121	81	13.8×	10.2 s	0.9 ms
19	How does the pusher decide fast-lane vs bulk?	3,292	256	12.9×	10.3 s	1.0 ms
20	How does the file watcher work in the sync daemon?	894	104	8.6×	10.5 s	0.7 ms
21	How is daemon config loaded?	1,412	284	5×	10.6 s	0.9 ms
22	How does the sync daemon track git branch/commit state?	422	103	4.1×	10.6 s	0.8 ms
23	What do the agent hooks in the sync daemon do?	556	60	9.3×	10.3 s	0.7 ms
24	How does connecting an agent to the sync daemon work?	3,238	302	10.7×	10.2 s	1.2 ms
25	What does the supervise command do?	2,329	263	8.9×	10.3 s	1.0 ms
26	What does the VS Code extension's main file do?	3,940	311	12.7×	10.2 s	0.9 ms
27	What's in the VS Code sidebar view?	2,023	305	6.6×	10.0 s	2.1 ms
28	What does the connector CLI do?	797	59	13.5×	10.4 s	0.8 ms
29	How does the connector CLI's core logic work?	2,837	340	8.3×	10.3 s	0.9 ms
30	What does the Cursor SDK helper do?	795	64	12.4×	10.2 s	0.5 ms
31	What does the ingestion service's main.go start up?	18,913	368	51.4×	10.0 s	1.4 ms
32	What does the raw query endpoint do?	1,617	90	18×	10.1 s	0.7 ms
33	What's in the graph intelligence activities file?	56,745	2,474	22.9×	5.1 s	3.2 ms
34	What do the proactive agent handlers do?	30,824	2,228	13.8×	10.1 s	3.2 ms
35	What do the ingestion Temporal workflows do?	27,800	709	39.2×	9.9 s	1.4 ms
36	What do the connector sync activities do?	27,260	1,538	17.7×	10.0 s	2.2 ms
37	What does the identity service's server.js set up?	498	41	12.1×	10.0 s	0.6 ms
38	What does the data service's main.py do?	653	71	9.2×	10.0 s	0.7 ms
39	What does the analytics service's main.py do?	935	39	24×	10.2 s	0.5 ms
40	What does the MCP backend's main.py do?	1,644	111	14.8×	10.3 s	0.6 ms
41	What is the graph engine positioned as, per its README?	4,850	18	269.4×	10.3 s	0.8 ms
42	What's the storage abstraction interface in the graph engine?	1,269	101	12.6×	10.4 s	0.8 ms
43	How does the graph engine's Bolt server work?	1,046	241	4.3×	10.3 s	1.0 ms
44	How does vectorized query planning work in the graph engine?	3,639	181	20.1×	10.4 s	0.7 ms
45	How does the query planner choose a seek key?	6,474	700	9.2×	10.4 s	1.4 ms
46	How does transaction management work in the graph engine?	687	186	3.7×	10.4 s	0.8 ms
47	How does conflict retry work in the graph engine's transactions?	325	63	5.2×	10.4 s	0.6 ms
48	How does full-text search work in the graph engine?	1,870	327	5.7×	10.4 s	1.0 ms
49	How is the fulltext index built in the graph engine?	2,160	338	6.4×	10.4 s	1.1 ms
50	How does the BadgerDB storage backend work?	8,154	1,256	6.5×	10.3 s	2.4 ms
51	How does the MongoDB storage backend work?	3,171	346	9.2×	10.4 s	1.0 ms
52	How does bulk writing work on the Mongo backend?	2,420	208	11.6×	10.3 s	0.7 ms
53	How does the graph engine's built-in MCP server work?	1,898	212	9×	10.3 s	0.8 ms
54	How does RBAC/auth work in the graph engine?	449	127	3.5×	10.4 s	0.9 ms
55	How does the bulk ingest pipeline work in the graph engine?	692	61	11.3×	10.3 s	1.7 ms
56	How does the bulk loader work in the graph engine?	1,174	298	3.9×	10.4 s	0.9 ms
57	What's documented in the graph engine's architecture doc?	787	18	43.7×	10.4 s	0.9 ms
58	What's documented about the graph engine's Cypher compatibility?	782	18	43.4×	10.2 s	0.5 ms
59	What does the graph engine's main entry point do?	5,219	326	16×	10.4 s	1.1 ms
60	Compare the graph engine's Badger and Mongo storage backends.	11,325	1,256	9×	10.3 s	2.0 ms
61	Where is MintBootstrap defined and what does it call?	2,010	118	17×	517 ms	1.0 ms
62	Who calls SignHS256?	984	228	4.3×	717 ms	640.6 ms
63	Where is handleCodeSyncBootstrap implemented?	2,696	198	13.6×	525 ms	2.6 ms
64	What does handleCodeSyncGitHubWebhook do?	2,696	142	19×	532 ms	1.4 ms
65	Where is JoinScope defined?	984	40	24.6×	510 ms	1.0 ms
66	Where is randomID defined and who uses it?	984	127	7.7×	702 ms	628.6 ms
67	Where is run_sync implemented in the sync daemon?	1,121	173	6.5×	561 ms	1.1 ms
68	Where is extract_repo defined in the extraction engine?	4,164	128	32.5×	523 ms	2.0 ms
69	Where is the seek-key ranking function implemented in the graph engine?	6,474	19	340.7×	510 ms	1.5 ms
70	Where is the merge seek-key chooser defined in the graph engine?	6,474	62	104.4×	525 ms	0.7 ms
71	Where is the endpoint-seek chooser defined in the graph engine?	3,639	19	191.5×	520 ms	0.6 ms
72	Where is is_passthrough defined in the extraction engine?	1,179	38	31×	514 ms	0.7 ms
73	Where is the connect function for agent hooks defined?	3,238	370	8.8×	422 ms	0.8 ms
74	Where is autoConnectSync implemented in the connector CLI?	2,837	72	39.4×	520 ms	0.7 ms
75	Where is forwardMcpStdio implemented?	2,837	134	21.2×	530 ms	0.6 ms
76	How many files does the codebase graph track for this project?no advantage	2	17	0.1×	79 ms	72.9 ms
77	How many functions are there in the codebase, broken down conceptually by size?no advantage	0	18	0×	524 ms	461.5 ms
78	Which open Linear issues mention sync?outlier	1,016,742	227	4,479×	106 ms	92.2 ms
79	What GitHub repositories are tracked in the ontology?	14,484	180	80.5×	4 ms	2.9 ms
80	Which files in the codebase are the biggest by lines of code?	96	193	0.5×	105 ms	83.9 ms
81	How many CodeImport nodes exist and what does that represent?	1,549	18	86.1×	369 ms	354.3 ms
82	What's the breakdown of files by language in this codebase?no advantage	11	80	0.1×	79 ms	75.6 ms
83	How many CodeClass nodes are there in this project?no advantage	0	17	0×	129 ms	153.6 ms
84	What connector is the codebase ingested through?fixed	2,143	6	—	>562 s	446.6 ms
85	Which repository is this project's code attached to in the graph?fixed	26	6	—	>562 s	218.1 ms
86	What does the decisions log record?	1,568	18	87.1×	9.5 s	1.4 ms
87	What's documented in the MCP connection guide?	2,160	18	120×	9.5 s	0.7 ms
88	What does the API gateway's README describe?	1,785	18	99.2×	9.6 s	0.6 ms
89	What does the OAuth bootstrap test cover?	246	44	5.6×	9.7 s	0.7 ms
90	What does the code-sync bootstrap test cover?	219	65	3.4×	10.0 s	0.8 ms
91	What does the gateway's integration test cover?	9,483	964	9.8×	10.1 s	1.6 ms
92	What does the auto-connect e2e test check?	860	18	47.8×	10.2 s	0.6 ms
93	What does the sync-connect setup script do?	605	18	33.6×	10.1 s	0.6 ms
94	What does the ops script manage?	13,149	18	730.5×	10.0 s	0.6 ms
95	What ports and services does the environment config define?	7,088	18	393.8×	10.0 s	0.5 ms
96	How do the OAuth module and code-sync bootstrap module interact?	4,705	410	11.5×	10.1 s	1.1 ms
97	Compare the sync daemon and its sync-job module.	4,000	305	13.1×	10.1 s	1.0 ms
98	Compare the graph engine's architecture doc with its README.	5,636	18	313.1×	10.3 s	0.6 ms
99	Compare the VS Code extension and the connector CLI as MCP entry points.	6,777	310	21.9×	10.0 s	1.0 ms
100	Compare the API gateway's OAuth flow with the graph engine's RBAC auth.	2,458	410	6×	10.2 s	0.9 ms

Accuracy

Verified against real source, not assumed

Token and latency numbers say nothing about whether the graph's answer is actually correct. A sample of results was cross-checked against independently-derived ground truth — real regex extraction from source, real grep for call sites, real shell counts — rather than trusting the graph's output at face value.

100%

Symbol listings

54/54 verified exact against real source, once corrected for test artifacts

100%

Reverse caller lookups

Every real call site correctly attributed (2 targets, 15 real sites)

8/9

Forward call resolution

1 inconclusive due to an unscoped test query, not a confirmed graph error

100%

Aggregate counts

5/5 — file count within 0.6% of a real filesystem count, an explainable scope difference

10 raw “mismatches,” all traced to root cause — none were graph errors.

3 (graph-engine files): the “missing” symbols were checked against the canonical, up-to-date engine repo — but the graph had ingested a clone two commits behind. Confirmed directly: those symbols don't exist in the clone that was actually synced. The graph was 100% correct relative to its real source.

7 (TypeScript/JS files): the ground-truth regex over-counted local constvariables as symbols and missed bare class methods without a function keyword — a limitation of the test harness, not a graph defect. Reclassified as inconclusive rather than scored either way.

3 of 4 call-graph “mismatches” were also test artifacts: the ground-truth check only grepped the first 2,000 characters of each function body. Reading the full bodies confirmed the flagged calls really do happen — just past where the truncated check stopped looking.

The one real, standing accuracy gap: a generic, commonly-named callee resolved to an unrelated function of the same name in a different service. The batch check only verifies a name was called somewhere in the body — not whether the graph's claimed definition location for a generic name is the right one (here, it wasn't). This affects generic/common callee names resolved via unscoped cross-service matching — not symbol listings, not counts, not reverse lookups, not distinctively-named callees.

Cost

Priced at real Claude Sonnet 5 rates

These tokens are context returned into the model — i.e. input tokens on the model's next turn — priced at published per-million-token rates.

Pricing tier	Reading files	CodeMesh	Saved
Intro — $2.00/MTok (through 2026-08-31)	$0.8163	$0.0523	$0.764
Standard — $3.00/MTok	$1.2245	$0.0785	$1.146
All 98 completed queries, intro rate	$2.8498	$0.0531	$2.7968

Addendum · one real incident

A schema-ambiguity edge case, found and fixed

Separate from the 100-query sweep above — this is a single real scenario, kept to its actual scale (not inflated into 100 cases). It matters because it shows token reduction and answer correctness are different axes, and this is the clearest example of them pulling in opposite directions.

Asked “what connectors are in the data service,” the graph first matched a small set of this project's configured integrations — when the real answer was a much larger internal plugin catalog that only exists as source files, not as connector records. Nothing distinguished the two meanings before the agent queried. Fix: the query-guidance tool now surfaces a disambiguation note whenever that ambiguity is possible, and shows both query shapes side by side. Verified with unit tests plus one live end-to-end run against the real code path.

	Before fix	After fix	Ground truth
Accuracy	0% — answered a different question	100% (91/91)	100% (ground truth)
Round-trip time	21 ms	165 ms	26 ms
Response tokens	30	120	232
Cost / 10,000 queries	$0.6	$2.4	$4.64

Fixing correctness cost time and tokens here — it didn't save them. The old query was fast only because it skipped the guidance step straight into a wrong answer. The corrected graph path still beats reading files on tokens (120 vs. 232 — 48% fewer) despite the added guidance overhead. Treat this as one worked example of the tradeoff, not a generalizable rate.

See it on your own repository

These numbers are from one real monorepo. The fastest way to know what CodeMesh does for yours is to open one.

Create your workspace Back to overview