Benchmark Explorer

All 98 questions from CodeMesh's published benchmark, in one interactive table. Filter by difficulty tier or by the real codebase area each question targets, sort by token counts or reduction, and search for a specific question. Nothing is excluded, including the 11 questions CodeMesh answered incorrectly.

For the full write-up of how this data was produced, see the methodology. For the canonical, non-interactive result set, see the full benchmark page.

98 of 98 questions
QuestionTierAreaBaseline tokensCodeMesh tokensReduction↓VerdictScoreRounds
What default instructions string does openclaw send for OpenAI Codex Responses requests, and what does it use for empty input text?Q060mediumai-providers2,498,0169,68199.6%correct1.002
What source id does openclaw assign to built-in API providers, and which file registers it?Q064mediumai-providers1,972,6779,47599.5%partial0.752
What are the default read-lines and max-chars limits when openclaw's memory host reads a memory file?Q048mediummemory945,4409,53499.0%correct1.002
When the recovery scanner reads a running main session's transcript to decide how to resume it after a restart, what exact message-count and byte-size caps does it pass to the transcript reader?Q115hardsession-transcript693,27813,52398.0%incorrect0.003
What HTTP path prefixes does openclaw's gateway use for board views and for the standalone MCP app?Q020mediumgateway448,4019,58697.9%correct1.002
What are the exact reconnect-backoff parameters and overall admission deadline a worker uses when repeatedly reconnecting to its gateway after a retryable close, and what pre-existing gateway-client timeout constant does the per-attempt admission timeout default to?Q123expertworker-connection-retry487,26010,10897.9%correct1.002
What are the documented default values for openclaw's gateway auth rate limiter config (max failed attempts, sliding window, lockout duration, max tracked identities)?Q013hardgateway-auth410,9589,74997.6%correct1.002
What is openclaw's default Azure OpenAI API version, and what first-event timeout does it apply to Azure Responses streams?Q059mediumai-providers570,82115,50997.3%partial0.753
When the openclaw gateway is still starting its sidecars and cannot serve a websocket connection yet, which WebSocket close code does it use, what retry-after hint does it advertise, and what are the reason strings? Name the file.Q003hardgateway-protocol524,18414,74297.2%partial0.753
What wall-clock limit and tool budget does openclaw apply to a headless cron trigger evaluation, and how many trigger evaluations may run concurrently?Q038hardcron339,5499,81997.1%correct1.002
In openclaw's channel plugin loading, what caps exist on bundled channel load contexts and boundary roots, and what is the read-only channel plugin resolution cache size?Q035hardchannels332,9799,84997.0%correct1.002
There are two different user-facing 'gateway restart' notice strings for an interrupted main session -- one telling the user their turn couldn't safely resume, and one telling them to run /new or /reset. What distinguishes the two code paths that decide which one gets sent, and how does each affect the session's stored recovery state going forward?Q114extremerestart-recovery641,36619,91596.9%incorrect0.004
The gateway's background loop that automatically retries restart-aborted main sessions uses an exponential backoff. Using its default constants, what is the sleep delay immediately before the third (final) recovery attempt?Q113extremerestart-recovery307,17010,04496.7%correct1.002
The worker protocol enforces two different byte ceilings for JSON payload content depending on RPC family. What are the two values, and which constant deliberately reuses the smaller one to bound opaque provider-replay state blobs?Q119hardworker-protocol-limits452,47515,47696.6%correct1.003
How many transient retries does openclaw's cron timer trigger default to, and after how many consecutive failures does the cron service raise a failure alert by default?Q039mediumcron276,2669,54996.5%correct1.002
How many session targets can a single sessions.patchMany request address in openclaw, and what constant name carries that limit?Q009mediumgateway-protocol267,6459,63596.4%correct1.002
What are openclaw's default thread-binding idle and max-age policy values, and what does the max-age default mean?Q030hardchannels283,10710,16196.4%correct1.002
When openclaw masks a credential-like value for logging, how many characters does it reveal from the start and end of the secret, and does that count change based on the secret's length?Q106hardsecurity-secret-masking273,0809,89196.4%incorrect0.002
How long does openclaw wait for session work admission to drain, and which constant carries that value?Q046mediumsessions252,0369,47796.2%correct1.002
What is openclaw's default run-activity heartbeat interval in the channel run state machine?Q034mediumchannels240,2269,45696.1%correct1.002
How does openclaw's gateway auth rate limiter treat loopback addresses (127.0.0.1 / ::1) by default, and what still happens to failed auth attempts from loopback?Q014hardgateway-auth252,4929,98196.0%correct1.002
What is openclaw's default channel connect grace period, and which module defines it?Q018mediumgateway237,2629,46196.0%correct1.002
What is the maximum number of chat history entries openclaw's gateway protocol schema allows, and which file defines that cap?Q008mediumgateway-protocol234,3449,62095.9%correct1.002
In openclaw's WorkerConnection client, which close reasons put the connection into the terminal "fenced" state versus the retryable reconnect path, and how can a successful heartbeat response, with no socket close at all, still trigger fencing?Q118extremeworker-connection-lifecycle263,35110,80595.9%correct1.002
When openclaw computes the remaining backoff delay for a released ingress event, what backoff factor and jitter does it use, and how is the attempt count bounded before the backoff is computed?Q024hardchannels-ingress467,25219,39195.8%correct1.004
What limits does openclaw apply when probing inbound media on a channel event: how many probes, at what concurrency, and within what time budget?Q033hardchannels234,4299,78195.8%correct1.002
What retry delay does openclaw apply after a cron job's 4th consecutive execution failure, and what happens to the delay once a job has failed 8 times in a row?Q102extremecron-retry-backoff335,96014,80895.6%partial0.503
What filename does openclaw's memory host SDK treat as the memory root file, and what is the current memory chunking version?Q047mediummemory202,7819,49295.3%correct1.002
What are the maximum lengths openclaw enforces for an OpenAI prompt-cache key and for a replayed Responses item id?Q062mediumai-providers408,70219,27395.3%correct1.004
What timeout does openclaw apply when waiting for a configured binding route to become ready?Q036mediumchannels195,7219,58795.1%correct1.002
In openclaw's cron command runner, what is the default command timeout, and what value is used for the "effectively unbounded" timeout constant?Q041hardcron194,6729,57195.1%correct1.002
Which helpers does openclaw's skill env-override runtime use to block dangerous host environment variables, and where do those helpers live?Q068hardsecurity294,71114,37195.1%incorrect0.003
How many memory items can a single openclaw memory migration carry, and which schema file defines that limit?Q050mediummemory190,6479,45795.0%correct1.002
What is the cron expression evaluation cache size in openclaw, and what is the maximum length of a declarative cron job label?Q042mediumcron361,70318,95994.8%correct1.004
In openclaw's host env security module, which two git-related environment variables are singled out by name, and what default protocol allowlist does it define?Q072hardsecurity183,1719,62094.7%correct1.002
In openclaw's worker inference protocol schema, what is the maximum number of context messages and the maximum output tokens allowed?Q007mediumworker-protocol176,7749,72894.5%correct1.002
When a cron job's isolated agent run finally reaches delivery (for example after a long gateway outage delayed the run), how does the delivery-dispatch pipeline decide the scheduled announce/direct delivery is too stale to send, and what is the exact threshold?Q101expertcron-delivery270,78514,78994.5%correct1.003
When openclaw builds the "current context" block for an isolated cron agent run, how many messages does it include and what per-line and per-block character caps apply?Q044hardcron165,2439,56894.2%correct1.002
How does openclaw's sleepWithAbort guard against invalid or oversized delays, and what error does it raise when the abort signal fires?Q027hardpackages/retry167,9419,97094.1%partial0.502
How many consecutive typing-indicator failures does openclaw tolerate by default before backing off, and which file holds that default?Q029mediumchannels162,7789,55794.1%correct1.002
What is openclaw's default initial delay before a progress draft is emitted, and what is the channel feedback-reflection cooldown default?Q032mediumchannels163,2259,69194.1%correct1.002
What are openclaw's default and maximum cron script timeouts (in seconds), and the default and maximum cron script tool budgets?Q037mediumcron158,6489,59694.0%correct1.002
What are openclaw's default gateway request timeout and default pre-auth handshake timeout, and which package defines them?Q011mediumgateway-client157,2269,54893.9%correct1.002
What is openclaw's gateway control-plane rate limit (requests per window) and window length?Q017mediumgateway158,6569,59993.9%correct1.002
Which openclaw workspace package implements computeBackoff, what is its exact formula, and what does src/infra/backoff.ts contribute?Q025hardpackages/retry160,7549,96193.8%correct1.002
What is openclaw's default maximum number of ingress delivery attempts, and what is the default maximum backoff delay between ingress retries?Q021mediumchannels-ingress151,9499,56093.7%correct1.002
On gateway restart, when the cron service replays cron jobs that were missed while it was down, how many missed jobs does it run immediately by default, what happens to the rest, and why are missed agentTurn jobs handled differently from other missed jobs?Q104expertcron-startup-catchup162,59610,42493.6%partial0.502
In openclaw's ingress failure disposition logic, what are the two possible disposition kinds, and what reason string is recorded when the retry limit is exceeded?Q023hardchannels-ingress145,3609,58193.4%correct1.002
What Anthropic beta header value does openclaw use for server-side fallback, and what value does it send for the fallback selection?Q057hardai-providers145,0949,54593.4%correct1.002
When Anthropic's server-side fallback beta reroutes a refused stream onto Opus 5 or 4.8 mid-generation, what happens to the pre-fallback assistant content blocks in openclaw's transcript, and what fixed cost table does it substitute for billing that turn?Q125extremellm-provider-anthropic158,65410,50993.4%partial0.752
Which file defines openclaw's gateway wire protocol version constant, and what is the current PROTOCOL_VERSION value?Q001mediumgateway-protocol142,5119,48193.3%correct1.002
What error code does openclaw's agent core use to signal that a transcript cannot be continued, and where is it defined?Q051hardagent-core142,5519,57393.3%correct1.002
What Claude Code version string does openclaw's Anthropic model contract pin, and in which file?Q056mediumai-providers142,1839,51793.3%correct1.002
Why does the helper that builds the local WebSocket URL for a worker-to-gateway connection explicitly throw when the socket path contains a colon?Q122hardworker-connection-transport218,84415,14893.1%correct1.003
When a new cloud worker turn tries to claim a session whose previous turn's workspace result is still reconciling (rather than truly active elsewhere), how long does the gateway wait for the prior claim to release before giving up, what error message does it surface, and what git-ref shape must a stored workspace-conflict record satisfy before it's trusted as durable?Q124extremeworker-turn-lifecycle302,72520,87593.1%partial0.504
What are the transcript-commit batch limits in openclaw's worker admission protocol: max messages per batch, max content parts, and max JSON nesting depth?Q006hardworker-protocol136,6689,63293.0%correct1.002
In openclaw's retry package, what does computeBackoffSchedule return for attempt <= 0, and what happens when the attempt exceeds the provided schedule length?Q026hardpackages/retry139,2049,71593.0%correct1.002
What does openclaw's stripUrlUserInfo do, which package owns it, and how does it behave when the input is not a parseable URL?Q067hardsecurity139,5669,73593.0%correct1.002
The gateway's auth rate limiter gives loopback (localhost) callers an escalating delay instead of a hard lockout after repeated failed auth attempts. How many of those failed-attempt timestamps does it retain per loopback key before trimming older ones, and how is that number derived from the delay constants rather than being a separate hardcoded value?Q107extremeauth-rate-limit217,82015,34693.0%correct1.003
When openclaw runs a user-configured 'safe' regex against an untrusted string, how does it avoid quadratic-time scanning on very long inputs, and what happens if a match only occurs in the middle of a long string but not within the tested windows?Q105expertsecurity-regex135,14910,04692.6%correct1.002
openclaw defines a CHARS_PER_TOKEN_ESTIMATE constant with the same value in two different workspace packages. Name both files and the value, and say what that duplication implies about package layering.Q054experttokenization129,4999,76092.5%correct1.002
What placeholder text does openclaw substitute when Anthropic reasoning content is omitted from a replayed message?Q058mediumai-providers124,0069,45092.4%correct1.002
There is a fixed threshold in the agents runtime for warning about a tool-call loop, and a comment ties its non-configurability to a specific past PR. What is that threshold value and what does the comment say was retired?Q116hardagents-runtime125,6709,74492.2%correct1.002
What is the worker heartbeat interval used by openclaw's worker admission protocol, and what is WORKER_RPC_SET_VERSION set to?Q004mediumworker-protocol119,7099,62392.0%correct1.002
What exact marker string does openclaw use as the system prompt cache boundary, and which file owns it?Q055hardai-providers118,2789,56391.9%correct1.002
Before validating a worker inference frame (start/cancel/event/terminal) against its typebox schema, openclaw runs a separate hand-rolled JSON safety walk. What three classes of value does this walk reject that the schema check alone wouldn't necessarily catch, and what is the exact nesting-depth limit it enforces?Q121extremeworker-inference-validation186,65815,08991.9%partial0.753
openclaw normalizes differences between OpenAI Responses and Azure Responses streaming events. What are the two content-part type strings and the two delta event type strings it maps between?Q061hardai-providers122,14910,02291.8%correct1.002
What is openclaw's outbound echo suppression window, and how many outbound message identities are retained?Q028mediumchannels114,0589,55991.6%correct1.002
openclaw's gateway protocol version file declares four minimum/current version constants. Name all four and give each value, and explain what the asymmetry between the client minimum and the node/probe minimum means in practice.Q002hardgateway-protocol119,69510,13091.5%correct1.002
What two conditions must both hold before openclaw dead-letters a retryable ingress event, and what happens to an event that has exceeded the attempt limit but not the other condition?Q022hardchannels-ingress120,03410,24691.5%partial0.502
In openclaw's safe-regex checker, what are the cache size and test window constants?Q066mediumsecurity111,0809,48791.5%correct1.002
What are the default minimum and maximum draft streaming chunk sizes in openclaw's draft streaming chunker?Q031mediumchannels108,8929,67191.1%correct1.002
During the memory host SDK's legacy-table-to-canonical migration, which two legacy tables get temporary indexes created before the row copy runs, and on exactly which columns?Q112hardmemory-schema-migration109,3459,72791.1%correct1.002
What is the maximum allowed length of a chat send session key in openclaw's gateway protocol primitives, and how many viewer-presence keys can a session track?Q010mediumgateway-protocol158,64514,30991.0%correct1.003
What are the default, minimum and maximum cron stream batch intervals and batch byte sizes in openclaw?Q040hardcron106,0579,60690.9%correct1.002
What exact string format must a worker's build-identity bundle hash satisfy in the gateway protocol schema, and against how many independent server-side sources does the gateway cross-check that same hash before admitting the connection?Q120expertworker-admission158,69814,44090.9%incorrect0.003
For the OpenAI ChatGPT Responses transport, openclaw defines two separate byte caps for reading the HTTP response body from the same endpoint family: one for error responses and one for streamed success bodies. What are the two constant values, and by what factor do they differ?Q128extremellm-provider-streaming109,5269,99290.9%correct1.002
Name the SQLite table-name constants openclaw's memory host SDK defines for its index (base schema plus FTS schema).Q049hardmemory113,41710,53090.7%correct1.002
What caps does openclaw apply to cron run diagnostics normalization (entries, chars per entry, summary chars), and what tail size does it keep for exec diagnostics?Q043hardcron304,27430,80089.9%correct1.007
The memory host SDK's FTS schema helper for the derived chunk-body index refuses to build if its target table name collides with a specific other table -- which one, and why does the check use a case-insensitive comparison rather than a plain equality check?Q111extremememory-schema-fts101,78810,25289.9%partial0.752
In openclaw's durable outbound delivery completion module, what are the two completion "kind" variants and what states can a completion result report?Q053harddelivery189,11119,43389.7%incorrect0.004
When creating an armable transport stall watchdog without an explicit check interval, what formula does openclaw use to derive the poll interval from the idle timeout, and what floor does it ultimately get clamped to?Q130hardchannels-transport95,3739,80689.7%correct1.002
The memory host SDK's chunk provenance table restricts origin_class and session_kind to closed enums and cascades deletes from the canonical chunk table. When the schema-ensure function backfills provenance rows for chunks that predate the provenance table, what exact origin_class and session_kind values does it assign, and where does it source the observed_at timestamp from?Q110extremememory-schema92,9269,97089.3%correct1.002
What exact literal marker does openclaw insert into a system prompt to separate the cache-stable prefix from dynamic runtime additions, and what does prependSystemPromptAdditionAfterCacheBoundary do differently when that marker is absent from the prompt versus when it is present with no dynamic suffix after it?Q131extremellm-provider-prompt-cache92,61310,29488.9%correct1.002
Name the rate-limit scope constants exported by openclaw's gateway auth rate limiter and say what a scope is for.Q015hardgateway-auth130,77815,06588.5%correct1.003
What limits does openclaw's session diff module enforce on files, untracked files, per-file patch bytes, total patch bytes and total changed lines?Q045hardsessions129,43015,03088.4%correct1.003
In openclaw's gateway client timeouts module, what is MIN_CONNECT_CHALLENGE_TIMEOUT_MS and what is MAX_SAFE_TIMEOUT_DELAY_MS?Q012mediumgateway-client76,9789,73387.4%correct1.002
How does openclaw compare secret strings without leaking length information, and what does its comparison return when both inputs are empty strings?Q065hardsecurity107,29714,14486.8%incorrect0.003
For an isolated cron agent run configured with a 90-second wall-clock job timeout, how long does the pre-execution watchdog wait (once the runner has started but before the run reaches an execution-stage phase) before it fires a timeout, and what formula produces that number?Q103extremecron-watchdog113,54614,94786.8%correct1.003
openclaw's gateway agents-workspace server method is deliberately read-only. What does its header comment say about mutations, and which API is named as the place edits belong?Q070expertgateway141,80920,05685.9%incorrect0.004
What are the default thresholds openclaw's bot-to-bot channel reply loop guard uses before it starts suppressing messages, and does it count a message from bot A to bot B separately from a reply from bot B to bot A?Q126extremechannels-loop-guard132,67019,83685.0%correct1.004
In the memory host SDK's SQLite schema, chunk-level recall metadata (importance, triggers, project_key) lives in its own table rather than on the chunk table itself. What numeric range is enforced for the importance column, and what does the schema-ensure function do if it finds those same three columns still present directly on the canonical chunks table from an unreleased build?Q109expertmemory-schema98,01115,07484.6%correct1.003
What are the default request-count and time-window limits for openclaw's in-memory webhook rate limiter, and how does its sibling anomaly tracker decide when to actually emit a log line for repeated failures from the same key?Q129extremeplugin-sdk-webhook97,01414,93784.6%partial0.253
In openclaw's gateway channel health policy, when a channel snapshot reports lifecycle "starting" or "recovering", under what condition is it judged healthy, what reason string is returned, and what happens if the start timestamp is missing or the window has passed?Q019hardgateway127,85520,64183.9%incorrect0.004
Why does openclaw give the pre-auth device bootstrap-token verify path its own dedicated rate-limit scope instead of reusing the shared-secret scope? Cite the mechanism named in the code.Q016expertgateway-auth155,83526,78682.8%incorrect0.005
How many 'unauthorized role' errors is a single WebSocket connection allowed to trigger by default before the gateway forcibly closes it, and is that closing event guaranteed to be logged even when it doesn't fall on the periodic log-sampling boundary?Q108hardgateway-flood-guard96,98320,36079.0%correct1.004
The before_compaction and after_compaction plugin hooks get a 30-second default timeout in openclaw's hook runner, the same as agent_end, instead of the tighter budget used by other modifying hooks. What specific mechanism in the codex agent harness does the code cite as the reason a hung compaction handler would be especially dangerous?Q127extremeplugin-hooks126,30527,30378.4%correct1.005
The gateway rejects a worker's connect request with reason "rpc-set-mismatch" under two distinct conditions, not just when the worker's declared rpcSetVersion disagrees with the stored credential. What is the second condition, and which of the initial admission checks (bundle hash, version, session, owner epoch, rpc set, protocol features) is never re-checked again once the worker is admitted and starts sending heartbeats/RPCs?Q117extremeworker-admission132,22531,90675.9%incorrect0.006

Tip: link directly to a question with a URL hash, e.g. #question-Q054.

Want to see it on your repository?

These numbers come from one public codebase. Connect your own to see what CodeMesh does for it.

Start Saving TokensSee the Benchmark