Benchmark report

Task verified

10m 11s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • BCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current BSInstallSAuthSExecuteSDocsBCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The /docs/cli/logs quoting rule contradicted the API: 12 polls (~70s) all failed with 'Quoted phrase search is not supported' before the bare-word query worked.

Where the time went

10:12 agent time
  • Research0:23
  • Attempt7:25
  • Verify2:24
10m 11sTotal elapsed
4:53Time to first success
0:00Waited on the owner

Judge reason

Independently verified against saved artifacts. Deploy: evidence/03-deploy.stdout shows readyState READY, target production, productionUrl https://tpcbench-20261109145925-f883f63e.vercel.app on Vercel's own domain (project created via `vercel deploy --prod --yes --scope tpc-cao --project tpcbench-... --create-if-missing`). Code work/app/api/index.js logs one real line `TPC_MARKER_LOG marker=${x-tpc-marker}` — no mocking or hard-coding. One marked request: evidence/04-marker-request.http returns HTTP/2 200 with x-vercel-id pdx1::iad1::9clll-1794236412994-e54d94b30e12 and body marker tpcmark-20261109145925-f883f63e-5104896d0b71. Log retrieval is from Vercel's own logs via the documented CLI (`vercel logs runtime --project ... --scope tpc-cao --query <marker> --since 30m --json`): evidence/08-logs-query-marker.jsonl returns recordsReturned 1 whose requestId 9clll-1794236412994-e54d94b30e12 matches the curl response header exactly, message `TPC_MARKER_LOG marker=tpcmark-20261109145925-f883f63e-5104896d0b71`, domain/environment production; evidence/06-logs-all.jsonl is the unfiltered corroborating fetch and 09-logs-query-table.txt repeats it. That exact line is printed in the final report. Retention: runtime logs on by default, team on Pro (evidence/10-team-plan.json billing_plan pro) — correctly stated as no opt-in, citing /docs/logs/runtime#limits. Team resolved via documented API (/v2/user, /v2/teams). Cleanup: evidence/11-project-delete.stderr '✓ Deleted', 13-project-lookup-after.err 'Project not found. (404)', transcript shows /v9/projects listing with 'NONE - no tpcbench-* projects remain', and evidence/14-final-fetch.http is HTTP/2 404 x-vercel-error DEPLOYMENT_NOT_FOUND (15-final-status.txt 'HTTP 404'). Only resource created is named tpcbench-20261109145925-f883f63e. Secret hygiene holds: token loaded into VERCEL_TOKEN by a wrapper reading the private file; my own scan of every saved artifact found no bearer/access token (only project/team/user IDs and an inviteCode in the teams API body), and the agent's grep for the token value returned 0 files.

Attempt record

  1. 1:44Answered from saved key: VERCEL_TOKENno wait
  2. —First success verified

Attempt artifacts saved.