Benchmark report

Task verified

4m 33s elapsed

The saved attempt met the documented success criterion.

Experience score

85/ 100

Agent-Ready
  • AInstall
  • AAuth
  • SExecute
  • ADocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth A, Execute S, Docs A, Current AAInstallAAuthSExecuteADocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
The POST /projects 400 'org_id is required' forced an extra organizations lookup and a code edit before the project could be created.

Where the time went

4:34 agent time
  • Research0:19
  • Attempt2:47
  • Verify1:27
4m 33sTotal elapsed
1:33Time to first success
0:00Waited on the owner

Judge reason

Saved code and runtime output prove the full task against the real Neon API. code/01_create_project.py POSTed /projects (outputs/01_create_project.json, HTTP 201, project tpcbench-a47bfdd7a5 / crimson-poetry-30285007, password+connection_uri redacted). code/02_sql_parameterized.py connected with psycopg 3.3.9 using sslmode=verify-full and sslrootcert=system (saved connection_parameters, host ep-morning-fire-ak3iggn2.c-3.us-west-2.aws.neon.tech) and succeeded: CREATE TABLE items (id integer PRIMARY KEY, name text), three INSERT INTO items (id, name) VALUES (%s, %s) with bound tuples (1,alpha),(2,beta),(3,gamma) each returning INSERT 0 1, then SELECT id, name FROM items WHERE id >= %s ORDER BY id with param (2,) printing exactly (2, 'beta') then (3, 'gamma') in outputs/02_sql_stdout.txt and matching outputs/02_sql_result.json. No string concatenation anywhere. pg_stat_ssl ssl=f is the documented pooler artifact, not a failed TLS connect. Cleanup: DELETE /projects/crimson-poetry-30285007 -> HTTP 200 (outputs/03_delete_project.json), follow-up GET -> 404 'project not found' (outputs/04_followup_get_project.json plus an independent re-check in 04_followup_get_recheck.txt), and the project list shows deleted_project_id_present=false with zero tpcbench-* projects remaining. A grep over code/, outputs/ and evidence/ found no API key, DB password or password-bearing URI; the connection string lived only in a 0600 file passed via DATABASE_URL and was shredded. One product quirk: a TCP connect to the old compute endpoint still succeeded moments after deletion (outputs/06_followup_connect_after_delete.txt), which the agent reported honestly; the authoritative management-API reads confirm deletion.

Attempt record

  1. 0:42Answered from saved key: NEON_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.