Benchmark report

Task verified

7m 2s elapsed

The saved attempt met the documented success criterion.

Experience score

86/ 100

Agent-Ready
  • SInstall
  • SAuth
  • AExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute A, Docs A, Current SSInstallSAuthAExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
No doc page lists delivery-status labels or a per-message delivered/bounced pull read, so the agent explored several pages and polled 60s for a state change that never came.

Where the time went

7:02 agent time
  • Starting0:01
  • Research0:34
  • Attempt4:54
  • Verify1:34
7m 2sTotal elapsed
3:46Time to first success
0:00Waited on the owner

Judge reason

Saved runnable code (agentmail-task/run_task.py, byte-identical copy in evidence/) ran for real: console_output.txt and full_run.json show GET /v0/domains->200 {count:0}, two inboxes created via POST /v0/inboxes, then exactly one send POST /v0/inboxes/bench-sender-1793603399@agentmail.to/messages/send -> 200 returning message_id <010001a19b17f121-e4242d65-f637-44a7-8608-00d215056b22-000000@email.amazonses.com>. Pull reads followed by that identifier, no webhook/dashboard: GET /v0/inboxes/{inbox}/messages/{message_id} -> 200 with state field labels=["sent"] and timestamp/updated_at 2026-11-02T07:09:59.910Z (polled 12x over ~60s), and GET /v0/inboxes/{inbox}/events -> 200 yielding event_id 01M6DHFWA0Y6FRD49MZ93E0000, event_type label.added, label sent, event_at 2026-11-02T07:10:00.000Z, filtered client-side on the returned message_id; plus GET /v0/metrics/events over the send window (message.sent 1, message.delivered 1, bounced 0). result.json reproduces these values exactly (verified field-by-field against full_run.json/get_message_response.json/inbox_events_response.json) and honestly states the documented gap: AgentMail exposes message.delivered/message.bounced only via webhook transports, the inbox events schema is limited to label.added/label.removed, and metrics events are aggregate buckets with no message_id filter. No mocks or hard-coded data; recipient was an inbox created for this run through the product's own API.

Attempt record

  1. 2:24Answered from saved key: AgentMail API keyno wait
  2. —First success verified

Attempt artifacts saved.