Benchmark report

Task verified

4m 36s elapsed

The saved attempt met the documented success criterion.

Experience score

90/ 100

Agent-Ready
  • SInstall
  • SAuth
  • AExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute A, Docs S, Current SSInstallSAuthAExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The crawl response exposes no page title field, so titles had to be derived from raw markdown, costing one failed extraction pass and a re-run.

Where the time went

4:36 agent time
  • Research0:32
  • Attempt2:22
  • Verify1:42
4m 36sTotal elapsed
1:43Time to first success
0:00Waited on the owner

Judge reason

Saved evidence proves a live, single, section-restricted Tavily crawl. crawl_request.json shows POST https://api.tavily.com/crawl with url=https://www.postgresql.org/docs/17/tutorial.html, limit=25 (set in source before the call, never lowered), and select_paths=["^/docs/17/tutorial.*"] — documented in the fetched API reference as "Include only URLs matching these path patterns", a hard include filter, not the ranking-only `instructions` parameter. crawl_response_meta.json records HTTP 200 and a live Date header; crawl_response.json carries request_id 35f621f1-aea4-4c63-9a9a-d12f7c84e27f, usage {credits: 7}, and 23 distinct results, all under the tutorial prefix (I re-parsed it: 23 results, 23 distinct URLs, 0 off-section). raw_content is genuine postgresql.org markdown. I independently re-extracted each page's title from its own embedded "PostgreSQL 17 - ..." link metadata and all 23 matched pages.json and crawl_stdout.txt exactly. One synchronous product-side job; no per-page fetching or local post-filtering in crawl_tutorial.py.

Attempt record

  1. 1:14Answered from saved key: Tavily API keyno wait
  2. —First success verified

Attempt artifacts saved.