Agent-Driven Benchmark Quickstart

This page is a compact quickstart. The complete guide is benchmark/manual.md.

Prerequisites

1. Python environment:

   cd benchmark
   uv sync

2. Judge API key if judge is enabled:

   export OPENROUTER_API_KEY=...

3. Either an existing agent daemon, or an environment bridge plus --auto-agent-setup.

Run Against An Existing Agent

cd benchmark

uv run python -m runner run \
  --suite suites/phone_control_v1.json \
  --agent-url http://127.0.0.1:8080

Skip judge for fast trace collection:

uv run python -m runner run \
  --suite suites/phone_control_v1.json \
  --agent-url http://127.0.0.1:8080 \
  --no-judge

Run selected tasks:

uv run python -m runner run \
  --suite suites/phone_control_v1.json \
  --task-id open_settings \
  --task-id scroll_page_down

Run With MobileGym

Start a MobileGym environment bridge:

uv run python -m runner start-mobilegym-env --envs 5 --bridge-port 19090

Run the suite with automatic isolated agent workers:

uv run python -m runner run \
  --suite suites/mobilegym_basic.json \
  --environment-url http://127.0.0.1:19090 \
  --auto-agent-setup

The runner reads bridge capacity from /api/concurrent. If the suite has more tasks than the bridge capacity, extra tasks wait in the queue.

WebUI

cd benchmark
uv run python -m runner webui

Open http://127.0.0.1:8765. The WebUI is the recommended path for routine MobileGym concurrency because it shows per-task worker status, screens, logs, and persisted job records.

Output

Each CLI run creates runs/<run_id>/:

<run_id>/
├── manifest.json
├── results.jsonl
├── summary.md
├── report.html
└── tasks/<task_id>/
    ├── pre.jpg
    ├── post.jpg
    ├── history.json
    ├── trace.json
    └── judge.json

pre.jpg and post.jpg are present when --environment-url is configured and the bridge returns screenshots through /api/screen. Judge uses those two images plus trace/final response.

Rejudge

Change rubric phrasing or judge model without re-running on hardware:

uv run python -m runner rejudge --run-dir runs/<id> --judge-model claude-sonnet-4-6

Compare Runs

uv run python -m runner compare --runs runs/<id_a> runs/<id_b>

Legacy

The previous benchmark (benchmark/suites/full_smoke.json) is deprecated and retained for reference only. The legacy entry point scripts/aiden_benchmark.py forwards to the new runner.