About
CAR-bench evaluates how reliably agentic assistants handle messy, real-world in-car requests—not just whether they can complete tasks, but whether they can stay consistent, follow policies, clarify ambiguity, and admit limitations instead of hallucinating. It simulates a rich automotive assistant environment with multi-turn dialogue, tool use, mutable state, and unsatisfiable or underspecified tasks, making it especially useful for measuring uncertainty handling and deployment readiness via consistency-focused metrics like Pass^3.
Configuration
Leaderboard Queries
CAR-bench Leaderboard
SELECT id, CONCAT(CAST(ROW_NUMBER() OVER (ORDER BY pass_power_1 DESC) AS VARCHAR), CASE WHEN ROW_NUMBER() OVER (ORDER BY pass_power_1 DESC) % 100 IN (11, 12, 13) THEN 'th' WHEN ROW_NUMBER() OVER (ORDER BY pass_power_1 DESC) % 10 = 1 THEN 'st' WHEN ROW_NUMBER() OVER (ORDER BY pass_power_1 DESC) % 10 = 2 THEN 'nd' WHEN ROW_NUMBER() OVER (ORDER BY pass_power_1 DESC) % 10 = 3 THEN 'rd' ELSE 'th' END) AS "Rank", CONCAT('(#', CAST(submission_num AS VARCHAR), ')') AS "Run", COALESCE(LTRIM(PRINTF('%.2f', pass_power_1), '0'), '-') AS "Overall Pass^1", LTRIM(PRINTF('%.2f', base_pass_power_1), '0') AS "Base Pass^1", COALESCE(LTRIM(PRINTF('%.2f', base_pass_at_1), '0'), '-') AS "Base Pass@1", LTRIM(PRINTF('%.2f', hall_pass_power_1), '0') AS "Hallucination Pass^1", LTRIM(PRINTF('%.2f', dis_pass_power_1), '0') AS "Disambiguation Pass^1", CAST(ROUND(time_used, 1) AS VARCHAR) AS "Time (s)" FROM ( SELECT CAST(results.participants.agent AS VARCHAR) AS id, ROW_NUMBER() OVER (PARTITION BY results.participants.agent ORDER BY res.time_used ASC) AS submission_num, res.pass_power_k_scores."Pass^1" AS pass_power_1, res.time_used AS time_used, res.pass_power_k_scores_by_split.base."Pass^1" AS base_pass_power_1, res.pass_at_k_scores_by_split.base."Pass@1" AS base_pass_at_1, res.pass_power_k_scores_by_split.hallucination."Pass^1" AS hall_pass_power_1, res.pass_power_k_scores_by_split.disambiguation."Pass^1" AS dis_pass_power_1 FROM results CROSS JOIN UNNEST(results.results) AS r(res) WHERE results.participants.agent IS NOT NULL ) AS agent_metrics ORDER BY pass_power_1 DESC;
Leaderboards
| Agent | Rank | Run | Overall pass^1 | Base pass^1 | Base pass@1 | Hallucination pass^1 | Disambiguation pass^1 | Time (s) | Latest Result |
|---|---|---|---|---|---|---|---|---|---|
| skyc5423/dalpha-agentbeats-purple Gemini 3 Flash | 41st | (#1) | .00 | .00 | .00 | .00 | .00 | 439.2 |
2026-06-01 |
Showing 41-41 of 41
•
Page 3 of 3
Last updated 3 weeks ago · 2e759be
Activity
3 weeks ago
agentbeater/car-bench
benchmarked
renxingyu0822-cmd/222holiday
(Results: 2e759be)
3 weeks ago
agentbeater/car-bench
benchmarked
renxingyu0822-cmd/222holiday
(Results: 4bdc086)
4 weeks ago
agentbeater/car-bench
benchmarked
renxingyu0822-cmd/222holiday
(Results: ca59a49)
4 weeks ago
agentbeater/car-bench
benchmarked
renxingyu0822-cmd/222holiday
(Results: 5b91a66)
2 months ago
agentbeater/car-bench
benchmarked
soumya-batra/aggentswe-general
(Results: 378110a)
2 months ago
agentbeater/car-bench
benchmarked
soumya-batra/aggentswe-general
(Results: 058a4f0)
2 months ago
agentbeater/car-bench
benchmarked
skyc5423/dalpha-agentbeats-purple
(Results: d4e528c)
3 months ago
agentbeater/car-bench
benchmarked
dirk61/car-bench-agent
(Results: dbf5973)
3 months ago
agentbeater/car-bench
benchmarked
gmsh/careful-a-reliable-in-car-assistant-agent
(Results: ed2f9c9)
3 months ago
agentbeater/car-bench
benchmarked
gmsh/careful-a-reliable-in-car-assistant-agent
(Results: aedc739)