Skip to content
Casualstack Docs

Welcome

AgentBench OS is a CI gate for AI coding agent output. It answers one question with a hard pass or fail: did this agent run violate the constraints you set, or not.

Source: https://github.com/casualstack/agentbench-os (MIT license, v0.1.0).

The problem

Section titled “The problem”

AI coding agents produce diffs fast, and the diffs frequently look correct without being correct. An agent asked to fix a failing test can make the test pass by deleting the assertion instead of fixing the code. An agent asked to touch one file can quietly edit a dozen. An agent debugging offline can run pip install mid-task and change what “passing” even means. None of this shows up in a code review that only reads the final diff, because the diff is exactly what the agent wants you to see.

AgentBench OS exists to check the process, not just the artifact: what the agent actually did, step by step, against rules you can write down and run in CI.

Three core ideas

Section titled “Three core ideas”

Property oracles. An oracle is a small, deterministic check against a recorded agent run: “the test suite still exits 0,” “this file is byte-identical to before,” “no network access happened,” “this regex still matches.” Oracles do not evaluate quality or style. They check one property and return pass or fail, with a message. See Writing Oracles for the oracle types that ship today (test_must_pass, file_not_modified, no_network, assertion_exists) and how to add your own.

Deterministic gates. A gate runs a set of oracles against a recorded trajectory (a JSON log of the agent’s tool calls) replayed into a clean workspace, and fails the moment any oracle fails. There is no partial credit and no score to interpret. agentbench gate exits 1 if any task fails, which is what branch protection rules and CI jobs need. See CI Integration.

Session watching. Gates need a recorded trajectory ahead of time. Watch mode does not: agentbench watch finds the AI coding sessions already on your machine (Claude Code, Cursor, Codex CLI, and detects Antigravity), tails them as they happen, and raises plain-English alerts the moment an agent does something like weaken an assertion or write to a .env file. Zero configuration, nothing leaves your machine. See Watch Mode.

What AgentBench OS is not

Section titled “What AgentBench OS is not”
  • It is not an eval score. There is a benchmark matrix runner (agentbench matrix) for comparing pass rates across models and prompts, but every underlying check is still a hard pass/fail oracle. AgentBench never produces a fuzzy quality number for a single run; it produces PASS or FAIL.
  • It is not an LLM judge. No oracle asks a model whether code “looks right.” Checks are regex matches, file diffs, and shell command exit codes. The tradeoff is real: oracles catch what you thought to check for, nothing more, and nothing less predictably.
  • It is not a live agent runner. The MVP evaluates a recorded trajectory (a JSON file of tool calls) against a task’s initial workspace. It does not drive Cursor or Claude Code itself. Watch mode is the exception that observes agents live, but even there it reads session logs rather than controlling the agent.

Where to go next

Section titled “Where to go next”

Repository: https://github.com/casualstack/agentbench-os