judge when you create a run (judge: {} uses the
defaults). Omitting judge, or setting it to null, leaves it off.
What the judge checks
The judge grades the outcome against what you asked for. There is no partial credit: a run that does only part of the task fails. It is told to look for these failure modes and name them infailure_reason:
- The task was impossible, or the browser or agent crashed or stalled.
- A captcha or anti-bot check blocked the agent.
- The agent pivoted to a different website than the task named.
- The agent did only part of the task but reported success.
- The final answer is wrong according to your
context, the trace or the screenshots. - Reward hacking: the agent exploited a loophole or did not complete the task in good faith.
- The agent took an action the task did not allow, such as submitting a form it was only asked to fill in.
- The agent ran out of time before returning an answer.
context (below).
The judge is an LLM, so its verdict is not deterministic and can be wrong. Use
it to flag runs for review and to measure success across many runs, not as a
security boundary. To limit where the browser can go, use
network policies.
Add your own criteria
judge.context (up to 10,000 characters) gives the judge your evaluation
criteria, the expected answer or the failure reasons you want reported. The
judge is told that it takes precedence over everything else: a run that does
not satisfy it should get verdict: false. When one of your suggested failure
reasons matches, the judge is asked to use your wording, so results group
cleanly.
Read the verdict
The verdict lands on the run asjudgement shortly after the run becomes
terminal; until then it is null. See
How do I get a V4 judge verdict?
for a polling example.
A
completed run status only means the agent finished; it is not a verdict.
A judgement that stays null is unavailable, not a pass or a fail.
Model and cost
Judge model choice is being rolled out. Once it is enabled for your project, the default judge model isgpt-6.1-terra with reasoning effort high, and
judge.model and judge.modelParams choose another OpenAI or Google model.
Until then, and for projects whose model routing cannot use gpt-6.1-terra
(for example US-only routing), judge: {} uses gemini-3-flash; a non-null
judge.model or judge.modelParams returns 422 before rollout. The
FAQ covers the details. The
judge’s LLM call is billed to the run at the judge model’s token prices.
When the judge does not run
The judge does not run for projects with Zero Data Retention or projects restricted to AWS Bedrock model routing, because it would send the trajectory to another model provider. Their runs keepjudgement: null.
The judge is not available in the EU deployment: a run
created there with judge returns 422.