Skip to main content
The judge is an optional LLM check that runs after a V4 run finishes. It reads the task, a summary of the agent’s trajectory (its steps, tool calls and results, with long parts shortened and very long runs cut off), the final answer and up to 20 screenshots, and returns a verdict: did the agent complete the task, in full and in good faith? Turn it on by passing judge when you create a run (judge: {} uses the defaults). Omitting judge, or setting it to null, leaves it off.

What the judge checks

The judge grades the outcome against what you asked for. There is no partial credit: a run that does only part of the task fails. It is told to look for these failure modes and name them in failure_reason:
  • The task was impossible, or the browser or agent crashed or stalled.
  • A captcha or anti-bot check blocked the agent.
  • The agent pivoted to a different website than the task named.
  • The agent did only part of the task but reported success.
  • The final answer is wrong according to your context, the trace or the screenshots.
  • Reward hacking: the agent exploited a loophole or did not complete the task in good faith.
  • The agent took an action the task did not allow, such as submitting a form it was only asked to fill in.
  • The agent ran out of time before returning an answer.
Prompt injection shows up through these checks. If a page tells the agent to drop its task, open another site or report a different answer, and the agent does so, the result no longer matches your task, so the judge should fail the run. To have the judge name it, say so in context (below).
The judge is an LLM, so its verdict is not deterministic and can be wrong. Use it to flag runs for review and to measure success across many runs, not as a security boundary. To limit where the browser can go, use network policies.

Add your own criteria

judge.context (up to 10,000 characters) gives the judge your evaluation criteria, the expected answer or the failure reasons you want reported. The judge is told that it takes precedence over everything else: a run that does not satisfy it should get verdict: false. When one of your suggested failure reasons matches, the judge is asked to use your wording, so results group cleanly.

Read the verdict

The verdict lands on the run as judgement shortly after the run becomes terminal; until then it is null. See How do I get a V4 judge verdict? for a polling example.
A completed run status only means the agent finished; it is not a verdict. A judgement that stays null is unavailable, not a pass or a fail.

Model and cost

Judge model choice is being rolled out. Once it is enabled for your project, the default judge model is gpt-6.1-terra with reasoning effort high, and judge.model and judge.modelParams choose another OpenAI or Google model. Until then, and for projects whose model routing cannot use gpt-6.1-terra (for example US-only routing), judge: {} uses gemini-3-flash; a non-null judge.model or judge.modelParams returns 422 before rollout. The FAQ covers the details. The judge’s LLM call is billed to the run at the judge model’s token prices.

When the judge does not run

The judge does not run for projects with Zero Data Retention or projects restricted to AWS Bedrock model routing, because it would send the trajectory to another model provider. Their runs keep judgement: null. The judge is not available in the EU deployment: a run created there with judge returns 422.