This is the number a CI gate would block a merge on. Pick it before you run, not after you see the result — otherwise you are fitting the gate to the prompt.
Judges drift, agree with themselves, and reward verbosity. Use tab 4 to measure the judge against your own labels before you trust a single judged score.
{{variable}} placeholders are filled from each test case's inputs.
Leave empty to evaluate A alone. Fill it to get a per-case A/B regression table.
| contains | Output contains this substring (case-insensitive). |
| not_contains | Output must not contain it. Good for leaked PII, banned phrases, pricing or SLA promises. |
| equals | Exact match after trimming. |
| regex | JavaScript regex, tested against the whole output. |
| json_valid | Output parses as JSON (fenced code blocks are stripped first). Value ignored. |
| json_has_keys | Output is JSON and has every comma-separated key at the top level. |
| max_words | Output is at most N words. Cheap guard against verbosity creep. |
| llm_judge | A model scores the output against your criterion. Calibrate it in tab 4 before trusting it. |
An LLM judge is itself a prompt, and it needs its own eval. Label each judged output by hand below, then compare. If agreement is under ~85%, the judged portion of your score is noise — fix the judge rubric or replace it with a deterministic assertion.
Run a suite containing at least one llm_judge assertion, then come back here.