Prompting

How do you write a good judge prompt?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 11 min read

How do you write a good judge prompt? A good judge prompt gives the judge model one narrow criterion, a written scale that defines each score, a request to reason before it scores and a fixed output format. The mistake that costs the point is trusting one ordering of the answers, and the third step below shows how often a judge flips when the order is reversed.

In this article

Before you start
Why interviewers ask this
Judge prompts from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page sits in the prompt engineering learning path and answers one question from it.

Why interviewers ask this

A judge prompt is how teams grade model outputs that have no single correct string, so a question about it shows whether you can measure quality at all. Interviewers use it to separate candidates who can write a rubric from candidates who also check whether the rubric works.

A weak answer says "I ask a generative pre-trained transformer (GPT) model to rate the answer from one to ten." A strong answer gives one criterion, defines the scale, asks for reasoning before the score and fixes the output format. A Staff-level answer adds the evidence step: run the judge on examples that humans already labeled, report agreement beyond chance, and test for position and length effects before trusting any number it produces. More questions of this kind are collected in prompt engineering interview questions and answers.

Judge prompts from first principles

Judge prompt: A judge prompt is the written instruction given to a judge model that states what to evaluate, how each score is defined, what the output must look like and which material to judge, so that the same model can grade many outputs in a consistent way.

In plain words

Think of a teacher with a marking scheme. Each essay is graded against the same written criteria, so two essays get comparable marks. A judge prompt is that marking scheme for a model. The analogy breaks in one place. A teacher is rarely swayed by which essay sits on top of the pile. A model sometimes is.

A strong large language model (LLM) that grades the outputs of another model on open-ended questions where fixed benchmarks fall short is called LLM judge (Zheng et al., 2023). It is needed because word-overlap scores such as Bilingual Evaluation Understudy (BLEU) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) correlate poorly with human judgment on tasks that need creativity and diversity (Liu et al., 2023).

Step 1: Write one criterion and define each score

A judge prompt should test one criterion, such as "every claim is supported by the supplied document", and define each score in words. A prompt that asks whether an answer is "good" mixes accuracy, tone and length, so the verdict cannot be debugged.

Input: the criterion, the scale and the material. Output: a rule the judge can apply. Pass or fail is the simplest scale; a short scale with a sentence per level is the next. The sources used for this page do not compare scale sizes, so treat the scale as a design choice you test, not a proven rule. Here is a template:

You are grading one answer against one criterion.
Criterion: every claim in the answer is supported by the document.
Pass: all claims are supported. Fail: at least one claim is not.
Length and style are not part of the criterion.
Document: {document}
Answer: {answer}
Think step by step, then reply with JSON only:
{"reasoning": "<one paragraph>", "assessment": "pass" | "fail"}

The reasoning field comes first so the assessment follows the reasoning, as Step 2 explains.

You now have a rubric the judge cannot misread. One criterion and a defined scale come first.

Step 2: Ask for reasoning, then a verdict in a fixed format

The judge should write its reasoning first and give the verdict in a fixed, machine-readable format afterwards. G-Eval is a framework that uses large language models with chain-of-thought and a form-filling paradigm to assess the quality of natural language generation (NLG) outputs. With Generative Pre-trained Transformer 4 (GPT-4) as the backbone it reaches a Spearman correlation of 0.514 with humans on a summarization task. Spearman correlation compares two rankings, where one means the same order, so a good judge still leaves room for disagreement.

Input: the filled prompt. Output: a reasoning paragraph and a verdict field. Put the reasoning before the verdict, because a verdict written first cannot be justified by later reasoning. Parse the verdict with a schema; see getting reliable JavaScript Object Notation (JSON) from an LLM. The judge reasons before it gives a verdict in a fixed format.

Step 3: Remove position bias by swapping the answers

For pairwise comparisons, run each pair in both orders and keep only the verdicts that agree. Zheng et al. (2023) examine position bias, verbosity bias, self-enhancement bias and limited reasoning ability in LLM judges, and show that some of the biases are minor or can be mitigated (Zheng et al., 2023). Liu et al. (2023) flag a possible bias toward LLM-written text.

Prompt wording matters too. The first paper's position-bias table reports 23.8% consistency for Claude-v1 with the default prompt and 56.2% with a prompt the authors call rename, which renames the assistants in the default prompt to test whether the bias is about positions or names (Zheng et al., 2023).

In an illustrative run on 20 invented answer pairs, the judge picked the same underlying answer in both orders for 14 pairs, a consistency of 0.70.¹ In the other 6 pairs it chose the first slot five times and the second slot once, so the order alone decided those verdicts.¹ Running both orders exposes position bias. For length bias, state in the prompt that length is not quality.

Step 4: Check the judge against human labels

A judge is trusted only after it is compared with human labels on the same items. Zheng et al. (2023) report that strong judges such as GPT-4 achieve over 80% agreement with humans, across 3,000 controlled expert votes and 3,000 crowdsourced votes, the same level as human-human agreement (Zheng et al., 2023). Measure judges before trusting them.

Here is an illustrative check on 20 invented items that humans labeled, 12 pass and 8 fail.¹ A vague prompt ("Is this answer good?") agreed with the humans on 13 items, a raw agreement of 0.65, and a kappa of 0.186, because it passed 17 items.¹ The rubric prompt agreed on 16 items, a raw agreement of 0.80, and a kappa of 0.5833.¹

On the same 20 invented items, the rubric prompt beats the vague prompt on raw agreement and on Cohen's kappa, and it passes far fewer items.
Prompt	Raw agreement	Cohen's kappa	Items passed (of 20)
Vague ("Is this answer good?")	0.65	0.186	17
Rubric (one criterion, defined scale)	0.80	0.5833	12

The rubric prompt raises raw agreement from 0.65 to 0.80, and Cohen's kappa rises from 0.186 to 0.5833.¹

With only 20 items, the 95% Wilson interval for the rubric prompt's raw agreement runs from about 0.58 to 0.92, so a small label set cannot separate two close prompts.¹

On the same invented items, a rubric prompt lifts both raw agreement and Cohen's kappa over a vague one.CC BY 4.0 — source: How do you write a good judge prompt?

Text description of the diagram

The rubric prompt raises raw agreement from 0.65 to 0.80.
Cohen's kappa rises from 0.186 to 0.5833.

A trustworthy judge needs all four stages: a narrow rubric, reasoning, a position check and a comparison with human labels.CC BY 4.0 — source: How do you write a good judge prompt?

Step-by-step description

One criterion, each score defined.
Reasoning first, then a parseable verdict.
Run both answer orders.
Compare with human labels.

One criterion and a defined scale come first.
The judge reasons before it gives a verdict in a fixed format.
Running both orders exposes position bias.
Human labels show whether the judge can be trusted.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.