---
title: "LLM as a Judge"
description: "Evaluating tasks performed by LLMs can be difficult due to their complexity and the diverse criteria involved. Traditional methods like rule-based assessment or similarity metrics (e.g., ROUGE, BLEU) often fall short when applied to the nuanced and varied outputs of LLMs."
---

<Frame>
![](https://storage.googleapis.com/arize-phoenix-assets/assets/images/PX%20docs%20graphic.jpg)
</Frame>

<Note>
  LLM as a Judge is a general evaluation concept that applies to both evaluation approaches in Phoenix. You can use it via the SDK (client-side) or configure LLM evaluators directly in the Phoenix UI (server-side).
</Note>


For instance, an AI assistant’s answer to a question can be:

* not grounded in context
* repetitive, repetitive, repetitive
* grammatically incorrect
* excessively lengthy and characterized by an overabundance of words
* incoherent

The list of criteria goes on. And even if we had a limited list, each of these would be hard to measure

To overcome this challenge, the concept of "LLM as a Judge" employs an LLM to evaluate another's output, combining human-like assessment with machine efficiency.

## How It Works

Here’s the step-by-step process for using an LLM as a judge:

<Steps>
  <Step title="Identify Evaluation Criteria">
   First, determine what you want to evaluate, be it faithfulness, toxicity, accuracy, or another characteristic. See our [pre-built evaluators](/docs/phoenix/evaluation/pre-built-metrics) for examples of what can be assessed.
  </Step>
  <Step title="Craft Your Evaluation Prompt">
   Write a prompt template that will guide the evaluation. This template should clearly define what variables are needed from both the initial prompt and the LLM's response to effectively assess the output.
  </Step>
  <Step title="Select an Evaluation LLM">
   Choose the most suitable LLM from our available options for conducting your specific evaluations.
  </Step>
  <Step title="Generate Evaluations and View Results">

   Execute the evaluations across your data. This process allows for comprehensive testing without the need for manual annotation, enabling you to iterate quickly and refine your LLM's prompts.
  </Step>
</Steps>

Using an LLM as a judge significantly enhances the scalability and efficiency of the evaluation process. By employing this method, you can run thousands of evaluations across curated data without the need for human annotation.

This capability will not only speed up the iteration process for refining your LLM's prompts but will also ensure that you can deploy your models to production with confidence.

## Using LLM as a Judge in Phoenix

<CardGroup cols={2}>
  <Card title="SDK Evaluations" icon="code" href="/docs/phoenix/evaluation/how-to-evals/custom-llm-evaluators">
    Write custom LLM evaluators in Python or TypeScript. See also: [Configuring the LLM](/docs/phoenix/evaluation/how-to-evals/configuring-the-llm) for model selection and prompt setup.
  </Card>
  <Card title="Server-Side Evaluators" icon="browser" href="/docs/phoenix/evaluation/server-evals/llm-evaluators">
    Configure LLM evaluators in the Phoenix UI — no local code or API key setup required.
  </Card>
</CardGroup>

## Additional Resources

<CardGroup cols={2}>
  <Card title="LLM as a Judge" href="https://arize.com/llm-as-a-judge/" icon="gavel" horizontal>
    Arize AI
  </Card>
  <Card title="LLM as a Jury:How To Implement" href="https://arize.com/llm-as-a-jury/" icon="scale-balanced" horizontal>
    Arize AI
  </Card>
</CardGroup>
