---
title: "Measure Performance with Evaluations"
---

An evaluation produces a score or label for an output, so you can track quality across runs. Evaluations attach quality signals to runs so that correctness or relevance can be reasoned about consistently instead of judged case by case. Traces tell us what happened during a run, but they don't tell us whether the output was good; evaluations fill that gap by letting us score outputs in a consistent, repeatable way.

In this guide, you’ll set up evaluations in Phoenix and run them on existing trace data so you can measure the quality of model outputs from a real application.

We’ll start with data that already exists in Phoenix, define a simple evaluation, and run it so we can see results directly in the UI. The goal is to move from “I have model outputs” to “I can measure quality in a repeatable way.”

Since we already have traces, we can take this a step further by scoring them against metrics like correctness, relevance, or custom checks that matter to your use case.

## **Before We Start**

To follow along, you’ll need to have completed [Get Started with Tracing](https://arize.com/docs/phoenix/get-started/get-started-tracing) which means we have:

- Financial Analysis and Research Chatbot
- Trace Data in Phoenix

---

<Info>
  **Follow along with code**: This guide has a companion notebook with runnable
  code examples. Find it
  [here](https://github.com/Arize-ai/phoenix/blob/main/tutorials/quickstarts/python_quickstart.ipynb).
</Info>

---

<Steps>
  <Step title={<span className="step-title">Make Sure You Have Data in Phoenix</span>}>
    Before we can run evaluations, we need something to evaluate.

    Evaluations in Phoenix run over existing trace data. If you followed the tracing guide, you should already have:

    - A project in Phoenix
    - Traces containing LLM inputs and outputs

    It’s best to have multiple traces so we can see how evaluation results vary from run to run. If needed, run your agent a few times with different inputs to generate more data.

    ```python
    test_queries = [
        {"tickers": "AAPL", "focus": "financial analysis and market outlook"},
        {"tickers": "NVDA", "focus": "valuation metrics and growth prospects"},
        {"tickers": "AMZN", "focus": "profitability and market share"},
        {"tickers": "AAPL, MSFT", "focus": "comparative financial analysis"},
        {"tickers": "META, SNAP, PINS", "focus": "social media sector trends"},
        {"tickers": "RIVN", "focus": "financial health and viability"},
        {"tickers": "SNOW", "focus": "revenue growth trajectory"},
        {"tickers": "KO", "focus": "dividend yield and stability"},
        {"tickers": "META", "focus": "latest developments and stock performance"},
        {"tickers": "AAPL, MSFT, GOOGL, AMZN, META", "focus": "big tech comparison and market outlook"},
        {"tickers": "AMC", "focus": "financial analysis and market sentiment"},
    ]

    for query in test_queries:
        crew.kickoff(inputs=query)
    ```
  </Step>

  <Step title={<span className="step-title">Define an Evaluation</span>}>
    Now that we have trace data, the next question is how we decide whether an output is actually good.

    An evaluation makes that decision explicit. Instead of manually inspecting outputs or relying on intuition, we define a rule that Phoenix can apply consistently across many runs.

    In Phoenix, evaluations can be written in different ways. In this guide, we’ll use an LLM-as-a-judge evaluation as a simple starting point. This works well for questions like correctness or relevance, and lets us get metrics quickly. (If you’d rather use code-based evaluations, you can follow [the guide](https://arize.com/docs/phoenix/evaluation/how-to-evals/code-evaluators#using-create-evaluator) on setting those up.)

    For LLM-as-a-judge evaluations, that means defining three things:

    - A prompt that describes the judgment criteria
    - An LLM that performs the evaluation
    - The data we want to score

    In this step, we'll define a basic completeness evaluation that checks whether the agent's output completely answers the input. Phoenix also provides [pre-built evaluation templates](https://arize.com/docs/phoenix/evaluation/running-pre-tested-evals) you can use or adapt for other metrics like relevance or faithfulness.

    ### **Define the Evaluation Prompt**

    We’ll start by defining the prompt that tells the evaluator how to judge an answer. We're using `attributes.input.value` & `attributes.output.value` as that is how our span data saves input & output.

    ```python
    financial_completeness_template = """
    You are evaluating whether a financial research report correctly completes ALL parts of the user's task with COMPREHENSIVE coverage.

    User input: {attributes.input.value}

    Generated report: {attributes.output.value}

    To be marked as "complete", the report MUST meet ALL of these strict requirements:

    1. TICKER COVERAGE (MANDATORY):
       - Cover ALL companies/tickers mentioned in the input
       - If multiple tickers are listed, EACH must have dedicated analysis (not just mentioned in passing)
       - For multiple tickers, the report must provide COMPARATIVE analysis when relevant

    2. FOCUS AREA COVERAGE (MANDATORY):
       - Address ALL focus areas mentioned in the input
       - If the focus mentions multiple topics (e.g., "earnings and outlook"), BOTH must be thoroughly addressed
       - Each focus area must have substantial content, not just a brief mention

    3. FINANCIAL DATA REQUIREMENTS (MANDATORY):
       - For EACH ticker, the report must include:
         * Current/recent stock price or performance data
         * At least 2 key financial ratios (P/E, P/B, debt-to-equity, ROE, etc.)
         * Revenue or earnings information
         * Recent news or developments (within last 6 months)
       - If focus mentions specific metrics (e.g., "P/E ratio"), those MUST be explicitly provided

    4. DEPTH REQUIREMENT (MANDATORY):
       - Each ticker must have at least 3-4 sentences of dedicated analysis
       - Generic statements without specific data do NOT count
       - The report must demonstrate thorough research, not superficial coverage

    5. COMPARISON REQUIREMENT (if multiple tickers):
       - If 2+ tickers are requested, the report MUST include direct comparisons
       - Comparisons should cover multiple key metrics side-by-side
       - Generic statements like "both companies are good" do NOT satisfy this requirement
       - Must explicitly state which company performs better/worse on specific metrics

    The report is "incomplete" if it fails ANY of the above requirements, including:
    - Missing any ticker or only mentioning it briefly
    - Failing to address any focus area or only addressing it superficially
    - Missing required financial data for any ticker
    - Providing generic analysis without specific metrics or data
    - Failing to provide comparisons when multiple tickers are requested
    - Not meeting the depth requirement for any ticker

    Respond with ONLY one word: "complete" or "incomplete"
    Then provide a detailed explanation of which specific requirements were met or failed.
    """
    ```

    This prompt defines what correctness means for our application.

    ### **Define the LLM Judge**

    Next, we’ll define the model that will act as the judge.

    ```python
    from phoenix.evals import LLM

    llm = LLM(model="gpt-4o", provider="openai")
    ```

    ### **Create the Evaluator**

    Now we can combine the prompt and model into an evaluator.

    ```python
    from phoenix.evals import ClassificationEvaluator

    completeness_evaluator = ClassificationEvaluator(
        name="completeness",
        prompt_template=financial_completeness_template,
        llm=llm,
        choices={"complete": 1.0, "incomplete": 0.0},
    )
    ```

    At this point, we’ve defined how Phoenix should evaluate correctness, but we haven’t run it yet.
  </Step>

  <Step title={<span className="step-title">Run the Evaluation</span>}>
    Next, we’ll pull our trace data from Phoenix and run the evaluator on it.

    First, fetch the spans we want to evaluate:

    ```python
    from phoenix.client import Client

    px_client = Client()
    df = px_client.spans.get_spans_dataframe(project_name="crewai-tracing-quickstart")
    parent_spans = df[df["span_kind"] == "CHAIN"]
    ```

    Then run the evaluator over that data. We are adding in `suppress_tracing()` since auto-instrumentation enabled and we do not want to trace each of these OpenAI evaluation calls in our project.

    ```python
    from phoenix.evals import async_evaluate_dataframe
    from phoenix.trace import suppress_tracing

    with suppress_tracing():
        results_df = await async_evaluate_dataframe(
            dataframe=parent_spans,
            evaluators=[completeness_evaluator],
            concurrency=10,
        )
    ```

    This produces evaluation results for each span in the dataset.
  </Step>

  <Step title={<span className="step-title">Log Evaluation Results to Phoenix</span>}>
    Finally, we’ll log the evaluation results back to Phoenix so they show up alongside our traces in the UI. This is what makes evaluations useful beyond a single run. Instead of living only in code, results become part of the same view you already use to understand behavior.

    ```python
    from phoenix.evals.utils import to_annotation_dataframe

    evaluations = to_annotation_dataframe(
        dataframe=results_df
    )

    Client().spans.log_span_annotations_dataframe(
        dataframe=evaluations
    )
    ```

    Once this completes, head back to Phoenix.

    You’ll now see evaluation results attached to your trace data in the annotations column, making it easy to understand which runs passed, which failed, and how quality varies across executions.

    <Frame caption="Tracing Project with Evaluation Annotations">
      <video
        controls
        className="w-full aspect-video rounded-xl"
        src="https://storage.googleapis.com/arize-phoenix-assets/assets/videos/completeness%20evals%20quickstart.mp4"
      />
    </Frame>

    **Congratulations**! You’ve run your first evaluation in Phoenix.
  </Step>
</Steps>

## **Learn More About Evals**

Now that you have evaluation results in Phoenix, you can start using them to guide iteration.

You can group traces with an incorrect label into a dataset, make changes to prompts or logic, and then run experiments on the same inputs to compare how outputs differ. The easiest and fastest way to make iterations to your application with no code is through prompt playground. The [Iterate on Your Prompts guide](/docs/phoenix/get-started/get-started-prompt-playground) walks through this workflow in more detail.

To go deeper on evaluations, the [Evaluations Tutorial](https://arize.com/docs/phoenix/evaluation/typescript-quickstart) covers writing more nuanced evaluators, using different scoring strategies, and comparing quality across runs as your application evolves.

This was a simple example, but evaluations in Phoenix can support much more advanced workflows over time.
