# Evaluating the RAG answer quality

[📺 Watch: (RAG Deep Dive series) Evaluating RAG answer quality](https://www.youtube.com/watch?v=lyCLu53fb3g)

Follow these steps to evaluate the quality of the answers generated by the RAG flow.

* [Deploy an evaluation model](#deploy-an-evaluation-model)
* [Setup the evaluation environment](#setup-the-evaluation-environment)
* [Generate ground truth data](#generate-ground-truth-data)
* [Run bulk evaluation](#run-bulk-evaluation)
* [Review the evaluation results](#review-the-evaluation-results)

## Deploy an evaluation model

1. Run this command to tell `azd` to deploy a reasoning model for evaluation:

    ```shell
    azd env set USE_EVAL true
    ```

2. Set both the evaluation model and chat model capacities to the highest possible values to ensure that the evaluation runs relatively quickly. The evaluation model grades responses, while the chat model generates target answers and query rewrites. Either deployment can be rate limited during a bulk run.

    ```shell
    azd env set AZURE_OPENAI_EVAL_DEPLOYMENT_CAPACITY 100
    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_CAPACITY 100
    ```

    By default, the evaluation deployment uses `gpt-5.4` version `2026-03-05`. To change those settings, set the azd environment variables `AZURE_OPENAI_EVAL_MODEL` and `AZURE_OPENAI_EVAL_MODEL_VERSION` to the desired values.

3. Then, run the following command to provision the model:

    ```shell
    azd provision
    ```

## Setup the evaluation environment

Make a new Python virtual environment and activate it. This is currently required due to incompatibilities between the dependencies of the evaluation script and the main project.

```bash
python -m venv .evalenv
```

```bash
source .evalenv/bin/activate
```

Install all the dependencies for the evaluation script by running the following command:

```bash
pip install -r evals/requirements.txt
```

## Generate ground truth data

Generate ground truth data by running the following command:

```bash
python evals/generate_ground_truth.py --numquestions=200 --numsearchdocs=1000
```

The options are:

* `numquestions`: The number of questions to generate. We suggest at least 200.
* `numsearchdocs`: The number of documents (chunks) to retrieve from your search index. You can leave off the option to fetch all documents, but that will significantly increase time it takes to generate ground truth data. You may want to at least start with a subset.
* `kgfile`: An existing RAGAS knowledge base JSON file, which is usually `ground_truth_kg.json`. You may want to specify this if you already created a knowledge base and just want to tweak the question generation steps.
* `groundtruthfile`: The file to write the generated ground truth answwers. By default, this is `evals/ground_truth.jsonl`.

🕰️ This may take a long time, possibly several hours, depending on the size of the search index.

Review the generated data in `evals/ground_truth.jsonl` after running that script, removing any question/answer pairs that don't seem like realistic user input.

## Run bulk evaluation

Review the configuration in `evals/evaluate_config.json` to ensure that everything is correctly setup. You may want to adjust the metrics used. See [the ai-rag-chat-evaluator README](https://github.com/Azure-Samples/ai-rag-chat-evaluator) for more information on the available metrics.

By default, the evaluation script will evaluate every question in the ground truth data.
Run the evaluation script by running the following command:

```bash
python evals/run_evaluate.py
```

The options are:

* `numquestions`: The number of questions to evaluate. By default, this is all questions in the ground truth data.
* `resultsdir`: The directory to write the evaluation results. By default, this is a timestamped folder in `evals/results`. This option can also be specified in `evaluate_config.json`.
* `targeturl`: The URL of the running application to evaluate. By default, this is `http://localhost:50505`. This option can also be specified in `evaluate_config.json`.

🕰️ This may take a long time, possibly several hours, depending on the number of ground truth questions, the TPM capacity of the evaluation model, and the number of LLM-based metrics requested.

### Check for hidden rate limiting

The application SDKs automatically retry HTTP 429 responses, so an evaluation request can eventually succeed while spending significant time waiting for capacity. The evaluation client only sees the final successful response.

When Application Insights is enabled for the target deployment, query its Logs after each run:

```kusto
dependencies
| where timestamp between (datetime(<RUN_START_UTC>) .. datetime(<RUN_END_UTC>))
| where resultCode == "429"
| summarize attempts=count(), affectedRequests=dcount(operation_Id) by target, name
```

Any returned rows mean the latency results were affected by throttling. Increase the relevant deployment capacity, run `azd provision`, and repeat the evaluation before comparing latency.

## Review the evaluation results

The evaluation script will output a summary of the evaluation results, inside the `evals/results` directory.

You can see a summary of results across all evaluation runs by running the following command:

```bash
cd evals
python -m evaltools summary results
```

Compare answers to the ground truth by running the following command,
replacing `RUNHERE` with the name of a run directory inside `evals/results`:

```bash
cd evals
python -m evaltools diff results/RUNHERE
```

Compare answers across two runs by running the following command,
replacing `FIRSTRUNHERE` and `SECONDRUNHERE` with run directory names inside `evals/results`:

```bash
cd evals
python -m evaltools diff results/FIRSTRUNHERE results/SECONDRUNHERE
```

## Evaluate multimodal RAG answers

The repository also includes an `evaluate_config_multimodal.json` file specifically for evaluating multimodal RAG answers. This configuration uses a different ground truth file, `ground_truth_multimodal.jsonl`, which includes questions based off the sample data that require both text and image sources to answer.

Note that the "groundedness" evaluator is not reliable for multimodal RAG, since it does not currently incorporate the image sources. We still include it in the metrics, but the more reliable metrics are "relevance" and "citations matched".
