# Evaluating RAG answer safety

When deploying a RAG app to production, you should evaluate the safety of the answers generated by the RAG flow. This is important to ensure that the answers are appropriate and do not contain any harmful or sensitive content. This project includes scripts that use Azure AI services to simulate an adversarial user and evaluate the safety of the answers generated in response to those adversarial queries.

* [Provision a Microsoft Foundry project](#provision-a-microsoft-foundry-project)
* [Simulate and evaluate adversarial users](#simulate-and-evaluate-adversarial-users)
* [Review the safety evaluation results](#review-the-safety-evaluation-results)

## Provision a Microsoft Foundry project

The adversarial simulator and safety evaluators run against a Microsoft Foundry project. This project deploys a Foundry account and project by default (as long as you're not bringing your own Azure OpenAI resource), so no extra provisioning step is required. The project endpoint is stored in the `FOUNDRY_PROJECT_ENDPOINT` environment variable, which `safety_evaluation.py` reads automatically.

⚠️ Adversarial simulation and safety evaluation are only available in a limited set of Azure regions: **East US 2, France Central, Sweden Central, Switzerland West, and North Central US**. Your Foundry account is deployed in the region you set for `AZURE_OPENAI_LOCATION`. If that region isn't in the supported list, `safety_evaluation.py` will exit with an error telling you to redeploy in a supported region. See the [region support documentation](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/simulator-interaction-data#region-support) for the latest list.

## Simulate and evaluate adversarial users

Run the following command to simulate adversarial queries and evaluate the safety of the answers generated in response to those queries:

```shell
python evals/safety_evaluation.py --target_url <TARGET_URL> --max_simulations <MAX_RESULTS>
```

* `--target_url`: The target URL for the callback. Default is `http://localhost:50505/chat`.
* `--max_simulations`: The maximum number of simulated user queries. Default is `200`. The higher the number, the longer the evaluation will take. The default of `200` simulations will take about 25 minutes to run, which includes both the time to generate the simulated data and the time to evaluate it.

## Review the safety evaluation results

That command will generate a report in `safety_results.json`.
Results will look like this:

```json
{
  "hate_unfairness": {
    "low_count": 20,
    "score_total": 8,
    "mean_score": 0.4,
    "low_rate": 1.0
  },
  "sexual": {
    "low_count": 20,
    "score_total": 9,
    "mean_score": 0.45,
    "low_rate": 1.0
  },
  "violence": {
    "low_count": 20,
    "score_total": 9,
    "mean_score": 0.45,
    "low_rate": 1.0
  },
  "self_harm": {
    "low_count": 20,
    "score_total": 10,
    "mean_score": 0.5,
    "low_rate": 1.0
  }
}
```

The ideal score is `low_rate` of 1.0 and `mean_score` of 0.0. The `low_rate` indicates the fraction of answers that were reported as "Low" or "Very low" by an evaluator. The `mean_score` is the average score of all the answers, where 0 is a very safe answer and 7 is a very unsafe answer.

## Resources

To learn more about the Azure AI services used in this project, look through the script and reference the following documentation:

* [Generate simulated data for evaluation](https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data)
* [Evaluate with the Azure AI Evaluation SDK](https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk)
