# LLM Auditor (Go)

This agent functions as an automated fact-checking layer specifically designed to evaluate and enhance the factual grounding of responses generated by Large Language Models (LLMs). Its primary role is to bolster the reliability of LLM outputs by systematically analyzing them against real-world information; it achieves this by identifying verifiable claims within the text, utilizing web search and its internal knowledge to determine their accuracy, producing a detailed report on its findings, and optionally rewriting the original response to correct any discovered inaccuracies.

## Overview
This agent evaluates and improves the factual grounding of responses generated
by LLMs. Its primary purpose is to serve as an automated fact-checking layer,
analyzing LLM answers against real-world information to enhance reliability.

*   Identifies and isolates specific, verifiable statements within an
    LLM-generated text.
*   Determines accuracy of claims using web search tools and knowledge it is
    trained on.
*   Produces a clear breakdown listing identified claims with their verification
    status.
*   Optionally rewrites original responses to correct inaccuracies based on
    verified findings.

This sample agent enables a user to query an LLM and the agent audits the
corresponding answer by extracting claims, utilizing search tools for
verification, generating an audit report, and optionally re-writing the
response.

## Agent Details

The key features of the LLM Auditor include:

| Feature | Description |
| --- | --- |
| **Interaction Type** | Workflow |
| **Complexity**  | Easy |
| **Agent Type**  | Multi Agent |
| **Components**  | Tools: built-in Google Search |
| **Vertical**  | Horizontal |

### Agent architecture:

This diagram shows the detailed architecture of the agents and tools used
to implement this workflow.
<img src="llm_auditor_architecture.webp" alt="LLM Auditor Architecture" width="800"/>

## Setup and Installation

1.  **Prerequisites**

    *   Go 1.24+
    *   A project on Google Cloud Platform
    *   Google Cloud CLI
        *   For installation, please follow the instruction on the official
            [Google Cloud website](https://cloud.google.com/sdk/docs/install).

2.  **Installation**

    ```bash
    # Clone this repository.
    git clone https://github.com/google/adk-samples.git
    cd adk-samples/go/agents/llm-auditor
    # Tidy up the dependencies
    go mod tidy
    ```

3.  **Configuration**

    *   Set up Google Cloud credentials.

        *   You may set the following environment variables in your shell.

        ```bash
        export GOOGLE_GENAI_USE_VERTEXAI=true
        export GOOGLE_CLOUD_PROJECT=<your-project-id>
        export GOOGLE_CLOUD_LOCATION=<your-project-location>
        ```

    *   Authenticate your GCloud account.

        ```bash
        gcloud auth application-default login
        gcloud auth application-default set-quota-project $GOOGLE_CLOUD_PROJECT
        ```

    *   If you'd prefer to run the agent locally without using Google Vertex AI or Cloud dependencies, you may set the following 
        environment variables in your shell.

        ```bash
        export GOOGLE_GENAI_USE_VERTEXAI=false
        export GOOGLE_API_KEY=PASTE_YOUR_ACTUAL_API_KEY_HERE
        ```

        Refer to the [Google Gemini API Quickstart](https://ai.google.dev/gemini-api/docs/quickstart?lang=go) for instructions on obtaining a Google Gemini API key.

## Running the Agent

**Using `adk`**

ADK provides convenient ways to bring up agents locally and interact with them.
You may talk to the agent using the CLI:

```bash
go run ./cmd
```

This will start a web server on your machine and print the URL.
You may open the URL, select "llm_auditor" in the top-left drop-down menu, and
a chatbot interface will appear on the right. The conversation is initially
blank. Here are some example requests you may ask the LLM Auditor to verify:

*   `Double check this: Earth is further away from the Sun than Mars.`
*   `Q: Why the blueberries are blue? A: Because blueberries have pigments on
    their skin.`

Sampled responses of these requests are shown below in the [Example
Interaction](#example-interaction) section.

### Example Interaction

Below is an example interaction with the LLM Auditor. Note that the exact output
of the agent may be different every time.

```
user: Double check this: Earth is further away from the Sun than Mars.

[critic_agent]: Okay, I will double-check the statement: "Earth is further away
from the Sun than Mars."

**Step 1: Identify the CLAIMS**

The answer contains one claim:

* CLAIM 1: Earth is further away from the Sun than Mars.

**Step 2: Verify each CLAIM**

* CLAIM 1: Earth is further away from the Sun than Mars.
    * Context: This is a direct statement of fact about the relative distances
      of Earth and Mars from the Sun.
    * External Sources: I will use Google Search to check the average distances
      of Earth and Mars from the Sun.
    * Verdict: **Inaccurate**.
    * Justification: According to the search results, the average distance of
      Earth from the Sun is approximately 150 million kilometers [2, 3, 6, 9],
      while the average distance of Mars from the Sun is approximately 228
      million kilometers [1, 4, 5, 8, 10]. Therefore, Mars is further away from
      the Sun than Earth, contradicting the claim.

**Step 3: Provide an overall assessment**

    * OVERALL VERDICT: Inaccurate
    * OVERALL JUSTIFICATION: The single claim made in the answer is demonstrably
      false based on widely available astronomical data. Therefore, the answer
      as a whole is incorrect.

Here's a summary of my verification results:

* Claim: Earth is further away from the Sun than Mars.
    * Part in answer text: "Earth is further away from the Sun than Mars."
    * Verdict: Inaccurate
    * Justification: The average distance of Earth from the Sun is about 150
      million kilometers, while the average distance of Mars from the Sun is
      about 228 million kilometers. Thus, Mars is farther from the sun than the
      Earth.

Reference:

<omitted for clarity>

[reviser_agent]: Mars is further away from the Sun than Earth.
```

## Disclaimer

This agent sample is provided for illustrative purposes only and is not intended for production use. It serves as a basic example of an agent and a foundational starting point for individuals or teams to develop their own agents.

This sample has not been rigorously tested, may contain bugs or limitations, and does not include features or optimizations typically required for a production environment (e.g., robust error handling, security measures, scalability, performance considerations, comprehensive logging, or advanced configuration options).

Users are solely responsible for any further development, testing, security hardening, and deployment of agents based on this sample. We recommend thorough review, testing, and the implementation of appropriate safeguards before using any derived agent in a live or critical system.
