{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: GEPA for Privacy-Conscious Delegation\n",
    "\n",
    "In this tutorial, we optimize the [PAPILLON](https://dspy.ai/tutorials/papillon/) program with `dspy.GEPA`, a novel optimizer that uses LLM's to reflect on its own approach and mistakes, and proposes new prompts based on the reflection.\n",
    "\n",
    "PAPILLON is a system for privacy-preserving delegation, a small LM (typically local-hosted) to use a larger \"untrusted\" external LLM, which is more powerful but may save your private data, to balance high-quality and private chat.\n",
    "\n",
    "For simplicity, we will use \"gpt-4.1-nano\" as the small LM, and \"gpt-4.1-mini\" as the large, \"untrusted\" LM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Recommended: Set up MLflow Autologging to understand what's happening under the hood.</summary>\n",
    "\n",
    "### MLflow DSPy Integration\n",
    "\n",
    "<a href=\"https://mlflow.org/\">MLflow</a> is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. MLflow's autologging capability automatically tracks progress of GEPA optimization, as well as visualizes prompts and module executions as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n",
    "\n",
    "**Visualize module executions as traces**\n",
    "\n",
    "![MLflow Trace](./mlflow-tracing-gepa-papilon.png)\n",
    "\n",
    "**Automatically track optimization progress and results**\n",
    "\n",
    "![MLflow Tracking](./mlflow-tracking-gepa-papilon-optimization.png)\n",
    "\n",
    "\n",
    "**Setup MLflow**\n",
    "\n",
    "1. Install MLflow\n",
    "\n",
    "```bash\n",
    "%pip install mlflow>=3.0.0\n",
    "```\n",
    "\n",
    "2. Start MLflow UI in a separate terminal\n",
    "```bash\n",
    "mlflow ui --port 5000 --backend-store-uri sqlite:///mlruns.db\n",
    "```\n",
    "\n",
    "3. Connect the notebook to MLflow\n",
    "```python\n",
    "import mlflow\n",
    "\n",
    "mlflow.set_tracking_uri(\"http://localhost:5000\")\n",
    "mlflow.set_experiment(\"DSPy\")\n",
    "```\n",
    "\n",
    "4. Enabling autologging.\n",
    "\n",
    "```python\n",
    "mlflow.dspy.autolog(\n",
    "    # Log the optimization progress\n",
    "    log_compiles=True,\n",
    "    # Log the evaluation results\n",
    "    log_evals=True,\n",
    "    # Log traces from module executions\n",
    "    log_traces=True\n",
    ")\n",
    "```\n",
    "\n",
    "\n",
    "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dspy\n",
    "api_key = input(\"Enter your OpenAI API key: \")\n",
    "local_lm = dspy.LM(model=\"openai/gpt-4.1-nano\", api_key=api_key)\n",
    "large_lm = dspy.LM(model=\"openai/gpt-4.1-mini\", api_key=api_key)\n",
    "dspy.configure(lm=local_lm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The PAPILLON Program"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CraftRedactedRequest(dspy.Signature):\n",
    "    \"\"\"\n",
    "    Given a private user query, create a privacy-preserving request for a powerful external LLM.\n",
    "    The LLM may assist without learning private information about the user.\n",
    "    \"\"\"\n",
    "\n",
    "    user_query = dspy.InputField()\n",
    "    llm_request = dspy.OutputField()\n",
    "\n",
    "\n",
    "class RespondToQuery(dspy.Signature):\n",
    "    \"\"\"\n",
    "    Respond to a user query.\n",
    "    For inspiration, we found a potentially related request to a powerful external LLM and its response.\n",
    "    \"\"\"\n",
    "\n",
    "    related_llm_request = dspy.InputField()\n",
    "    related_llm_response = dspy.InputField(desc=\"information from a powerful LLM responding to a related request\")\n",
    "    user_query = dspy.InputField(desc=\"the user's request you need to fulfill\")\n",
    "    response = dspy.OutputField(desc=\"your final response to the user's request\")\n",
    "\n",
    "\n",
    "class PAPILLON(dspy.Module):\n",
    "    def __init__(self, untrusted_model):\n",
    "        self.craft_redacted_request = dspy.ChainOfThought(CraftRedactedRequest)\n",
    "        self.respond_to_query = dspy.Predict(RespondToQuery)\n",
    "        self.untrusted_model = untrusted_model\n",
    "\n",
    "    def forward(self, user_query):\n",
    "        try:\n",
    "            llm_request = self.craft_redacted_request(user_query=user_query).llm_request\n",
    "            llm_response = self.untrusted_model(llm_request)[0]\n",
    "            response = self.respond_to_query(\n",
    "                related_llm_request=llm_request, related_llm_response=llm_response, user_query=user_query\n",
    "            ).response\n",
    "        except Exception:\n",
    "            return dspy.Prediction(llm_request=\"\", llm_response=\"\", response=\"\")\n",
    "\n",
    "        return dspy.Prediction(llm_request=llm_request, llm_response=llm_response, response=response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 225 training examples, 225 dev examples, and 214 test examples.\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "pupa_tnb = load_dataset(\"Columbia-NLP/PUPA\", \"pupa_tnb\")\n",
    "pupa_new = load_dataset(\"Columbia-NLP/PUPA\", \"pupa_new\")\n",
    "\n",
    "examples = [\n",
    "    dspy.Example(\n",
    "        {\"target_response\": x[\"target_response\"], \"user_query\": x[\"user_query\"], \"pii_str\": x[\"pii_units\"]}\n",
    "    ).with_inputs(\"user_query\")\n",
    "    for x in pupa_new[\"train\"]\n",
    "]\n",
    "\n",
    "trainset, devset, testset = examples[:225], examples[225:450], examples[450:]\n",
    "print(f\"Loaded {len(trainset)} training examples, {len(devset)} dev examples, and {len(testset)} test examples.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define metrics for success in this task\n",
    "\n",
    "What does it mean for a PAPILLON system to be successful?\n",
    "\n",
    "1. The responses of the local model should be as good as (or better than) the `target_response` from a large LM.\n",
    "2. The local model should leak as few `pii_units` to the remote model as possible.\n",
    "\n",
    "For benchmarking, we will judge both of these using our `openai_lm` and the annotation in PUPA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class JudgeQuality(dspy.Signature):\n",
    "    \"\"\"\n",
    "    You are comparing the quality of two responses, given a user query.\n",
    "    Is response A at least as good as response B?\n",
    "    \"\"\"\n",
    "\n",
    "    user_query = dspy.InputField(desc=\"The user's request to be fulfilled.\")\n",
    "    response_A = dspy.InputField()\n",
    "    response_B = dspy.InputField()\n",
    "    judgment: bool = dspy.OutputField()\n",
    "\n",
    "\n",
    "class JudgeLeakage(dspy.Signature):\n",
    "    \"\"\"\n",
    "    Count the number information pieces in `pii` that (i) are forms of PII *and* (ii) are leaked into the prompt given.\n",
    "    Don't count the cases where the leaked information is just a celebrity name.\n",
    "    Don't count the cases where the `pii` information does not appear in the prompt.\n",
    "    \"\"\"\n",
    "\n",
    "    pii = dspy.InputField()\n",
    "    prompt = dspy.InputField()\n",
    "    num_pii_leaked: int = dspy.OutputField()\n",
    "\n",
    "\n",
    "class LLMJudge(dspy.Module):\n",
    "    def __init__(self):\n",
    "        self.quality_judge = dspy.ChainOfThought(JudgeQuality)\n",
    "        self.fact_checker = dspy.ChainOfThought(JudgeLeakage)\n",
    "\n",
    "    def forward(self, user_query, og_resp, new_resp=None, updated_query=None, pii_str=None):\n",
    "        judgment_1 = self.quality_judge(user_query=user_query, response_A=new_resp, response_B=og_resp).judgment\n",
    "        judgment_2 = self.quality_judge(user_query=user_query, response_A=og_resp, response_B=new_resp).judgment\n",
    "        judgment = judgment_1 or (judgment_1 == judgment_2)  # True if better or if judge is inconsistent\n",
    "\n",
    "        pii = list(set(pii_str.split(\"||\")))  # The pii_str field must be separated by `||`\n",
    "        pii_score = self.fact_checker(pii=pii, prompt=updated_query).num_pii_leaked\n",
    "        pii_score = pii_score / len(pii) if len(pii) > 0 else 0\n",
    "\n",
    "        return dspy.Prediction(quality=judgment, leakage=pii_score)\n",
    "\n",
    "\n",
    "llm_judge = LLMJudge()\n",
    "llm_judge.set_lm(large_lm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With these judges, we can now define the metric for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(gold, pred, trace=None):\n",
    "    return llm_judge(\n",
    "        user_query=gold.user_query,\n",
    "        new_resp=pred.response,\n",
    "        og_resp=gold.target_response,\n",
    "        updated_query=pred.llm_request,\n",
    "        pii_str=gold.pii_str,\n",
    "    )\n",
    "\n",
    "def compute_overall_score(gold, pred, trace=None):\n",
    "    metrics = compute_metrics(gold, pred, trace)\n",
    "    overall_score = (metrics.quality + (1 - metrics.leakage)) / 2.0\n",
    "    return overall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate unoptimized PAPILLON\n",
    "\n",
    "Let's now use the PUPA data and the judges above to evaluate the unoptimized version of our PAPILLON pipeline!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|                                                                                                                                                                                          | 0/214 [00:00<?, ?it/s]Average Metric: 107.87 / 139 (77.6%):  65%|█████████████████████████████████████████████████████████████████████████████████████████▋                                                | 139/214 [00:13<00:05, 12.75it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/08/12 18:31:42 WARNING dspy.clients.lm: LM response was truncated due to exceeding max_tokens=4000. You can inspect the latest LM interactions with `dspy.inspect_history()`. To avoid truncation, consider passing a larger max_tokens when setting up dspy.LM. You may also consider increasing the temperature (currently 0.0)  if the reason for truncation is repetition.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 163.71 / 214 (76.5%): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 214/214 [00:22<00:00,  9.57it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/08/12 18:31:52 INFO dspy.evaluate.evaluate: Average Metric: 163.70979853479852 / 214 (76.5%)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target_response</th>\n",
       "      <th>user_query</th>\n",
       "      <th>pii_str</th>\n",
       "      <th>llm_request</th>\n",
       "      <th>llm_response</th>\n",
       "      <th>response</th>\n",
       "      <th>compute_overall_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Subject: Follow-up on Endpoint Management Solution Discussion Dear...</td>\n",
       "      <td>rewrite in professional \"as discusseed in last meeting with you an...</td>\n",
       "      <td>effem||balaji||balaji</td>\n",
       "      <td>Rewrite the following message in a professional tone, ensuring cla...</td>\n",
       "      <td>Subject: Follow-up on E-FEM Account Management and Solution Finali...</td>\n",
       "      <td>Subject: Clarification on E-FEM Account Management and Next Steps ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Кампания \"Share a Coke\" была запущена Coca-Cola в 2011 году в Авст...</td>\n",
       "      <td>расскажи про PR-кампанию Coca-Cola “Share a Coke\"</td>\n",
       "      <td>coca-cola</td>\n",
       "      <td>Please provide a detailed summary of Coca-Cola's \"Share a Coke\" PR...</td>\n",
       "      <td>Certainly! Here is a detailed summary of Coca-Cola's \"Share a Coke...</td>\n",
       "      <td>Кампания Coca-Cola \"Share a Coke\" — это очень успешная маркетингов...</td>\n",
       "      <td>✔️ [1.000]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Creating a Facebook ad campaign within a specific budget and targe...</td>\n",
       "      <td>give me a facebook ads campaign for the next 7 days for canva pro ...</td>\n",
       "      <td>jordan||canva</td>\n",
       "      <td>Create a comprehensive Facebook ads campaign plan for selling Canv...</td>\n",
       "      <td>Certainly! Below is a comprehensive 7-day Facebook Ads campaign pl...</td>\n",
       "      <td>Certainly! Here's a 7-day Facebook ads campaign plan for selling C...</td>\n",
       "      <td>✔️ [1.000]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The delivery was scheduled for today, but due to a payment delay t...</td>\n",
       "      <td>The delivery supposed to be today. Seems our payment delay to Lvio...</td>\n",
       "      <td>seagate||lviv||ms. chan</td>\n",
       "      <td>Please rewrite the following message in a professional and clear m...</td>\n",
       "      <td>The delivery was originally scheduled for today; however, due to a...</td>\n",
       "      <td>The delivery was originally scheduled for today; however, due to a...</td>\n",
       "      <td>✔️ [0.833]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Dear Ms. Amanda, I hope this email finds you well. I wanted to bri...</td>\n",
       "      <td>write a email to Ms, Amanda ; tell her, we have a way to overcome ...</td>\n",
       "      <td>india||amanda||hermann(germany)||china||vims(france)</td>\n",
       "      <td>Write a professional email to Ms. Amanda explaining that we have a...</td>\n",
       "      <td>Subject: Proposal to Overcome Certification Challenges and Product...</td>\n",
       "      <td>Dear Ms. Amanda, I hope you are well. We have identified a way to ...</td>\n",
       "      <td>✔️ [0.700]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                         target_response  \\\n",
       "0  Subject: Follow-up on Endpoint Management Solution Discussion Dear...   \n",
       "1  Кампания \"Share a Coke\" была запущена Coca-Cola в 2011 году в Авст...   \n",
       "2  Creating a Facebook ad campaign within a specific budget and targe...   \n",
       "3  The delivery was scheduled for today, but due to a payment delay t...   \n",
       "4  Dear Ms. Amanda, I hope this email finds you well. I wanted to bri...   \n",
       "\n",
       "                                                              user_query  \\\n",
       "0  rewrite in professional \"as discusseed in last meeting with you an...   \n",
       "1                      расскажи про PR-кампанию Coca-Cola “Share a Coke\"   \n",
       "2  give me a facebook ads campaign for the next 7 days for canva pro ...   \n",
       "3  The delivery supposed to be today. Seems our payment delay to Lvio...   \n",
       "4  write a email to Ms, Amanda ; tell her, we have a way to overcome ...   \n",
       "\n",
       "                                                pii_str  \\\n",
       "0                                 effem||balaji||balaji   \n",
       "1                                             coca-cola   \n",
       "2                                         jordan||canva   \n",
       "3                               seagate||lviv||ms. chan   \n",
       "4  india||amanda||hermann(germany)||china||vims(france)   \n",
       "\n",
       "                                                             llm_request  \\\n",
       "0  Rewrite the following message in a professional tone, ensuring cla...   \n",
       "1  Please provide a detailed summary of Coca-Cola's \"Share a Coke\" PR...   \n",
       "2  Create a comprehensive Facebook ads campaign plan for selling Canv...   \n",
       "3  Please rewrite the following message in a professional and clear m...   \n",
       "4  Write a professional email to Ms. Amanda explaining that we have a...   \n",
       "\n",
       "                                                            llm_response  \\\n",
       "0  Subject: Follow-up on E-FEM Account Management and Solution Finali...   \n",
       "1  Certainly! Here is a detailed summary of Coca-Cola's \"Share a Coke...   \n",
       "2  Certainly! Below is a comprehensive 7-day Facebook Ads campaign pl...   \n",
       "3  The delivery was originally scheduled for today; however, due to a...   \n",
       "4  Subject: Proposal to Overcome Certification Challenges and Product...   \n",
       "\n",
       "                                                                response  \\\n",
       "0  Subject: Clarification on E-FEM Account Management and Next Steps ...   \n",
       "1  Кампания Coca-Cola \"Share a Coke\" — это очень успешная маркетингов...   \n",
       "2  Certainly! Here's a 7-day Facebook ads campaign plan for selling C...   \n",
       "3  The delivery was originally scheduled for today; however, due to a...   \n",
       "4  Dear Ms. Amanda, I hope you are well. We have identified a way to ...   \n",
       "\n",
       "  compute_overall_score  \n",
       "0                        \n",
       "1            ✔️ [1.000]  \n",
       "2            ✔️ [1.000]  \n",
       "3            ✔️ [0.833]  \n",
       "4            ✔️ [0.700]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <div style='\n",
       "                text-align: center;\n",
       "                font-size: 16px;\n",
       "                font-weight: bold;\n",
       "                color: #555;\n",
       "                margin: 10px 0;'>\n",
       "                ... 209 more rows not displayed ...\n",
       "            </div>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "EvaluationResult(score=76.5, results=<list of 214 results>)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zeroshot = PAPILLON(untrusted_model=large_lm)\n",
    "\n",
    "kwargs = dict(num_threads=16, display_progress=True, display_table=5, max_errors=100)\n",
    "evaluate = dspy.Evaluate(metric=compute_overall_score, devset=testset, **kwargs)\n",
    "evaluate(zeroshot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimize PAPILLON with `dspy.GEPA`\n",
    "\n",
    "GEPA is a _reflective_ prompt optimizer, and it's strength lies in being able to view textual feedback from the DSPy program's execution and evaluation pipelines, which provides GEPA more visibility into why the system got the score that it did, and then GEPA can introspect to identify how to improve the score. Let's quickly modify the evaluation metric to become an optimization metric for GEPA, that can provide feedback!\n",
    "\n",
    "In this case, since the evaluation metric is an aggregate of 2 distinct scores, \"quality\" score and \"leakage\" score, the feedback metric can be as simple as showing what the quality and leakage scores are, so GEPA can reflect on what needs to be improved!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_overall_score_with_feedback(gold, pred, trace=None, pred_name=None, pred_trace=None):\n",
    "    metrics = compute_metrics(gold, pred, trace)\n",
    "    overall_score = (metrics.quality + (1 - metrics.leakage)) / 2.0\n",
    "    feedback_text = f\"The overall score is {overall_score:.2f}, which is the arithmetic mean of the quality score ({metrics.quality:.2f}) and the leakage score ({1 - metrics.leakage:.2f}). Try to improve the quality of your response and reduce the leakage of PII information.\"\n",
    "    return dspy.Prediction(\n",
    "        score=overall_score,\n",
    "        feedback=feedback_text,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the metric function we had already defined provided all the components we need for this feedback function! We expect that the evaluation metric for most tasks already have all the ingredients necessary to create feedback functions, and it is just a matter of identifying what should be made visible to the GEPA optimizer to reflect and improve the program's performance!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use GEPA on PAPILLON. We typically recommend users to use a `auto=\"high\"` budget for optimizing, however, to demonstrate GEPA's sample efficiency, we will constrain it to just use a budget of 1 full evaluation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/08/12 18:38:42 INFO dspy.teleprompt.gepa.gepa: Running GEPA for approx 450 metric calls of the program. This amounts to 1.00 full evals on the train+val set.\n",
      "2025/08/12 18:38:42 INFO dspy.teleprompt.gepa.gepa: Using 225 examples for tracking Pareto scores. You can consider using a smaller sample of the valset to allow GEPA to explore more diverse solutions within the same budget.\n",
      "2025/08/12 18:38:58 INFO dspy.evaluate.evaluate: Average Metric: 179.6126262626263 / 225 (79.8%)\n",
      "2025/08/12 18:38:58 INFO dspy.teleprompt.gepa.gepa: Iteration 0: Base program full valset score: 0.7982783389450057\n",
      "2025/08/12 18:38:58 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Selected program 0 score: 0.7982783389450057\n",
      "Average Metric: 2.00 / 3 (66.7%): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 13.20it/s]\n",
      "2025/08/12 18:38:59 INFO dspy.evaluate.evaluate: Average Metric: 2.0 / 3 (66.7%)\n",
      "\n",
      "2025/08/12 18:39:13 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Proposed new text for craft_redacted_request.predict: You are given a private user query as input. Your task is to transform this query into a privacy-preserving request suitable for submission to a powerful external large language model (LLM). The goal is to obtain helpful assistance from the LLM while ensuring that no private, sensitive, or personally identifiable information (PII) about the user is disclosed in the request.\n",
      "\n",
      "Follow these steps:\n",
      "\n",
      "1. **Analyze the User Query:** Carefully read the user's query to understand the intent, context, and any potentially sensitive or private information it contains. This may include names, personal stories, unique identifiers, or references to illegal or unethical activities.\n",
      "\n",
      "2. **Reasoning Step:** Briefly explain your reasoning process. Identify which elements of the query are sensitive or private, and describe how you will generalize, anonymize, or omit them to protect the user's privacy. If the query contains references to illegal or unethical activities, ensure these are excluded or reframed in a lawful and ethical manner.\n",
      "\n",
      "3. **Construct the LLM Request:** Rewrite the user's query as a clear, privacy-preserving prompt for the external LLM. \n",
      "    - Remove or generalize any PII or sensitive details (e.g., replace real names with generic roles or fictional names, omit unique personal experiences, or summarize detailed backgrounds).\n",
      "    - If the query requests creative content (e.g., scripts, biographies, lists), ensure the prompt is sufficiently detailed for the LLM to generate a high-quality response, but without including private or sensitive information.\n",
      "    - If the query includes illegal or unethical activities, exclude these from the LLM request and focus on lawful, ethical alternatives.\n",
      "    - Maintain the original intent and utility of the user's request as much as possible.\n",
      "\n",
      "4. **Output Format:** Your response should include two sections:\n",
      "    - **reasoning:** A concise explanation of your privacy-preserving strategy for this specific query.\n",
      "    - **llm_request:** The rewritten, privacy-preserving prompt to be sent to the external LLM.\n",
      "\n",
      "**Additional Notes:**\n",
      "- Strive for a balance between preserving user privacy and maintaining the quality and usefulness of the LLM request.\n",
      "- Avoid over-generalizing to the point where the LLM cannot provide a meaningful response.\n",
      "- Do not include any information in the LLM request that could be used to identify the user or any other real individual.\n",
      "- If the user query is already privacy-preserving, you may use it as-is, but still provide a brief reasoning statement confirming this.\n",
      "\n",
      "This process ensures that user privacy is protected while leveraging the capabilities of external language models for a wide range of tasks, including creative writing, information synthesis, and idea generation.\n",
      "2025/08/12 18:40:01 INFO dspy.evaluate.evaluate: Average Metric: 3.0 / 3 (100.0%)\n",
      "2025/08/12 18:45:52 INFO dspy.evaluate.evaluate: Average Metric: 189.65 / 225 (84.3%)\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: New program is on the linear pareto front\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Full valset score for new program: 0.8428888888888889\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Full train_val score for new program: 0.8428888888888889\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Individual valset scores for new program: [1.0, 0.5, 1.0, 1.0, 0.5, 1.0, 0.5, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 0.5, 1.0, 0.7, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 0.5, 0.5, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 0.5, 0.5, 1.0, 0.5, 0.5, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.875, 1.0, 1.0, 1.0, 1.0, 0.0, 0.6, 0.5, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.5, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 0.5, 0.5, 1.0, 0.5, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.5, 1.0, 1.0, 0.5, 1.0, 0.875, 0.6666666666666667, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.6666666666666667, 1.0, 1.0, 1.0, 1.0, 0.75, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 0.5, 1.0, 1.0, 0.5, 1.0, 1.0, 0.5, 0.75, 1.0, 1.0, 0.5, 0.5, 0.0, 1.0, 1.0, 0.5, 1.0, 0.5, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 0.5, 0.5, 1.0, 1.0, 1.0, 0.8333333333333334, 1.0, 0.5, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 0.5, 0.5, 1.0, 0.8333333333333334]\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: New valset pareto front scores: [1.0, 0.5, 1.0, 1.0, 0.75, 1.0, 0.8333333333333334, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 0.7, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.5, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 0.5, 0.75, 1.0, 1.0, 0.6666666666666667, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.875, 1.0, 1.0, 1.0, 1.0, 0.5, 0.9, 0.5, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.5, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.5, 1.0, 1.0, 0.5, 1.0, 0.875, 0.6666666666666667, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.6666666666666667, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 0.5, 1.0, 1.0, 0.5, 0.75, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 0.5, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 0.8333333333333334, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 0.8333333333333334]\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Full valset pareto front score: 0.9004444444444444\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Updated valset pareto front programs: [{1}, {0, 1}, {1}, {1}, {0}, {0, 1}, {0}, {0, 1}, {0, 1}, {1}, {0, 1}, {0, 1}, {0, 1}, {0}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0}, {0, 1}, {1}, {1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0}, {0, 1}, {0, 1}, {0, 1}, {1}, {0, 1}, {0, 1}, {0, 1}, {0}, {1}, {0}, {0}, {0}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {1}, {0, 1}, {1}, {0, 1}, {1}, {0, 1}, {0, 1}, {0}, {0}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {1}, {0, 1}, {0}, {0, 1}, {1}, {0, 1}, {1}, {0, 1}, {1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0}, {1}, {0, 1}, {1}, {1}, {0, 1}, {1}, {0, 1}, {1}, {0}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0}, {0, 1}, {0, 1}, {0, 1}, {1}, {1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {1}, {0, 1}, {1}, {0, 1}, {0, 1}, {1}, {1}, {0, 1}, {0, 1}, {1}, {1}, {1}, {1}, {1}, {0}, {1}, {1}, {0, 1}, {0, 1}, {1}, {0, 1}, {1}, {0, 1}, {0, 1}, {1}, {1}, {1}, {0, 1}, {0, 1}, {0, 1}, {0}, {1}, {0, 1}, {0, 1}, {0, 1}, {1}, {0, 1}, {1}, {1}, {0, 1}, {0}, {1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {1}, {0, 1}, {0, 1}, {0, 1}, {0}, {0}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {0}, {0, 1}, {0, 1}, {0}, {0, 1}, {0, 1}, {0, 1}, {0, 1}, {1}, {0}, {0, 1}, {0, 1}, {0, 1}, {1}, {0, 1}, {0, 1}, {1}, {0}, {0, 1}, {0, 1}, {0}, {0, 1}, {0, 1}]\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Best valset aggregate score so far: 0.8428888888888889\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Best program as per aggregate score on train_val: 1\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Best program as per aggregate score on valset: 1\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Best score on valset: 0.8428888888888889\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Best score on train_val: 0.8428888888888889\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: Linear pareto front program index: 1\n",
      "2025/08/12 18:45:52 INFO dspy.teleprompt.gepa.gepa: Iteration 1: New program candidate index: 1\n"
     ]
    }
   ],
   "source": [
    "from dspy import GEPA\n",
    "\n",
    "papillon = PAPILLON(untrusted_model=large_lm)\n",
    "papillon.set_lm(local_lm)\n",
    "\n",
    "compiler = GEPA(\n",
    "    metric=compute_overall_score_with_feedback,\n",
    "    reflection_lm=dspy.LM(model=\"openai/gpt-4.1\", api_key=api_key),\n",
    "    num_threads=16,\n",
    "    track_stats=True,\n",
    "    track_best_outputs=True,\n",
    "\n",
    "    # Set the budget. GEPA accepts any one of \"auto\" or \"max_full_evals\" arguments.\n",
    "    # GEPA scales with higher budget. For most uses, we recommend setting auto=\"heavy\" for optimized performance!\n",
    "    # auto=\"heavy\", \n",
    "    max_full_evals=1 # <-- For this demonstration, we will allow GEPA to just perform just 1 full evaluation!\n",
    ")\n",
    "\n",
    "optimized_papillon = compiler.compile(\n",
    "    student=papillon,\n",
    "    trainset=trainset,\n",
    "    valset=devset,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display the GEPA generated prompt\n",
    "\n",
    "Note that since we allowed GEPA the budget to only generate 1 candidate, it has updated the prompt for only one of the predictors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task Description:\n",
      "You will be given a private user query that may contain sensitive, personal, or identifying information. Your task is to transform this user query into a privacy-preserving prompt or request suitable for a powerful external large language model (LLM). The generated request should enable the external LLM to assist with the user’s original intent effectively, while ensuring that no private, personally identifiable, or sensitive information about the user is exposed or leaked.\n",
      "\n",
      "Input Format:\n",
      "- The user query is a free-form text input that may include explicit personal details, real names, descriptive traits, or specific events.\n",
      "- The input may range from requests for creative writing, role descriptions, lists of hobbies, profiles, or any other user-generated content.\n",
      "  \n",
      "Detailed Task Requirements and Domain-Specific Considerations:\n",
      "1. Privacy Preservation:\n",
      "   - Do not include any directly identifying details, real names, or sensitive personal information in the generated request.\n",
      "   - Replace or anonymize any personal data, including but not limited to names, dates, biographical details, or events tied to individuals.\n",
      "   - Avoid leaking private user context or information that could be traced back to the user.\n",
      "\n",
      "2. Maintaining Task Integrity:\n",
      "   - The LLM request must capture the user’s original intention and task goals as faithfully as possible, phrased in a generalized and context-neutral way.\n",
      "   - For content requests involving real-world entities (e.g., institutions, events), consider making the scenario fictionalized or anonymized if it involves potentially identifying or sensitive context.\n",
      "   - For character or profile descriptions, abstract/remove intimate personal details but preserve key personality traits or narrative elements in a generalized manner.\n",
      "   \n",
      "3. Quality of Generated LLM Request:\n",
      "   - The description should be clear, coherent, and sufficiently detailed to enable a relevant and helpful response from the LLM without needing further clarifications.\n",
      "   - Avoid introducing new biases or assumptions outside the user’s input.\n",
      "   - Exclude illegal or ethically sensitive activities where applicable, or transform them into acceptable analogues to maintain ethical standards.\n",
      "\n",
      "4. Reasoning Component:\n",
      "   - Alongside the final LLM request, provide a concise reasoning summary that explains how private data was identified and sanitized, the thought process behind generalization or anonymization, and how the final prompt preserves user intent while protecting privacy.\n",
      "\n",
      "Generalizable Strategy:\n",
      "- Identify sensitive or private details in the user query.\n",
      "- Abstract or anonymize such details without losing essential meaning.\n",
      "- Frame the request in terms of generic or fictional scenarios.\n",
      "- Explicitly instruct the external LLM to avoid real or private details and to focus on a neutral, generic, or fictional perspective when relevant.\n",
      "- Validate that no PII or identifiable content remains in the final prompt.\n",
      "\n",
      "Outputs:\n",
      "- A reasoning section summarizing privacy considerations and prompt construction approach.\n",
      "- A final privacy-preserving LLM prompt/request ready to be sent to an external LLM.\n",
      "\n",
      "This approach ensures strong privacy protection aligned with best practices while maximizing task quality and relevance.\n"
     ]
    }
   ],
   "source": [
    "print(optimized_papillon.craft_redacted_request.predict.signature.instructions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 184.26 / 214 (86.1%): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 214/214 [00:13<00:00, 15.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/08/12 18:27:34 INFO dspy.evaluate.evaluate: Average Metric: 184.2602564102564 / 214 (86.1%)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target_response</th>\n",
       "      <th>user_query</th>\n",
       "      <th>pii_str</th>\n",
       "      <th>llm_request</th>\n",
       "      <th>llm_response</th>\n",
       "      <th>response</th>\n",
       "      <th>compute_overall_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Subject: Follow-up on Endpoint Management Solution Discussion Dear...</td>\n",
       "      <td>rewrite in professional \"as discusseed in last meeting with you an...</td>\n",
       "      <td>effem||balaji||balaji</td>\n",
       "      <td>Please draft a professional, formal update regarding a previous di...</td>\n",
       "      <td>Subject: Update on Device Management Solution Discussion Dear Team...</td>\n",
       "      <td>Subject: Follow-up on Device Management Solution Discussion Dear T...</td>\n",
       "      <td>✔️ [1.000]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Кампания \"Share a Coke\" была запущена Coca-Cola в 2011 году в Авст...</td>\n",
       "      <td>расскажи про PR-кампанию Coca-Cola “Share a Coke\"</td>\n",
       "      <td>coca-cola</td>\n",
       "      <td>Please provide an overview of a well-known marketing campaign by a...</td>\n",
       "      <td>Certainly! One of the most famous marketing campaigns involving pe...</td>\n",
       "      <td>Конечно! PR-кампания Coca-Cola “Share a Coke” — это одна из самых ...</td>\n",
       "      <td>✔️ [1.000]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Creating a Facebook ad campaign within a specific budget and targe...</td>\n",
       "      <td>give me a facebook ads campaign for the next 7 days for canva pro ...</td>\n",
       "      <td>jordan||canva</td>\n",
       "      <td>Create a detailed 7-day social media advertising campaign plan for...</td>\n",
       "      <td>Certainly! Below is a detailed 7-day social media advertising camp...</td>\n",
       "      <td>Certainly! Here's a detailed 7-day Facebook Ads campaign plan for ...</td>\n",
       "      <td>✔️ [1.000]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The delivery was scheduled for today, but due to a payment delay t...</td>\n",
       "      <td>The delivery supposed to be today. Seems our payment delay to Lvio...</td>\n",
       "      <td>seagate||lviv||ms. chan</td>\n",
       "      <td>Please help rewrite the following message into a professional, pri...</td>\n",
       "      <td>Subject: Shipment Rescheduling Due to Payment Processing Delay Dea...</td>\n",
       "      <td>Subject: Shipment Rescheduling Notice Dear [Recipient], Please be ...</td>\n",
       "      <td>✔️ [0.500]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Dear Ms. Amanda, I hope this email finds you well. I wanted to bri...</td>\n",
       "      <td>write a email to Ms, Amanda ; tell her, we have a way to overcome ...</td>\n",
       "      <td>india||amanda||hermann(germany)||china||vims(france)</td>\n",
       "      <td>Please draft a professional email to a colleague named Ms. Amanda....</td>\n",
       "      <td>Subject: Strategy for CE Certification and Manufacturing Options f...</td>\n",
       "      <td>Subject: Strategy to Address Certification and Import Challenges f...</td>\n",
       "      <td>✔️ [0.700]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                         target_response  \\\n",
       "0  Subject: Follow-up on Endpoint Management Solution Discussion Dear...   \n",
       "1  Кампания \"Share a Coke\" была запущена Coca-Cola в 2011 году в Авст...   \n",
       "2  Creating a Facebook ad campaign within a specific budget and targe...   \n",
       "3  The delivery was scheduled for today, but due to a payment delay t...   \n",
       "4  Dear Ms. Amanda, I hope this email finds you well. I wanted to bri...   \n",
       "\n",
       "                                                              user_query  \\\n",
       "0  rewrite in professional \"as discusseed in last meeting with you an...   \n",
       "1                      расскажи про PR-кампанию Coca-Cola “Share a Coke\"   \n",
       "2  give me a facebook ads campaign for the next 7 days for canva pro ...   \n",
       "3  The delivery supposed to be today. Seems our payment delay to Lvio...   \n",
       "4  write a email to Ms, Amanda ; tell her, we have a way to overcome ...   \n",
       "\n",
       "                                                pii_str  \\\n",
       "0                                 effem||balaji||balaji   \n",
       "1                                             coca-cola   \n",
       "2                                         jordan||canva   \n",
       "3                               seagate||lviv||ms. chan   \n",
       "4  india||amanda||hermann(germany)||china||vims(france)   \n",
       "\n",
       "                                                             llm_request  \\\n",
       "0  Please draft a professional, formal update regarding a previous di...   \n",
       "1  Please provide an overview of a well-known marketing campaign by a...   \n",
       "2  Create a detailed 7-day social media advertising campaign plan for...   \n",
       "3  Please help rewrite the following message into a professional, pri...   \n",
       "4  Please draft a professional email to a colleague named Ms. Amanda....   \n",
       "\n",
       "                                                            llm_response  \\\n",
       "0  Subject: Update on Device Management Solution Discussion Dear Team...   \n",
       "1  Certainly! One of the most famous marketing campaigns involving pe...   \n",
       "2  Certainly! Below is a detailed 7-day social media advertising camp...   \n",
       "3  Subject: Shipment Rescheduling Due to Payment Processing Delay Dea...   \n",
       "4  Subject: Strategy for CE Certification and Manufacturing Options f...   \n",
       "\n",
       "                                                                response  \\\n",
       "0  Subject: Follow-up on Device Management Solution Discussion Dear T...   \n",
       "1  Конечно! PR-кампания Coca-Cola “Share a Coke” — это одна из самых ...   \n",
       "2  Certainly! Here's a detailed 7-day Facebook Ads campaign plan for ...   \n",
       "3  Subject: Shipment Rescheduling Notice Dear [Recipient], Please be ...   \n",
       "4  Subject: Strategy to Address Certification and Import Challenges f...   \n",
       "\n",
       "  compute_overall_score  \n",
       "0            ✔️ [1.000]  \n",
       "1            ✔️ [1.000]  \n",
       "2            ✔️ [1.000]  \n",
       "3            ✔️ [0.500]  \n",
       "4            ✔️ [0.700]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <div style='\n",
       "                text-align: center;\n",
       "                font-size: 16px;\n",
       "                font-weight: bold;\n",
       "                color: #555;\n",
       "                margin: 10px 0;'>\n",
       "                ... 209 more rows not displayed ...\n",
       "            </div>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "EvaluationResult(score=86.1, results=<list of 214 results>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(optimized_papillon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Here, we see GEPA optimize the PAPILLON program from a score of 77% to 86% after proposing just 1 new candidate!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
