{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Classification Fine-tuning\n",
    "\n",
    "Let's walk through a quick example of fine-tuning the LM weights within a DSPy program. We'll apply to a simple 77-way classification task.\n",
    "\n",
    "Our finetuned program will use a tiny `Llama-3.2-1B` language model, hosted locally on your GPU. To make this more interesting, we'll assume that (i) we don't have **any training labels** but (ii) we have 500 unlabeled training examples.\n",
    "\n",
    "### Install dependencies and download data\n",
    "\n",
    "Install the latest DSPy via `pip install -U dspy` and follow along (or `uv pip`, if you prefer). This tutorial depends on DSPy >= 2.6.0. You also need to run `pip install datasets`.\n",
    "\n",
    "This tutorial requires a local GPU at the moment for inference, though we plan to support ollama serving for finetuned models as well.\n",
    "\n",
    "You will also need the following dependencies:\n",
    "1. Inference: We use SGLang to run local inference servers. You can install the latest version by following the instructions here: https://docs.sglang.ai/start/install.html\n",
    "Shared below is the most recent install command as of 04/02/2025, but we recommend that you follow the instructions in the most up to date version by navigating to the installation link.\n",
    "This ensures that the fine-tuning packages and the `sglang` package are in sync.\n",
    "    ```shell\n",
    "    > pip install --upgrade pip\n",
    "    > pip install uv\n",
    "    > uv pip install \"sglang[all]>=0.4.4.post3\" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python\n",
    "    ```\n",
    "1. Fine-tuning: We use the following packages. Note that we specify the version for the transformers package as a temporary fix to a recent issue: https://github.com/huggingface/trl/issues/2338\n",
    "    ```shell\n",
    "    > uv pip install -U torch transformers==4.48.3 accelerate trl peft\n",
    "    ```\n",
    "\n",
    "We recommend using `uv` package manager to speed up the installation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Recommended: Set up MLflow Tracing to understand what's happening under the hood.</summary>\n",
    "\n",
    "### MLflow DSPy Integration\n",
    "\n",
    "<a href=\"https://mlflow.org/\">MLflow</a> is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. In this tutorial, you can use MLflow to visualize prompts and optimization progress as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n",
    "\n",
    "![MLflow Trace](./mlflow-tracing-classification.png)\n",
    "\n",
    "1. Install MLflow\n",
    "\n",
    "```bash\n",
    "%pip install mlflow>=2.20\n",
    "```\n",
    "\n",
    "2. Start MLflow UI in a separate terminal\n",
    "```bash\n",
    "mlflow ui --port 5000\n",
    "```\n",
    "\n",
    "3. Connect the notebook to MLflow\n",
    "```python\n",
    "import mlflow\n",
    "\n",
    "mlflow.set_tracking_uri(\"http://localhost:5000\")\n",
    "mlflow.set_experiment(\"DSPy\")\n",
    "```\n",
    "\n",
    "4. Enabling tracing.\n",
    "```python\n",
    "mlflow.dspy.autolog()\n",
    "```\n",
    "\n",
    "\n",
    "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "For this tutorial, we will use the Banking77 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dspy\n",
    "import random\n",
    "from dspy.datasets import DataLoader\n",
    "from datasets import load_dataset\n",
    "\n",
    "# Load the Banking77 dataset.\n",
    "CLASSES = load_dataset(\"PolyAI/banking77\", split=\"train\", trust_remote_code=True).features['label'].names\n",
    "kwargs = dict(fields=(\"text\", \"label\"), input_keys=(\"text\",), split=\"train\", trust_remote_code=True)\n",
    "\n",
    "# Load the first 2000 examples from the dataset, and assign a hint to each *training* example.\n",
    "raw_data = [\n",
    "    dspy.Example(x, label=CLASSES[x.label]).with_inputs(\"text\")\n",
    "    for x in DataLoader().from_huggingface(dataset_name=\"PolyAI/banking77\", **kwargs)[:1000]\n",
    "]\n",
    "\n",
    "random.Random(0).shuffle(raw_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset has 77 different categories for classification. Let's review some of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(77,\n",
       " ['activate_my_card',\n",
       "  'age_limit',\n",
       "  'apple_pay_or_google_pay',\n",
       "  'atm_support',\n",
       "  'automatic_top_up',\n",
       "  'balance_not_updated_after_bank_transfer',\n",
       "  'balance_not_updated_after_cheque_or_cash_deposit',\n",
       "  'beneficiary_not_allowed',\n",
       "  'cancel_transfer',\n",
       "  'card_about_to_expire'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(CLASSES), CLASSES[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us sample 500 (unlabeled) queries from Banking77. We'll use these for our bootstrapped finetuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Example({'text': 'What if there is an error on the exchange rate?'}) (input_keys={'text'})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unlabeled_trainset = [dspy.Example(text=x.text).with_inputs(\"text\") for x in raw_data[:500]]\n",
    "\n",
    "unlabeled_trainset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DSPy program\n",
    "\n",
    "Let's say that we want a program that takes the `text` and reasons step by step and then selects one of the classes from Banking77.\n",
    "\n",
    "Note that this is meant mainly for illustration, or for cases where you want to inspect the model's reasoning, e.g. for a small degree of explainability. In other words, this type of task is not necessarily likely to benefit very much from explicit reasoning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Literal\n",
    "\n",
    "classify = dspy.ChainOfThought(f\"text -> label: Literal{CLASSES}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bootstrapped finetuning\n",
    "\n",
    "There are many ways to go about this, e.g. allowing the model to teach itself or using inference-time compute (e.g., ensembling) to identify cases of high confidence without labels.\n",
    "\n",
    "Perhaps the simplest is to use a model that we'd expect can do a reasonable job at this task as a teacher of reasoning and classification, and to distill that to our small model. All of these patterns can be expressed in a handful of lines of code.\n",
    "\n",
    "Let's set up the tiny `Llama-3.2-1B-Instruct` as a student LM. We'll use GPT-4o-mini as a teacher LM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dspy.clients.lm_local import LocalProvider\n",
    "\n",
    "student_lm_name = \"meta-llama/Llama-3.2-1B-Instruct\"\n",
    "student_lm = dspy.LM(model=f\"openai/local:{student_lm_name}\", provider=LocalProvider(), max_tokens=2000)\n",
    "teacher_lm = dspy.LM('openai/gpt-4o-mini', max_tokens=3000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's assign classifiers to our LMs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "student_classify = classify.deepcopy()\n",
    "student_classify.set_lm(student_lm)\n",
    "\n",
    "teacher_classify = classify.deepcopy()\n",
    "teacher_classify.set_lm(teacher_lm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now launch the bootstrapped finetuning. The word \"bootstrapped\" here means that the program itself will be invoked on the training inputs and the resulting traces seen over all modules will be recorded and used for finetuning. This is the weight-optimizing variant of the various BootstrapFewShot methods in DSPy.\n",
    "\n",
    "On every question in the (unlabeled) training set, this will invoke the teacher program, which will produce reasoning and select a class. This will be traced and then constitute a training set for all modules (in this case, just the one CoT module) in the student program.\n",
    "\n",
    "When the `compile` method is called, the `BootstrapFinetune` optimizer will use the passed teacher program (or programs, you can pass a list!) to create a training dataset.\n",
    "It will then use this training dataset to create a fine-tuned version of the LM set for the `student` program, replacing it with the trained LM.\n",
    "Note that the trained LM will be a new LM instance (the `student_lm` object we instantiated here will be untouched!)\n",
    "\n",
    "Note: If you have labels, you can pass `metric` to the constructor of `BootstrapFinetune`. If you want to apply this in practice, you can pass `train_kwargs` to the constructor to control local LM training settings: `device`, `use_peft`, `num_train_epochs`, `per_device_train_batch_size`, `gradient_accumulation_steps`, `learning_rate`, `max_seq_length`, `packing`, `bf16`, and `output_dir`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional:\n",
    "# [1] You can set `DSPY_FINETUNEDIR` environment variable to control where the directory that will be used to store the\n",
    "#     checkpoints and fine-tuning data. If this is not set, `DSPY_CACHEDIR` is used by default.\n",
    "# [2] You can set the `CUDA_VISIBLE_DEVICES` environment variable to control the GPU that will be used for fine-tuning\n",
    "#     and inference. If this is not set and the default GPU that's used by HuggingFace's `transformers` library is\n",
    "#     occupied, an OutOfMemoryError might be raised.\n",
    "#\n",
    "# import os\n",
    "# os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\"\n",
    "# os.environ[\"DSPY_FINETUNEDIR\"] = \"/path/to/dir\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "dspy.settings.experimental = True  # fine-tuning is an experimental feature, so we set a flag to enable it\n",
    "\n",
    "optimizer = dspy.BootstrapFinetune(num_threads=16)  # if you *do* have labels, pass metric=your_metric here!\n",
    "classify_ft = optimizer.compile(student_classify, teacher=teacher_classify, trainset=unlabeled_trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is a local model, we need to explicitly launch it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "classify_ft.get_lm().launch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validating the finetuned program\n",
    "\n",
    "Let's now figure out if this was successful. We can ask the system one question and inspect its behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(\n",
       "    reasoning='The user is inquiring about a specific issue, which they did not receive and is still showing as a pending transaction. This situation typically indicates a problem with the cash withdrawal process, as the user is not receiving the money they attempted to withdraw. The appropriate label for this scenario is \"pending_cash_withdrawal,\" as it directly relates to the status of the cash withdrawal transaction.',\n",
       "    label='pending_cash_withdrawal'\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classify_ft(text=\"I didn't receive my money earlier and it says the transaction is still in progress. Can you fix it?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also get a small set of gold labels and see if the system can generalize to unseen queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Example({'text': 'Which fiat currencies do you currently support? Will this change in this future?', 'label': 'fiat_currency_support'}) (input_keys={'text'})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "devset = raw_data[500:600]\n",
    "devset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define an evaluator on this small dev set, where the metric ignores the reasoning and checks that the label is exactly correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric = (lambda x, y, trace=None: x.label == y.label)\n",
    "evaluate = dspy.Evaluate(devset=devset, metric=metric, display_progress=True, display_table=5, num_threads=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's evaluate the finetuned 1B classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 51.00 / 99 (51.5%): 100%|██████████| 100/100 [00:35<00:00,  2.79it/s]"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>example_label</th>\n",
       "      <th>reasoning</th>\n",
       "      <th>pred_label</th>\n",
       "      <th>&lt;lambda&gt;</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Which fiat currencies do you currently support? Will this change i...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>The user is inquiring about the current support for fiat currencie...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I didn't receive my money earlier and it says the transaction is s...</td>\n",
       "      <td>pending_cash_withdrawal</td>\n",
       "      <td>The user is inquiring about a specific issue, which they did not r...</td>\n",
       "      <td>pending_cash_withdrawal</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>what currencies do you accept?</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>The user is inquiring about the currencies that are accepted, whic...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Where can I find your exchange rates?</td>\n",
       "      <td>exchange_rate</td>\n",
       "      <td>The user is inquiring about where to find exchange rates, which re...</td>\n",
       "      <td>exchange_rate</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>why hasnt my card come in yet?</td>\n",
       "      <td>card_arrival</td>\n",
       "      <td>The user is inquiring about the status of their card, which sugges...</td>\n",
       "      <td>card_arrival</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    text  \\\n",
       "0  Which fiat currencies do you currently support? Will this change i...   \n",
       "1  I didn't receive my money earlier and it says the transaction is s...   \n",
       "2                                         what currencies do you accept?   \n",
       "3                                  Where can I find your exchange rates?   \n",
       "4                                         why hasnt my card come in yet?   \n",
       "\n",
       "             example_label  \\\n",
       "0    fiat_currency_support   \n",
       "1  pending_cash_withdrawal   \n",
       "2    fiat_currency_support   \n",
       "3            exchange_rate   \n",
       "4             card_arrival   \n",
       "\n",
       "                                                               reasoning  \\\n",
       "0  The user is inquiring about the current support for fiat currencie...   \n",
       "1  The user is inquiring about a specific issue, which they did not r...   \n",
       "2  The user is inquiring about the currencies that are accepted, whic...   \n",
       "3  The user is inquiring about where to find exchange rates, which re...   \n",
       "4  The user is inquiring about the status of their card, which sugges...   \n",
       "\n",
       "                pred_label   <lambda> label  \n",
       "0    fiat_currency_support  ✔️ [True]   NaN  \n",
       "1  pending_cash_withdrawal  ✔️ [True]   NaN  \n",
       "2    fiat_currency_support  ✔️ [True]   NaN  \n",
       "3            exchange_rate  ✔️ [True]   NaN  \n",
       "4             card_arrival  ✔️ [True]   NaN  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 95 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "51.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(classify_ft)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Tracking Evaluation Results in MLflow Experiment</summary>\n",
    "\n",
    "<br/>\n",
    "\n",
    "To track and visualize the evaluation results over time, you can record the results in MLflow Experiment.\n",
    "\n",
    "\n",
    "```python\n",
    "import mlflow\n",
    "\n",
    "with mlflow.start_run(run_name=\"classifier_evaluation\"):\n",
    "    evaluate_correctness = dspy.Evaluate(\n",
    "        devset=devset,\n",
    "        metric=extraction_correctness_metric,\n",
    "        num_threads=16,\n",
    "        display_progress=True,\n",
    "    )\n",
    "\n",
    "    # Evaluate the program as usual\n",
    "    result = evaluate_correctness(people_extractor)\n",
    "\n",
    "    # Log the aggregated score\n",
    "    mlflow.log_metric(\"exact_match\", result.score)\n",
    "    # Log the detailed evaluation results as a table\n",
    "    mlflow.log_table(\n",
    "        {\n",
    "            \"Text\": [example.text for example in devset],\n",
    "            \"Expected\": [example.example_label for example in devset],\n",
    "            \"Predicted\": [output[1] for output in result.results],\n",
    "            \"Exact match\": [output[2] for output in result.results],\n",
    "        },\n",
    "        artifact_file=\"eval_results.json\",\n",
    "    )\n",
    "```\n",
    "\n",
    "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad, given that we started with no labels of the task. Even though we have no labels, you can use various strategies to boost the quality of the bootstrapped training data.\n",
    "\n",
    "To try that next, let's free our GPU memory by killing the finetuned LM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "classify_ft.get_lm().kill()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bootstrapped finetuning against a metric\n",
    "\n",
    "If you have labels, you can generally boost this by a large margin. To do so, you can pass a `metric` to BootstrapFinetune, which it will use for filtering the trajectories over your program before it builds the finetuning data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = dspy.BootstrapFinetune(num_threads=16, metric=metric)\n",
    "classify_ft = optimizer.compile(student_classify, teacher=teacher_classify, trainset=raw_data[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now launch and evaluate this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "classify_ft.get_lm().launch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 85.00 / 98 (86.7%): 100%|██████████| 100/100 [00:46<00:00,  2.14it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>example_label</th>\n",
       "      <th>reasoning</th>\n",
       "      <th>pred_label</th>\n",
       "      <th>&lt;lambda&gt;</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Which fiat currencies do you currently support? Will this change i...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>The user is inquiring about the fiat currencies currently supporte...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I didn't receive my money earlier and it says the transaction is s...</td>\n",
       "      <td>pending_cash_withdrawal</td>\n",
       "      <td>The user is inquiring about an unexpected fee on their account, wh...</td>\n",
       "      <td>extra_charge_on_statement</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>what currencies do you accept?</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>The user is inquiring about the types of currencies that are accep...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Where can I find your exchange rates?</td>\n",
       "      <td>exchange_rate</td>\n",
       "      <td>The user is inquiring about where to find exchange rates, which re...</td>\n",
       "      <td>exchange_rate</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>why hasnt my card come in yet?</td>\n",
       "      <td>card_arrival</td>\n",
       "      <td>The user is inquiring about the status of their card delivery, whi...</td>\n",
       "      <td>card_arrival</td>\n",
       "      <td>✔️ [True]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    text  \\\n",
       "0  Which fiat currencies do you currently support? Will this change i...   \n",
       "1  I didn't receive my money earlier and it says the transaction is s...   \n",
       "2                                         what currencies do you accept?   \n",
       "3                                  Where can I find your exchange rates?   \n",
       "4                                         why hasnt my card come in yet?   \n",
       "\n",
       "             example_label  \\\n",
       "0    fiat_currency_support   \n",
       "1  pending_cash_withdrawal   \n",
       "2    fiat_currency_support   \n",
       "3            exchange_rate   \n",
       "4             card_arrival   \n",
       "\n",
       "                                                               reasoning  \\\n",
       "0  The user is inquiring about the fiat currencies currently supporte...   \n",
       "1  The user is inquiring about an unexpected fee on their account, wh...   \n",
       "2  The user is inquiring about the types of currencies that are accep...   \n",
       "3  The user is inquiring about where to find exchange rates, which re...   \n",
       "4  The user is inquiring about the status of their card delivery, whi...   \n",
       "\n",
       "                  pred_label   <lambda> label  \n",
       "0      fiat_currency_support  ✔️ [True]   NaN  \n",
       "1  extra_charge_on_statement              NaN  \n",
       "2      fiat_currency_support  ✔️ [True]   NaN  \n",
       "3              exchange_rate  ✔️ [True]   NaN  \n",
       "4               card_arrival  ✔️ [True]   NaN  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 95 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "85.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(classify_ft)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's quite a bit better, given just 500 labels. In fact, it seems to be a lot stronger than the teacher LM gets out of the box!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 55.00 / 100 (55.0%): 100%|██████████| 100/100 [00:11<00:00,  8.88it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/01/08 12:38:35 INFO dspy.evaluate.evaluate: Average Metric: 55 / 100 (55.0%)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>example_label</th>\n",
       "      <th>reasoning</th>\n",
       "      <th>pred_label</th>\n",
       "      <th>&lt;lambda&gt;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Which fiat currencies do you currently support? Will this change i...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>The user is inquiring about the fiat currencies supported by the s...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I didn't receive my money earlier and it says the transaction is s...</td>\n",
       "      <td>pending_cash_withdrawal</td>\n",
       "      <td>The user is experiencing an issue with a transaction that is still...</td>\n",
       "      <td>pending_transfer</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>what currencies do you accept?</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>The question is asking about the types of currencies accepted, whi...</td>\n",
       "      <td>fiat_currency_support</td>\n",
       "      <td>✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Where can I find your exchange rates?</td>\n",
       "      <td>exchange_rate</td>\n",
       "      <td>The user is inquiring about where to find exchange rates, which re...</td>\n",
       "      <td>exchange_rate</td>\n",
       "      <td>✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>why hasnt my card come in yet?</td>\n",
       "      <td>card_arrival</td>\n",
       "      <td>The user is inquiring about the status of their card delivery, whi...</td>\n",
       "      <td>card_delivery_estimate</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    text  \\\n",
       "0  Which fiat currencies do you currently support? Will this change i...   \n",
       "1  I didn't receive my money earlier and it says the transaction is s...   \n",
       "2                                         what currencies do you accept?   \n",
       "3                                  Where can I find your exchange rates?   \n",
       "4                                         why hasnt my card come in yet?   \n",
       "\n",
       "             example_label  \\\n",
       "0    fiat_currency_support   \n",
       "1  pending_cash_withdrawal   \n",
       "2    fiat_currency_support   \n",
       "3            exchange_rate   \n",
       "4             card_arrival   \n",
       "\n",
       "                                                               reasoning  \\\n",
       "0  The user is inquiring about the fiat currencies supported by the s...   \n",
       "1  The user is experiencing an issue with a transaction that is still...   \n",
       "2  The question is asking about the types of currencies accepted, whi...   \n",
       "3  The user is inquiring about where to find exchange rates, which re...   \n",
       "4  The user is inquiring about the status of their card delivery, whi...   \n",
       "\n",
       "               pred_label   <lambda>  \n",
       "0   fiat_currency_support  ✔️ [True]  \n",
       "1        pending_transfer             \n",
       "2   fiat_currency_support  ✔️ [True]  \n",
       "3           exchange_rate  ✔️ [True]  \n",
       "4  card_delivery_estimate             "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 95 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "55.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(teacher_classify)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And thanks to bootstrapping, the model learns to apply our modules to get the right label, in this case, reasoning explicitly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\u001b[34m[2025-01-08T12:39:42.143798]\u001b[0m\n",
      "\n",
      "\u001b[31mSystem message:\u001b[0m\n",
      "\n",
      "Your input fields are:\n",
      "1. `text` (str)\n",
      "\n",
      "Your output fields are:\n",
      "1. `reasoning` (str)\n",
      "2. `label` (Literal[activate_my_card, age_limit, apple_pay_or_google_pay, atm_support, automatic_top_up, balance_not_updated_after_bank_transfer, balance_not_updated_after_cheque_or_cash_deposit, beneficiary_not_allowed, cancel_transfer, card_about_to_expire, card_acceptance, card_arrival, card_delivery_estimate, card_linking, card_not_working, card_payment_fee_charged, card_payment_not_recognised, card_payment_wrong_exchange_rate, card_swallowed, cash_withdrawal_charge, cash_withdrawal_not_recognised, change_pin, compromised_card, contactless_not_working, country_support, declined_card_payment, declined_cash_withdrawal, declined_transfer, direct_debit_payment_not_recognised, disposable_card_limits, edit_personal_details, exchange_charge, exchange_rate, exchange_via_app, extra_charge_on_statement, failed_transfer, fiat_currency_support, get_disposable_virtual_card, get_physical_card, getting_spare_card, getting_virtual_card, lost_or_stolen_card, lost_or_stolen_phone, order_physical_card, passcode_forgotten, pending_card_payment, pending_cash_withdrawal, pending_top_up, pending_transfer, pin_blocked, receiving_money, Refund_not_showing_up, request_refund, reverted_card_payment?, supported_cards_and_currencies, terminate_account, top_up_by_bank_transfer_charge, top_up_by_card_charge, top_up_by_cash_or_cheque, top_up_failed, top_up_limits, top_up_reverted, topping_up_by_card, transaction_charged_twice, transfer_fee_charged, transfer_into_account, transfer_not_received_by_recipient, transfer_timing, unable_to_verify_identity, verify_my_identity, verify_source_of_funds, verify_top_up, virtual_card_not_working, visa_or_mastercard, why_verify_identity, wrong_amount_of_cash_received, wrong_exchange_rate_for_cash_withdrawal])\n",
      "\n",
      "All interactions will be structured in the following way, with the appropriate values filled in.\n",
      "\n",
      "[[ ## text ## ]]\n",
      "{text}\n",
      "\n",
      "[[ ## reasoning ## ]]\n",
      "{reasoning}\n",
      "\n",
      "[[ ## label ## ]]\n",
      "{label}        # note: the value you produce must be one of: activate_my_card; age_limit; apple_pay_or_google_pay; atm_support; automatic_top_up; balance_not_updated_after_bank_transfer; balance_not_updated_after_cheque_or_cash_deposit; beneficiary_not_allowed; cancel_transfer; card_about_to_expire; card_acceptance; card_arrival; card_delivery_estimate; card_linking; card_not_working; card_payment_fee_charged; card_payment_not_recognised; card_payment_wrong_exchange_rate; card_swallowed; cash_withdrawal_charge; cash_withdrawal_not_recognised; change_pin; compromised_card; contactless_not_working; country_support; declined_card_payment; declined_cash_withdrawal; declined_transfer; direct_debit_payment_not_recognised; disposable_card_limits; edit_personal_details; exchange_charge; exchange_rate; exchange_via_app; extra_charge_on_statement; failed_transfer; fiat_currency_support; get_disposable_virtual_card; get_physical_card; getting_spare_card; getting_virtual_card; lost_or_stolen_card; lost_or_stolen_phone; order_physical_card; passcode_forgotten; pending_card_payment; pending_cash_withdrawal; pending_top_up; pending_transfer; pin_blocked; receiving_money; Refund_not_showing_up; request_refund; reverted_card_payment?; supported_cards_and_currencies; terminate_account; top_up_by_bank_transfer_charge; top_up_by_card_charge; top_up_by_cash_or_cheque; top_up_failed; top_up_limits; top_up_reverted; topping_up_by_card; transaction_charged_twice; transfer_fee_charged; transfer_into_account; transfer_not_received_by_recipient; transfer_timing; unable_to_verify_identity; verify_my_identity; verify_source_of_funds; verify_top_up; virtual_card_not_working; visa_or_mastercard; why_verify_identity; wrong_amount_of_cash_received; wrong_exchange_rate_for_cash_withdrawal\n",
      "\n",
      "[[ ## completed ## ]]\n",
      "\n",
      "In adhering to this structure, your objective is: \n",
      "        Given the fields `text`, produce the fields `label`.\n",
      "\n",
      "\n",
      "\u001b[31mUser message:\u001b[0m\n",
      "\n",
      "[[ ## text ## ]]\n",
      "why hasnt my card come in yet?\n",
      "\n",
      "Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## label ## ]]` (must be formatted as a valid Python Literal[activate_my_card, age_limit, apple_pay_or_google_pay, atm_support, automatic_top_up, balance_not_updated_after_bank_transfer, balance_not_updated_after_cheque_or_cash_deposit, beneficiary_not_allowed, cancel_transfer, card_about_to_expire, card_acceptance, card_arrival, card_delivery_estimate, card_linking, card_not_working, card_payment_fee_charged, card_payment_not_recognised, card_payment_wrong_exchange_rate, card_swallowed, cash_withdrawal_charge, cash_withdrawal_not_recognised, change_pin, compromised_card, contactless_not_working, country_support, declined_card_payment, declined_cash_withdrawal, declined_transfer, direct_debit_payment_not_recognised, disposable_card_limits, edit_personal_details, exchange_charge, exchange_rate, exchange_via_app, extra_charge_on_statement, failed_transfer, fiat_currency_support, get_disposable_virtual_card, get_physical_card, getting_spare_card, getting_virtual_card, lost_or_stolen_card, lost_or_stolen_phone, order_physical_card, passcode_forgotten, pending_card_payment, pending_cash_withdrawal, pending_top_up, pending_transfer, pin_blocked, receiving_money, Refund_not_showing_up, request_refund, reverted_card_payment?, supported_cards_and_currencies, terminate_account, top_up_by_bank_transfer_charge, top_up_by_card_charge, top_up_by_cash_or_cheque, top_up_failed, top_up_limits, top_up_reverted, topping_up_by_card, transaction_charged_twice, transfer_fee_charged, transfer_into_account, transfer_not_received_by_recipient, transfer_timing, unable_to_verify_identity, verify_my_identity, verify_source_of_funds, verify_top_up, virtual_card_not_working, visa_or_mastercard, why_verify_identity, wrong_amount_of_cash_received, wrong_exchange_rate_for_cash_withdrawal]), and then ending with the marker for `[[ ## completed ## ]]`.\n",
      "\n",
      "\n",
      "\u001b[31mResponse:\u001b[0m\n",
      "\n",
      "\u001b[32m[[ ## reasoning ## ]]\n",
      "The user is inquiring about the status of their card delivery, which suggests they are concerned about when they will receive their card. This aligns with the topic of card arrival and delivery estimates.\n",
      "\n",
      "[[ ## label ## ]]\n",
      "card_arrival\n",
      "\n",
      "[[ ## completed ## ]]\u001b[0m\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "classify_ft(text=\"why hasnt my card come in yet?\")\n",
    "dspy.inspect_history()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Saving fine-tuned programs in MLflow Experiment</summary>\n",
    "\n",
    "<br/>\n",
    "\n",
    "To deploy the fine-tuned program in production or share it with your team, you can save it in MLflow Experiment. Compared to simply saving it to a local file, MLflow offers the following benefits:\n",
    "\n",
    "1. **Dependency Management**: MLflow automatically save the frozen environment metadata along with the program to ensure reproducibility.\n",
    "2. **Experiment Tracking**: With MLflow, you can track the program's performance and cost along with the program itself.\n",
    "3. **Collaboration**: You can share the program and results with your team members by sharing the MLflow experiment.\n",
    "\n",
    "To save the program in MLflow, run the following code:\n",
    "\n",
    "```python\n",
    "import mlflow\n",
    "\n",
    "# Start an MLflow Run and save the program\n",
    "with mlflow.start_run(run_name=\"optimized_classifier\"):\n",
    "    model_info = mlflow.dspy.log_model(\n",
    "        classify_ft,\n",
    "        artifact_path=\"model\", # Any name to save the program in MLflow\n",
    "    )\n",
    "\n",
    "# Load the program back from MLflow\n",
    "loaded = mlflow.dspy.load_model(model_info.model_uri)\n",
    "```\n",
    "\n",
    "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n",
    "\n",
    "</details>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py311_202501",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
