{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UaDIwQOOjgAO"
   },
   "source": [
    "# Supervised Fine-Tuning (SFT) Ministral-3 with QLoRA using TRL\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_ministral3_vl.ipynb)\n",
    "\n",
    "![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4f0hzSo4kKEc"
   },
   "source": [
    "With [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl), you can fine-tune cutting edge vision language models. It comes with support for quantized parameter efficient fine-tuning technique **QLoRA**, so we can use free Colab (T4 GPU) to fine-tune models like [Ministral-3](https://huggingface.co/collections/mistralai/ministral-3).\n",
    "\n",
    "\n",
    "- [TRL GitHub Repository](https://github.com/huggingface/trl) — star us to support the project!  \n",
    "- [Official TRL Examples (notebooks and scripts)](https://huggingface.co/docs/trl/example_overview)  \n",
    "- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pGXgIbj2kXEP"
   },
   "source": [
    "## Install dependencies\n",
    "\n",
    "We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, and **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "V8rqrGq3hmeU",
    "outputId": "4a15adc2-e895-4c40-d174-c52e0b208dd5"
   },
   "outputs": [],
   "source": [
    "!pip install -Uq \"trl[peft]\" bitsandbytes trackio git+https://github.com/huggingface/transformers mistral-common"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ou0VO1gHklS-"
   },
   "source": [
    "### Log in to Hugging Face\n",
    "\n",
    "Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "C5eHAVFthmeU"
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vNylrNdqkoN-"
   },
   "source": [
    "## Load dataset\n",
    "\n",
    "\n",
    "We'll load the [**trl-lib/llava-instruct-mix**](https://huggingface.co/datasets/trl-lib/llava-instruct-mix) dataset from the Hugging Face Hub using the `datasets` library.\n",
    "\n",
    "This dataset is a set of GPT-generated multimodal instruction-following data. We use a processed version for conveniency here. You can check out more details about how to configure your own multimodal dataset for traininig with SFT in the [docs](https://huggingface.co/docs/trl/en/sft_trainer#training-vision-language-models). Fine-tuning Ministral-3 on it helps refine its response style and visual understanding.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "referenced_widgets": [
      "e0bb4423267a4572b1b9cc894edd25c5"
     ]
    },
    "id": "hOPra_x5hmeU",
    "outputId": "112a213e-0036-452f-e0a8-9295da13c3d1"
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset_name = \"trl-lib/llava-instruct-mix\"\n",
    "train_dataset = load_dataset(dataset_name, split=\"train[:10%]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JFtR4Xyx4FYO"
   },
   "source": [
    "Let's review one example to understand the internal structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vYJGczm6hmeV",
    "outputId": "0a9d8771-51bd-4b68-c1b0-d8b96da9b56d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'images': [<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x480>],\n",
       " 'prompt': [{'content': \"How can the presentation of this meal influence one's eating experience?\",\n",
       "   'role': 'user'}],\n",
       " 'completion': [{'content': \"The presentation of this meal can positively influence one's eating experience. In the image, colorful plastic trays and bowls are used to hold a variety of foods, including meat, vegetables, fruit, and bread. The vibrant presentation can make the meal more visually appealing and enticing, which may encourage healthier eating habits as the dishes include nutritious options like broccoli and oranges. Diverse food options and attractive meal presentation can also make the dining experience more enjoyable and satisfying. Moreover, the bright colors and well-organized food placement can create a positive atmosphere, enhancing one's overall dining experience.\",\n",
       "   'role': 'assistant'}]}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qeZCtRB1m5xj"
   },
   "source": [
    "## Load model and configure LoRA/QLoRA\n",
    "\n",
    "This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration.\n",
    "\n",
    "> **Note:**\n",
    "> In older GPUs (including those available on Colab), **FP8 support** is limited, so we use the BF16 version of the model.\n",
    "> In that case, you can select the official checkpoint or the one from Unsloth.\n",
    "> If you have access to GPUs with **FP8 support**, you can switch to that version instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "referenced_widgets": [
      "5fa6df349d314dd8b1baf79c1ed4eb0b",
      "f561335214d84bf1b8985dd4573e9ae1",
      "0c557dfead7d46e99af2c6e77b050930"
     ]
    },
    "id": "8dggHeG2hmeV",
    "outputId": "58ceedc1-26b3-467d-f466-e76d850aca5f"
   },
   "outputs": [],
   "source": [
    "from transformers import Mistral3ForConditionalGeneration, FineGrainedFP8Config, BitsAndBytesConfig\n",
    "import torch\n",
    "\n",
    "FP8 = False\n",
    "\n",
    "if FP8:\n",
    "    model_name = \"mistralai/Ministral-3-3B-Instruct-2512\"\n",
    "    quantization_config = FineGrainedFP8Config(dequantize=False)\n",
    "else:\n",
    "    model_name = \"mistralai/Ministral-3-3B-Instruct-2512-BF16\" # \"unsloth/Ministral-3-3B-Instruct-2512\"\n",
    "    quantization_config = BitsAndBytesConfig(\n",
    "        load_in_4bit=True,  # Load the model in 4-bit precision to save memory\n",
    "        bnb_4bit_compute_dtype=torch.float16,  # Data type used for internal computations in quantization\n",
    "        bnb_4bit_use_double_quant=True,  # Use double quantization to improve accuracy\n",
    "        bnb_4bit_quant_type=\"nf4\",  # Type of quantization. \"nf4\" is recommended for recent LLMs\n",
    "    )\n",
    "\n",
    "model = Mistral3ForConditionalGeneration.from_pretrained(\n",
    "    model_name,\n",
    "    dtype=\"float32\",\n",
    "    device_map=\"auto\",\n",
    "    quantization_config=quantization_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jyklRvNxnHmy"
   },
   "source": [
    "The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter** — a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8wI1Cqk4hmeV"
   },
   "outputs": [],
   "source": [
    "from peft import LoraConfig\n",
    "\n",
    "# You may need to update `target_modules` depending on the architecture of your chosen model.\n",
    "# For example, different VLMs might have different attention/projection layer names.\n",
    "peft_config = LoraConfig(\n",
    "    r=32,\n",
    "    lora_alpha=32,\n",
    "    target_modules=['down_proj','o_proj','k_proj','q_proj','gate_proj','up_proj','v_proj'],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mBAfaiA-nbdm"
   },
   "source": [
    "## Train model\n",
    "\n",
    "We'll configure **SFT** using `SFTConfig`, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the [TRL SFTConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.SFTConfig)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FrbfENGThmeV"
   },
   "outputs": [],
   "source": [
    "from trl import SFTConfig\n",
    "\n",
    "output_dir = \"Ministral-3-3B-Instruct-trl-sft\"\n",
    "\n",
    "training_args = SFTConfig(\n",
    "    # Training schedule / optimization\n",
    "    #num_train_epochs=1,\n",
    "    max_steps=10,                                         # Number of dataset passes. For full trainings, use `num_train_epochs` instead\n",
    "    per_device_train_batch_size=2,                        # Batch size per GPU/CPU\n",
    "    gradient_accumulation_steps=8,                        # Gradients are accumulated over multiple steps → effective batch size = 4 * 8 = 32\n",
    "    warmup_steps=5,                                       # Gradually increase LR during first N steps\n",
    "    learning_rate=2e-4,                                   # Learning rate for the optimizer\n",
    "    optim=\"adamw_8bit\",                                   # Optimizer\n",
    "    max_length=None,                                      # For VLMs, truncating may remove image tokens, leading to errors during training. max_length=None avoids it\n",
    "\n",
    "    # Logging / reporting\n",
    "    output_dir=output_dir,                                # Where to save model checkpoints and logs\n",
    "    logging_steps=1,                                      # Log training metrics every N steps\n",
    "    report_to=\"trackio\",                                  # Experiment tracking tool\n",
    "    trackio_space_id = output_dir,\n",
    "\n",
    "    # Hub integration\n",
    "    push_to_hub=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bF4GtNO2ne1k"
   },
   "source": [
    "Configure the SFT Trainer. We pass the previously configured `training_args`. We don't use eval dataset to maintain memory usage low but you can configure it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JjLhVbO_hmeV"
   },
   "outputs": [],
   "source": [
    "from trl import SFTTrainer\n",
    "\n",
    "trainer = SFTTrainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=train_dataset,\n",
    "    peft_config=peft_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K9Ub3jTDnfcD"
   },
   "source": [
    "Show memory stats before training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "p3UUrqCWhmeV",
    "outputId": "992da74b-7e6b-41f0-cb71-320203c1d6d9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU = Tesla T4. Max memory = 14.741 GB.\n",
      "6.346 GB of memory reserved.\n"
     ]
    }
   ],
   "source": [
    "gpu_stats = torch.cuda.get_device_properties(0)\n",
    "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
    "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
    "\n",
    "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",
    "print(f\"{start_gpu_memory} GB of memory reserved.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4NiFu9tcniBP"
   },
   "source": [
    "And train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MA8py8DghmeV",
    "outputId": "b68f35e1-cfdd-413f-b3d7-5afa4d30aac5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Trackio project initialized: huggingface\n",
      "* Trackio metrics will be synced to Hugging Face Dataset: sergiopaniego/Ministral-3-3B-Instruct-trl-sft-dataset\n",
      "* Creating new space: https://huggingface.co/spaces/sergiopaniego/Ministral-3-3B-Instruct-trl-sft\n",
      "* View dashboard by going to: https://sergiopaniego-Ministral-3-3B-Instruct-trl-sft.hf.space/\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"https://sergiopaniego-Ministral-3-3B-Instruct-trl-sft.hf.space/\" width=\"100%\" height=\"1000px\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Created new run: sergiopaniego-1764766746\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py:1044: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. Starting in PyTorch 2.9, calling checkpoint without use_reentrant will raise an exception. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  return fn(*args, **kwargs)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='10' max='10' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [10/10 44:39, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.979992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.894323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.924157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.396819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.357613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.345677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.356363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.399492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.356316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.307108</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Run finished. Uploading logs to Trackio (please wait...)\n"
     ]
    }
   ],
   "source": [
    "trainer_stats = trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "miZ2I1A9nnM4"
   },
   "source": [
    "Show memory stats after training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eUi4ww17hmeV",
    "outputId": "24b18fc3-cb0f-40a1-954d-c3b4d799dfbb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2492.1064 seconds used for training.\n",
      "41.54 minutes used for training.\n",
      "Peak reserved memory = 13.881 GB.\n",
      "Peak reserved memory for training = 7.535 GB.\n",
      "Peak reserved memory % of max memory = 94.166 %.\n",
      "Peak reserved memory for training % of max memory = 51.116 %.\n"
     ]
    }
   ],
   "source": [
    "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
    "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
    "used_percentage = round(used_memory / max_memory * 100, 3)\n",
    "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
    "\n",
    "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
    "print(f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\")\n",
    "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
    "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
    "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
    "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3lrrYfPunloQ"
   },
   "source": [
    "## Saving fine tuned model\n",
    "\n",
    "In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "referenced_widgets": [
      "ee7d9fe2f71343a4a4832fd2b0fef66f",
      "7afb48c0ac514171aa052210e9063fe7",
      "7047f7b527284d958df2ec5e800cce56",
      "8a106d43be4948ffbb2a2a5281882fe0",
      "1f3deb9e16554fd0a4666d52fa9992b2",
      "4cb20a75e85a495a8a3b36eb513db36b",
      "7ded518429454b5aa2230f98ba0d014c",
      "7112497495a742a887c4c956cdf46777",
      "cae8cfc256f649d5aa367f923b4aeb5f",
      "1a2cfcd18f5f496fa4f6bcf5c0a33966"
     ]
    },
    "id": "S7TzHDwXhmeV",
    "outputId": "0d465e97-5459-4f59-84cf-d5c374503a3b"
   },
   "outputs": [],
   "source": [
    "trainer.save_model(output_dir)\n",
    "trainer.push_to_hub(dataset_name=dataset_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pFq51FWEK1DX"
   },
   "source": [
    "## Load the fine-tuned model and run inference\n",
    "\n",
    "Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Xh4fo-WzhmeV"
   },
   "outputs": [],
   "source": [
    "output_dir = \"Ministral-3-3B-Instruct-trl-sft\"\n",
    "# model_name = \"mistralai/Ministral-3-3B-Instruct-2512\"\n",
    "model_name = \"mistralai/Ministral-3-3B-Instruct-2512-BF16\" # \"unsloth/Ministral-3-3B-Instruct-2512\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "referenced_widgets": [
      "5b4514b95d7742a5a2cc777478a152b7",
      "fdfc2ee13efa40218ab315641eb62fb7",
      "9aad01ac11794c9ca74d1a442e715da5",
      "384a08933c11424cbc48dc28f33ec90e"
     ]
    },
    "id": "z9S319H-hmeV",
    "outputId": "6b5ef891-cc0d-44ba-b112-3c577247a295"
   },
   "outputs": [],
   "source": [
    "from transformers import Mistral3ForConditionalGeneration, MistralCommonBackend\n",
    "from peft import PeftModel\n",
    "\n",
    "base_model = model_name\n",
    "adapter_model = f\"{output_dir}\" # Replace with your HF username or organization + fine-tuned model name\n",
    "\n",
    "model = Mistral3ForConditionalGeneration.from_pretrained(base_model, dtype=\"float32\", device_map=\"auto\")\n",
    "model = PeftModel.from_pretrained(model, adapter_model)\n",
    "\n",
    "tokenizer = MistralCommonBackend.from_pretrained(base_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EvObNndEhmeW"
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "from io import BytesIO\n",
    "\n",
    "problem = train_dataset[0]['prompt'][0]['content']\n",
    "image = train_dataset[0]['images'][0]\n",
    "\n",
    "buffer = BytesIO()\n",
    "image.save(buffer, format=\"JPEG\")\n",
    "image_bytes = buffer.getvalue()\n",
    "image_b64 = base64.b64encode(image_bytes).decode(\"utf-8\")\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"image_url\",\n",
    "                \"image_url\": {\n",
    "                    \"url\": f\"data:image/jpeg;base64,{image_b64}\"\n",
    "                },\n",
    "            },\n",
    "            {\"type\": \"text\", \"text\": problem},\n",
    "        ],\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yjBaVAevhmeW",
    "outputId": "47695454-856a-40b9-8ab2-d621c6e6b3da"
   },
   "outputs": [],
   "source": [
    "messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CD2BVqCBhmeW",
    "outputId": "fe1e612b-209f-4223-eaf3-9432deb0d467"
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "[encoded data, 83440 characters]",
      "image/png": "[encoded data, 759984 characters]",
      "text/plain": [
       "<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x480>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YU3hwodphmeW",
    "outputId": "7e1757f6-0c77-477a-c35a-2429d8020374"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The presentation of the meal, which is divided into three compartments in a lunch box, can significantly influence one's eating experience. The compartmentalized design allows for a visually appealing and organized presentation of the food, making it more enticing and appetizing. The variety of food items, such as the sandwich, chips, and fruit, also adds to the visual appeal and encourages a balanced and nutritious meal. The compartmentalized layout can also make it easier for the person eating the meal to access and enjoy each component of the meal, as they can easily reach for the food they want without having to rearrange the entire meal. Overall, the presentation of the meal can enhance the eating experience by making it more enjoyable, visually appealing, and convenient.</s>\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "tokenized = tokenizer.apply_chat_template(messages, return_tensors=\"pt\", return_dict=True)\n",
    "tokenized[\"input_ids\"] = tokenized[\"input_ids\"].to(device=\"cuda\")\n",
    "tokenized[\"pixel_values\"] = tokenized[\"pixel_values\"].to(dtype=torch.bfloat16, device=\"cuda\")\n",
    "image_sizes = [tokenized[\"pixel_values\"].shape[-2:]]\n",
    "\n",
    "output = model.generate(\n",
    "    **tokenized,\n",
    "    image_sizes=image_sizes,\n",
    "    max_new_tokens=512,\n",
    ")[0]\n",
    "\n",
    "decoded_output = tokenizer.decode(output[len(tokenized[\"input_ids\"][0]):])\n",
    "print(decoded_output)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
