{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UaDIwQOOjgAO"
   },
   "source": [
    "# Supervised Fine-Tuning (SFT) Qwen3-VL with QLoRA using TRL\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_qwen_vl.ipynb)\n",
    "\n",
    "![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4f0hzSo4kKEc"
   },
   "source": [
    "With [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl), you can fine-tune cutting edge vision language models. It comes with support for quantized parameter efficient fine-tuning technique **QLoRA**, so we can use free Colab (T4 GPU) to fine-tune models like [Qwen3-VL](https://huggingface.co/collections/Qwen/qwen3-vl-68d2a7c1b8a8afce4ebd2dbe).\n",
    "\n",
    "\n",
    "- [TRL GitHub Repository](https://github.com/huggingface/trl) — star us to support the project!  \n",
    "- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview)  \n",
    "- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\n",
    "- [More Qwen3-VL Fine-tuning Examples (including TRL scripts)](https://github.com/QwenLM/Qwen3-VL/tree/main/qwen-vl-finetune/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pGXgIbj2kXEP"
   },
   "source": [
    "## Install dependencies\n",
    "\n",
    "We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, and **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8CfZlUevmkg7"
   },
   "outputs": [],
   "source": [
    "!pip install -Uq \"trl[peft]\" bitsandbytes trackio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ou0VO1gHklS-"
   },
   "source": [
    "### Log in to Hugging Face\n",
    "\n",
    "Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4Ncx0wYtnYCW"
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vNylrNdqkoN-"
   },
   "source": [
    "## Load dataset\n",
    "\n",
    "\n",
    "We'll load the [**trl-lib/llava-instruct-mix**](https://huggingface.co/datasets/trl-lib/llava-instruct-mix) dataset from the Hugging Face Hub using the `datasets` library.\n",
    "\n",
    "This dataset is a set of GPT-generated multimodal instruction-following data. We use a processed version for conveniency here. You can check out more details about how to configure your own multimodal dataset for traininig with SFT in the [docs](https://huggingface.co/docs/trl/en/sft_trainer#training-vision-language-models). Fine-tuning Qwen3-VL on it helps refine its response style and visual understanding.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0AcyX6Jd1_hp"
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset_name = \"trl-lib/llava-instruct-mix\"\n",
    "train_dataset = load_dataset(dataset_name, split=\"train[:10%]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JFtR4Xyx4FYO"
   },
   "source": [
    "Let's review one example to understand the internal structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YLrEY_v8m0eA"
   },
   "outputs": [],
   "source": [
    "train_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qeZCtRB1m5xj"
   },
   "source": [
    "## Load model and configure LoRA/QLoRA\n",
    "\n",
    "This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gt05dgXgm9QR"
   },
   "outputs": [],
   "source": [
    "from transformers import Qwen3VLForConditionalGeneration, BitsAndBytesConfig\n",
    "import torch\n",
    "\n",
    "model_name = \"Qwen/Qwen3-VL-4B-Instruct\" # \"Qwen/Qwen3-VL-8B-Instruct\"\n",
    "\n",
    "model = Qwen3VLForConditionalGeneration.from_pretrained(\n",
    "    model_name,\n",
    "    dtype=\"float32\",\n",
    "    device_map=\"auto\",\n",
    "    quantization_config=BitsAndBytesConfig(\n",
    "        load_in_4bit=True,                        # Load the model in 4-bit precision to save memory\n",
    "        bnb_4bit_compute_dtype=torch.float16,     # Data type used for internal computations in quantization\n",
    "        bnb_4bit_use_double_quant=True,           # Use double quantization to improve accuracy\n",
    "        bnb_4bit_quant_type=\"nf4\"                 # Type of quantization. \"nf4\" is recommended for recent LLMs\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jyklRvNxnHmy"
   },
   "source": [
    "The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter** — a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ME1im5gh2LFg"
   },
   "outputs": [],
   "source": [
    "from peft import LoraConfig\n",
    "\n",
    "# You may need to update `target_modules` depending on the architecture of your chosen model.\n",
    "# For example, different VLMs might have different attention/projection layer names.\n",
    "peft_config = LoraConfig(\n",
    "    r=32,\n",
    "    lora_alpha=32,\n",
    "    target_modules=['down_proj','o_proj','k_proj','q_proj','gate_proj','up_proj','v_proj'],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mBAfaiA-nbdm"
   },
   "source": [
    "## Train model\n",
    "\n",
    "We'll configure **SFT** using `SFTConfig`, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the [TRL SFTConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.SFTConfig)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GQPxXvu-2Ngc"
   },
   "outputs": [],
   "source": [
    "from trl import SFTConfig\n",
    "\n",
    "output_dir = \"Qwen3-VL-4B-Instruct-trl-sft\"\n",
    "\n",
    "# Configure training arguments using SFTConfig\n",
    "training_args = SFTConfig(\n",
    "    # Training schedule / optimization\n",
    "    #num_train_epochs=1,\n",
    "    max_steps=10,                                         # Number of dataset passes. For full trainings, use `num_train_epochs` instead\n",
    "    per_device_train_batch_size=2,                        # Batch size per GPU/CPU\n",
    "    gradient_accumulation_steps=8,                        # Gradients are accumulated over multiple steps → effective batch size = 4 * 8 = 32\n",
    "    warmup_steps=5,                                       # Gradually increase LR during first N steps\n",
    "    learning_rate=2e-4,                                   # Learning rate for the optimizer\n",
    "    optim=\"adamw_8bit\",                                   # Optimizer\n",
    "    max_length=None,                                      # For VLMs, truncating may remove image tokens, leading to errors during training. max_length=None avoids it\n",
    "\n",
    "    # Logging / reporting\n",
    "    output_dir=output_dir,                                # Where to save model checkpoints and logs\n",
    "    logging_steps=1,                                      # Log training metrics every N steps\n",
    "    report_to=\"trackio\",                                  # Experiment tracking tool\n",
    "\n",
    "    # Hub integration\n",
    "    push_to_hub=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bF4GtNO2ne1k"
   },
   "source": [
    "Configure the SFT Trainer. We pass the previously configured `training_args`. We don't use eval dataset to maintain memory usage low but you can configure it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TwBeQKQC2RfZ"
   },
   "outputs": [],
   "source": [
    "from trl import SFTTrainer\n",
    "\n",
    "trainer = SFTTrainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=train_dataset,\n",
    "    peft_config=peft_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K9Ub3jTDnfcD"
   },
   "source": [
    "Show memory stats before training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "u6_Vsv_1KtVU"
   },
   "outputs": [],
   "source": [
    "gpu_stats = torch.cuda.get_device_properties(0)\n",
    "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
    "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
    "\n",
    "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",
    "print(f\"{start_gpu_memory} GB of memory reserved.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4NiFu9tcniBP"
   },
   "source": [
    "And train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pbJXrhA0ywra"
   },
   "outputs": [],
   "source": [
    "trainer_stats = trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "miZ2I1A9nnM4"
   },
   "source": [
    "Show memory stats after training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8jegvQGlKyEu"
   },
   "outputs": [],
   "source": [
    "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
    "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
    "used_percentage = round(used_memory / max_memory * 100, 3)\n",
    "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
    "\n",
    "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
    "print(f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\")\n",
    "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
    "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
    "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
    "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3lrrYfPunloQ"
   },
   "source": [
    "## Saving fine tuned model\n",
    "\n",
    "In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MNfRlfIGKSHI"
   },
   "outputs": [],
   "source": [
    "trainer.save_model(output_dir)\n",
    "trainer.push_to_hub(dataset_name=dataset_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pFq51FWEK1DX"
   },
   "source": [
    "## Load the fine-tuned model and run inference\n",
    "\n",
    "Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wAm1iQc8K1uY"
   },
   "outputs": [],
   "source": [
    "from transformers import Qwen3VLForConditionalGeneration, AutoProcessor\n",
    "from peft import PeftModel\n",
    "\n",
    "base_model = model_name\n",
    "adapter_model = f\"{output_dir}\" # Replace with your HF username or organization + fine-tuned model name\n",
    "\n",
    "model = Qwen3VLForConditionalGeneration.from_pretrained(base_model, dtype=\"float32\", device_map=\"auto\")\n",
    "model = PeftModel.from_pretrained(model, adapter_model)\n",
    "\n",
    "processor = AutoProcessor.from_pretrained(base_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "a5rLWJdOvwGQ"
   },
   "outputs": [],
   "source": [
    "problem = train_dataset[0]['prompt'][0]['content']\n",
    "image = train_dataset[0]['images'][0]\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"image\", \"image\": image},\n",
    "            {\"type\": \"text\", \"text\": problem},\n",
    "        ],\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qiu-ROFeBPhA"
   },
   "outputs": [],
   "source": [
    "messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 497
    },
    "id": "qGzEXSH5BQwG",
    "outputId": "611d6bfd-fb72-4737-e847-1611132d49ed"
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "[encoded data, 83440 characters]",
      "image/png": "[encoded data, 759984 characters]",
      "text/plain": [
       "<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x480>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SG64xtCtv24-",
    "outputId": "cdc347b4-cdf2-49c8-e308-71dc99f27476"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"The presentation of this meal can influence one's eating experience by making it more visually appealing and appetizing. The vibrant colors of the fruits, vegetables, and bread, as well as the careful arrangement of the food in the colorful containers, can create a sense of variety and abundance. This can make the meal more enjoyable and satisfying to eat, encouraging individuals to eat more and feel more satisfied after eating. Additionally, the presentation can make the meal more inviting and appealing to others, increasing the likelihood that they will eat it.\"]\n"
     ]
    }
   ],
   "source": [
    "inputs = processor.apply_chat_template(\n",
    "    messages,\n",
    "    add_generation_prompt=True,\n",
    "    tokenize=True,\n",
    "    return_tensors=\"pt\"\n",
    "    return_dict=True,\n",
    ").to(model.device)\n",
    "\n",
    "# Inference: Generation of the output\n",
    "generated_ids = model.generate(**inputs, max_new_tokens=128)\n",
    "generated_ids_trimmed = [\n",
    "    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n",
    "]\n",
    "output_text = processor.batch_decode(\n",
    "    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n",
    ")\n",
    "print(output_text)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
