Install-ready Unsloth Studio llama.cpp bundles for b8475.

Install-ready Unsloth Studio llama.cpp bundles for b8457.

Hey everyone, please update Unsloth to use the latest updates! 🦥 - Unsloth now has its own **🐋 Docker image**! Start training with no setup: [Read our Guide](https://docs.unsloth.ai/new/how-to-train-llms-with-unsloth-and-docker) • [Docker image](https://hub.docker.com/r/unsloth/unsloth) - We collabed with NVIDIA for **Blackwell** and **DGX Spark** support. Read our [Blackwell guide](https://docs.unsloth.ai/basics/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) and [DGX guide](https://docs.unsloth.ai/new/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth). <a href="https://docs.unsloth.ai/new/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth" target="_blank" rel="noopener noreferrer"><img width="200" height="2076" alt="unsloth nvidia dgx spark" src="https://github.com/user-attachments/assets/b1c6e807-59ff-4283-8614-794a06738454" /></a> ### New model updates - **Qwen3-VL** models are all now supported: [Blogpost](https://docs.unsloth.ai/models/qwen3-vl-run-and-fine-tune) • [SFT 8B notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision.ipynb) • [GRPO 8B notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) - **IBM Granite-4.0** models are now supported. [Granite-4.0 guide](https://docs.unsloth.ai/models/ibm-granite-4.0) • [Notebook](https://docs.unsloth.ai/models/ibm-granite-4.0#fine-tuning-granite-4.0-in-unsloth) - OpenAI showcased our new **gpt-oss RL** notebook for autonomously solving the 2048 game. [Blogpost](https://docs.unsloth.ai/new/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) • [Notebook](https://colab.research.google.com/github/openai/gpt-oss/blob/main/examples/reinforcement-fine-tuning.ipynb) - Read about our **GLM-4.6** chat template fixes and how to run the model [here](https://docs.unsloth.ai/models/glm-4.6-how-to-run-locally#unsloth-chat-template-fixes) ### New features - Introducing **Quantization-Aware Training**: We collabed with Pytorch for QAT, recovering as much 70% accuracy. [Read blog](https://docs.unsloth.ai/new/quantization-aware-training-qat) <a href="https://docs.unsloth.ai/new/quantization-aware-training-qat" target="_blank" rel="noopener noreferrer"><img width="200" height="3100" alt="qat2" src="https://github.com/user-attachments/assets/7a39cec9-c1cc-4037-bcec-693885fc17c2" /></a> - Unsloth supports OpenEnv to allow for **open RL environments**. Blog coming soon • [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb) - New [customer support agent](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb) notebook to enable real-time analysis & solving of customer interactions. You'll also learn how to train models using data from Google Sheets. - Support for **Python 3.13**, **PyTorch 2.9** and the latest Hugging Face TRL and transformers are now fixed. > [!TIP] > Update Unsloth via `pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth unsloth_zoo` > If you want PyTorch 2.9: `pip install --upgrade unsloth unsloth_zoo` ### RL Improvements 1. Fixed [Standby](https://docs.unsloth.ai/get-started/reinforcement-learning-rl-guide/memory-efficient-rl) consuming more VRAM than usual. Auto selects the maximum 80% to 95% of GPU utilization if `import os; os.environ["UNSLOTH_VLLM_STANDBY"] = "1"` is used. 2. Fixed GRPO training hangs with better environment timers - works on DGX Spark and all other GPUs. 4. Fixes GRPO `RuntimeError: shape '[1, 887, 1, 128]' is invalid for input of size 3633152` for all models ### RL Environment functions 1. New `execute_with_time_limit` function to force functions to execute within a time limit. E.g. with a 2 second time limit, use: ```python from unsloth import execute_with_time_limit @execute_with_time_limit(2) def execute_strategy(strategy, game): return _execute_strategy(strategy, game) try: execute_strategy(strategy, game) except TimeoutError as e: print(f"Timed out with error = {str(e)}") ``` 2. To check if only Python standard modules are used in a function, use `check_python_modules`. 3. Use `create_locked_down_function` to create a function without leakage of global variables. 4. Use `Benchmarker` ie `from unsloth import Benchmarker` to benchmark functions accurately. It wipes the L1 to L3 cache approximately to reduce chances of benchmark cheating. 5. Use `launch_openenv` to launch a continuous reloaded OpenEnv environment process (to stop it from closing down) ie `from unsloth import launch_openenv` It will auto find a port that is not used. ### Bug fixes 1. GPT-OSS BF16 The GPTOSSRouter works with `load_in_4bit = True` AttributeError: 'GptOssTopKRouter' object has no attribute 'weight' 2. Mistral training fixed - sentencepiece proto issue fixed (any protobuf version works) 3. Fix evaluation ie `UNSLOTH_RETURN_LOGITS="1"` works. Fixes https://github.com/unslothai/unsloth/issues/3126 https://github.com/unslothai/unsloth/issues/3071 4. Fixes `Output 0 of UnslothFusedLossBackward is a view and is being modified inplace.` for Gemma 3 and `transformers>=4.57.1` 6. If you see `ImportError: cannot import name '_Ink' from 'PIL._typing' (/usr/local/lib/python3.12/dist-packages/PIL/_typing.py)` please update and use our new notebooks **Don't forget to also join our Reddit: [r/unsloth](https://www.reddit.com/r/unsloth/) 🥰** ## What's Changed * Fix loading as 8bit by @Etherll in https://github.com/unslothai/unsloth/pull/3384 * Nightly by @danielhanchen in https://github.com/unslothai/unsloth/pull/3392 * Nightly by @danielhanchen in https://github.com/unslothai/unsloth/pull/3394 * Update int8-int4 QAT config to use Int8DynamicActivationIntxWeightConfig by @metascroy in https://github.com/unslothai/unsloth/pull/3391 * Gemma 3 bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/3410 * Transformers Fix v4.57 rename from PretrainedConfig to PreTrainedConfig by @mmathew23 in https://github.com/unslothai/unsloth/pull/3445 * improve qat by @Etherll in https://github.com/unslothai/unsloth/pull/3446 * Fix eval metric issue by @pluesclues in https://github.com/unslothai/unsloth/pull/3420 * [Part2] Reinstate llama.cpp Compatibility and GGUF Conversion with Multiple Quantizations and Automated Ollama Modelfile Creation by @rolandtannous in https://github.com/unslothai/unsloth/pull/3356 * vLLM FP8 quantized support for SFT/GRPO by @Datta0 in https://github.com/unslothai/unsloth/pull/3414 * Fix by @danielhanchen in https://github.com/unslothai/unsloth/pull/3466 * AMD fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/3467 * Fix transformers 4.57.1 by @danielhanchen in https://github.com/unslothai/unsloth/pull/3473 * GRPO bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/3474 * EOL LF (unix line endings) normalization by @djsaunde in https://github.com/unslothai/unsloth/pull/3478 * Fix out of resources issue for llama3.2 sft on amd gpu by @wangxunx in https://github.com/unslothai/unsloth/pull/3455 * Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/3483 * Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/3484 * Patch sleep mode properly for trl by @Datta0 in https://github.com/unslothai/unsloth/pull/3492 * Sleep trl patch by @Datta0 in https://github.com/unslothai/unsloth/pull/3494 * fix cross entropy loss issue for small vocab size on amd gpu by @wangxunx in https://github.com/unslothai/unsloth/pull/3503 * Gemma 3n fix by @mmathew23 in https://github.com/unslothai/unsloth/pull/3499 * enable intel for torch2.8 by @leizhenyuan in https://github.com/unslothai/unsloth/pull/3381 * add code for intel qlora by @leizhenyuan in https://github.com/unslothai/unsloth/pull/3370 * fix for intel memory calculation by @leizhenyuan in https://github.com/unslothai/unsloth/pull/3513 * [intel] enable support 2.9 for intel xpu by @leizhenyuan in https://github.com/unslothai/unsloth/pull/3514 * FP8 training enhancements by @Datta0 in https://github.com/unslothai/unsloth/pull/3496 ## New Contributors * @metascroy made their first contribution in https://github.com/unslothai/unsloth/pull/3391 * @djsaunde made their first contribution in https://github.com/unslothai/unsloth/pull/3478 * @wangxunx made their first contribution in https://github.com/unslothai/unsloth/pull/3455 **Full Changelog**: https://github.com/unslothai/unsloth/compare/September-2025-v3...October-2025

We’re excited to introduce Unsloth Flex Attention support for OpenAI gpt-oss training that enables **>8× longer context** lengths, **>50% less VRAM** usage and **>1.5× faster** training compared to all implementations including those using Flash Attention 3 (FA3). Unsloth Flex Attention makes it possible to train with a 60K context length on just 80GB of VRAM for BF16 LoRA. Also: - You can now export/save your QLoRA fine-tuned gpt-oss model to llama.cpp, vLLM, or HF. - We fixed gpt-oss training losses going to infinity on float16 GPUs (like T4 Colab) - We fixed gpt-oss implementation issues, most notably ensuring that `swiglu_limit = 7.0` is properly applied during MXFP4 inference [in transformers](https://github.com/huggingface/transformers/pull/40197) Full details in our blogpost: https://docs.unsloth.ai/basics/long-context-gpt-oss-training <a href="https://docs.unsloth.ai/basics/long-context-gpt-oss-training"><img src="https://github.com/user-attachments/assets/ea47ed4a-7a29-4c75-b9ac-2de4e13c91e5" width="600"></a> ## What's Changed * Add Qwen3 Instruct / Thinking chat templates by @Etherll in https://github.com/unslothai/unsloth/pull/3110 * Add Qwen3 4B to mapper.py by @Etherll in https://github.com/unslothai/unsloth/pull/3120 * Nightly by @danielhanchen in https://github.com/unslothai/unsloth/pull/3148 * Fix GPT OSS by @danielhanchen in https://github.com/unslothai/unsloth/pull/3154 * Nightly by @danielhanchen in https://github.com/unslothai/unsloth/pull/3169 * Update Blackwell install instructions for latest vLLM release by @qingy1337 in https://github.com/unslothai/unsloth/pull/3175 * Fix potential generator exhaustion bug in model loading file detection by @rolandtannous in https://github.com/unslothai/unsloth/pull/3167 * Fix vision model GGUF quantization_method error type by @rolandtannous in https://github.com/unslothai/unsloth/pull/3173 * Replace back ticks with single quotes by @rnowling in https://github.com/unslothai/unsloth/pull/3157 * Fix original_push_to_hub fallback by @Thiraput01 in https://github.com/unslothai/unsloth/pull/3115 * Add support for QAT + LoRA by @andrewor14 in https://github.com/unslothai/unsloth/pull/2976 * Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/3180 * Torch 2.8 by @danielhanchen in https://github.com/unslothai/unsloth/pull/3186 * Fix extras transformers typo in pyproject.toml by @parth2510 in https://github.com/unslothai/unsloth/pull/3187 * Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/3195 * allow torch.float32 dtype in FastLanguageModel by @mmathew23 in https://github.com/unslothai/unsloth/pull/3204 * fix is casual for qwen3 by @leizhenyuan in https://github.com/unslothai/unsloth/pull/3213 * Support `model.save_pretrained_torchao` by @jerryzh168 in https://github.com/unslothai/unsloth/pull/3111 * Fix gemma-3n by @mmathew23 in https://github.com/unslothai/unsloth/pull/3219 * Handle transformers move to dtype from torch_dtype by @mmathew23 in https://github.com/unslothai/unsloth/pull/3225 * chore: Fix Typos by @DefiWimar7 in https://github.com/unslothai/unsloth/pull/3224 ## New Contributors * @rnowling made their first contribution in https://github.com/unslothai/unsloth/pull/3157 * @Thiraput01 made their first contribution in https://github.com/unslothai/unsloth/pull/3115 * @andrewor14 made their first contribution in https://github.com/unslothai/unsloth/pull/2976 * @parth2510 made their first contribution in https://github.com/unslothai/unsloth/pull/3187 * @jerryzh168 made their first contribution in https://github.com/unslothai/unsloth/pull/3111 * @DefiWimar7 made their first contribution in https://github.com/unslothai/unsloth/pull/3224 **Full Changelog**: https://github.com/unslothai/unsloth/compare/August-2025...August-2025-v2

> [!TIP] > Update Unsloth via `pip install --upgrade --force-reinstall unsloth unsloth_zoo` # New Features **🔥 Gemma 3n Models Now Available** - Download optimized Gemma 3n models in 2B (E2B) and 4B (E4B) sizes - Available in safetensors, GGUF quantized format, and dynamic 4bit bnb quantization for finetuning. - HuggingFace Collection Link: [Gemma3-N](https://huggingface.co/collections/unsloth/gemma-3n-685d3874830e49e1c93f9339) **Key Gemma 3n Architecture Features:** - **Multimodal Support**: Audio, visual, and text input processing capabilities - **PLE Caching**: Per-Layer Embedding parameters can be cached to reduce memory usage - **MatFormer Architecture**: Matryoshka Transformer with nested models for flexible compute - **Conditional Parameter Loading**: Skip loading audio/visual parameters to save memory - **Parameter Efficiency**: E2B runs with ~1.91B effective parameters despite 5B+ total - **Wide Language Support**: Trained on 140+ languages with 32K token context **🧠 DeepSeek-R1 Support with Dynamic 1-bit GGUFs** - Fine-tune DeepSeek-R1-0528-Qwen3 with GRPO! Our new reward function increases multilingual response rates by 40%+ - Dynamic 1-bit GGUFs shrink the full 715GB model to just 185GB (-75% size) with optimal accuracy - DeepSeek-R1 notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_(8B)_GRPO.ipynb **📈 Dynamic 2.0 GGUFs** - New quantization method outperforms leading quantization methods - Sets new benchmarks for 5-shot MMLU and KL Divergence - Selectively quantizes layers for optimal accuracy - For more information: https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs **🎵 Text-to-Speech (TTS) Fine-tuning** - Train TTS models like Sesame-CSM and OpenAI's Whisper locally! - Clone voices, learn new emotions, tones & styles with 1.5x faster training and -50% VRAM - TTS notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Sesame_CSM_(1B)-TTS.ipynb **🔧 Native Tool Calling Support** - DeepSeek-Qwen3 now supports native tool calling with 93.25% on BFCL Berkeley Function-Calling Leaderboard - Compatible with llama.cpp (use `--jinja`), transformers, and vLLM - Qwen2.5 Coder tool calling: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb **📊 Synthetic Data Generation Partnership with Meta** - Turn documents into high-quality synthetic datasets using Llama - Parse PDFs, websites, videos and auto-generate QA pairs - Meta partnership notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_(3B).ipynb **⚡ Advanced Qwen3 GRPO notebook** - Proximity scoring for more nuanced reward functions - OpenR1 dataset support with advanced templates - Prefinetuning to skip GRPO format learning - https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb ```python # DeepSeek-R1 GRPO Fine-tuning Example: convert DeepSeek-R1-0528-Qwen3-8B into a reasoning model via GRPO by using OpenR1's Math dataset. from unsloth import FastLanguageModel import torch max_seq_length = 1024 # Can increase for longer reasoning traces lora_rank = 32 # Larger rank = smarter, but slower model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/DeepSeek-R1-0528-Qwen3-8B", max_seq_length = max_seq_length, load_in_4bit = True, # False for LoRA 16bit fast_inference = True, # Enable vLLM fast inference max_lora_rank = lora_rank, gpu_memory_utilization = 0.7, # Reduce if out of memory ) model = FastLanguageModel.get_peft_model( model, r = lora_rank, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_alpha = lora_rank*2, # *2 speeds up training use_gradient_checkpointing = "unsloth", # Reduces memory usage random_state = 3407, ) reasoning_start = None reasoning_end = None user_token = None assistant_token = None for token in tokenizer.get_added_vocab().keys(): if "think" in token and "/" in token: reasoning_end = token elif "think" in token: reasoning_start = token elif "user" in token: user_token = token elif "assistant" in token: assistant_token = token system_prompt = \ f"""You are given a problem. Think about the problem and provide your working out. You must think in Bahasa Indonesia.""" print(tokenizer.apply_chat_template([ {"role" : "user", "content" : "What is 1+1?"}, {"role" : "assistant", "content" : f"<think>I think it's 2.2</think>2"}, {"role" : "user", "content" : "What is 1+1?"}, {"role" : "assistant", "content" : f"<think>I think it's 2.2</think>2"}, ], tokenize = False, add_generation_prompt = True)) from datasets import load_dataset dataset = load_dataset("open-r1/DAPO-Math-17k-Processed", "en", split = "train") def extract_hash_answer(text): # if "####" not in text: return None # return text.split("####")[1].strip() return text dataset = dataset.map(lambda x: { "prompt" : [ {"role": "system", "content": system_prompt}, {"role": "user", "content": x["prompt"]}, ], "answer": extract_hash_answer(x["solution"]), }) # Add optional EOS token matching solution_end_regex = rf"{reasoning_end}(.*)" match_format = re.compile(solution_end_regex, re.DOTALL) match_format """We verify it works:""" match_format.findall( "Let me think!</think>"\ f"Hence, the solution is 2.", ) match_format.findall( "<think>Let me think!</think>"\ f"\n\nHence, the solution is 2", ) def match_format_exactly(completions, **kwargs): scores = [] for completion in completions: score = 0 response = completion[0]["content"] # Match if format is seen exactly! if match_format.search(response) is not None: score += 3.0 scores.append(score) return scores """If it fails, we want to reward the model if it at least follows the format partially, by counting each symbol:""" def match_format_approximately(completions, **kwargs): scores = [] for completion in completions: score = 0 response = completion[0]["content"] # Count how many keywords are seen - we penalize if too many! # If we see 1, then plus some points! # No need to reward <think> since we always prepend it! score += 0.5 if response.count(reasoning_start) == 1 else -1.0 score += 0.5 if response.count(reasoning_end) == 1 else -1.0 scores.append(score) return scores """We want to extract the generated answer, and reward or penalize it! We also reward it based on how close the answer is to the true one via ratios:""" def check_answer(prompts, completions, answer, **kwargs): question = prompts[0][-1]["content"] responses = [completion[0]["content"] for completion in completions] extracted_responses = [ guess.group(1) if (guess := match_format.search(r)) is not None else None \ for r in responses ] scores = [] for guess, true_answer in zip(extracted_responses, answer): score = 0 if guess is None: scores.append(-2.0) continue # Correct answer gets 5 points! if guess == true_answer: score += 5.0 # Match if spaces are seen, but less reward elif guess.strip() == true_answer.strip(): score += 3.5 else: # We also reward it if the answer is close via ratios! # Ie if the answer is within some range, reward it! try: ratio = float(guess) / float(true_answer) if ratio >= 0.9 and ratio <= 1.1: score += 2.0 elif ratio >= 0.8 and ratio <= 1.2: score += 1.5 else: score -= 2.5 # Penalize wrong answers except: score -= 4.5 # Penalize scores.append(score) return scores match_numbers = re.compile( r".*?[\s]{0,}([-]?[\d\.\,]{1,})", flags = re.MULTILINE | re.DOTALL ) print(match_numbers.findall(" 0.34 ")) print(match_numbers.findall(" 123,456 ")) print(match_numbers.findall(" -0.234 ")) print(match_numbers.findall("17")) import langid def get_lang(text: str) -> str: if not text: return "und" lang, _ = langid.classify(text) return lang print(get_lang("Hello, How are you")) # This should return en print(get_lang("Aku berpikir kalau aku adalah kamu")) # This should return id print(get_lang("我在这里")) # This should return zh import re def format_and_language_reward_func(completions, **kwargs): scores = [] for completion_item in completions: if not completion_item or not isinstance(completion_item[0], dict) or "content" not in completion_item[0]: scores.append(-5.0) print(f"Warning: Malformed completion item, assigning default low score: {completion_item}") continue content = completion_item[0]["content"] lang = get_lang(content) if lang == 'id': score = 5.0 elif lang == 'en': score = -3.0 elif lang == 'zh': score = -3.0 else: score = -5.0 scores.append(score) return scores prompts = [ [{"role": "assistant", "content": "What is the result of (1 + 2) * 4?"}], [{"role": "assistant", "content": "What is the result of (3 + 1) * 2?"}], ] completions = [ [{"role": "assistant", "content": "<think>The sum of 1 and 2 is 3, which we multiply by 4 to get 12.</think><answer>(1 + 2) * 4 = 12</answer>"}], [{"role": "assistant", "content": "The sum of 3 and 1 is 4, which we multiply by 2 to get 8. So (3 + 1) * 2 = 8."}], ] format_and_language_reward_func(prompts=prompts, completions=completions) global PRINTED_TIMES PRINTED_TIMES = 0 global PRINT_EVERY_STEPS PRINT_EVERY_STEPS = 5 def check_numbers(prompts, completions, answer, **kwargs): question = prompts[0][-1]["content"] responses = [completion[0]["content"] for completion in completions] extracted_responses = [ guess.group(1) if (guess := match_numbers.search(r)) is not None else None \ for r in responses ] scores = [] # Print only every few steps global PRINTED_TIMES global PRINT_EVERY_STEPS if PRINTED_TIMES % PRINT_EVERY_STEPS == 0: print( '*'*20 + f"Question:\n{question}", f"\nAnswer:\n{answer[0]}", f"\nResponse:\n{responses[0]}", f"\nExtracted:\n{extracted_responses[0]}" ) PRINTED_TIMES += 1 for guess, true_answer in zip(extracted_responses, answer): if guess is None: scores.append(-2.5) continue # Convert to numbers try: true_answer = float(true_answer.strip()) # Remove commas like in 123,456 guess = float(guess.strip().replace(",", "")) scores.append(3.5 if guess == true_answer else -1.5) except: scores.append(0) continue return scores tokenized = dataset.map( lambda x: {"tokens" : tokenizer.apply_chat_template(x["prompt"], add_generation_prompt = True, tokenize = True)}, batched = True, ) print(tokenizer.decode(tokenized[0]["tokens"])) tokenized = tokenized.map(lambda x: {"L" : len(x["tokens"])}) import numpy as np maximum_length = int(np.quantile(tokenized["L"], 0.9)) print("Max Length = ", maximum_length) # Filter only samples smaller than 90% max length dataset = dataset.select(np.where(np.array(tokenized["L"]) <= maximum_length)[0]) del tokenized max_prompt_length = maximum_length + 1 # + 1 just in case! max_completion_length = max_seq_length - max_prompt_length from vllm import SamplingParams vllm_sampling_params = SamplingParams( min_p = 0.1, top_p = 1.0, top_k = -1, seed = 3407, stop = [tokenizer.eos_token], include_stop_str_in_output = True, ) from trl import GRPOConfig, GRPOTrainer training_args = GRPOConfig( vllm_sampling_params = vllm_sampling_params, temperature = 1.0, learning_rate = 5e-6, weight_decay = 0.01, warmup_ratio = 0.1, lr_scheduler_type = "linear", optim = "adamw_8bit", logging_steps = 1, per_device_train_batch_size = 1, gradient_accumulation_steps = 1, # Increase to 4 for smoother training num_generations = 4, # Decrease if out of memory max_prompt_length = max_prompt_length, max_completion_length = max_completion_length, # num_train_epochs = 1, # Set to 1 for a full training run max_steps = 100, save_steps = 100, report_to = "none", # Can use Weights & Biases output_dir = "outputs", # For optional training + evaluation # fp16_full_eval = True, # per_device_eval_batch_size = 4, # eval_accumulation_steps = 1, # eval_strategy = "steps", # eval_steps = 1, ) trainer = GRPOTrainer( model = model, processing_class = tokenizer, reward_funcs = [ match_format_exactly, match_format_approximately, check_answer, check_numbers, format_and_language_reward_func, ], args = training_args, train_dataset = dataset, # For optional training + evaluation # train_dataset = new_dataset["train"], # eval_dataset = new_dataset["test"], ) trainer.train() ``` **🎯 Magistral Conversational Reasoning** - Fine-tune Magistral-24B for advanced conversational reasoning - Magistral notebook: https://github.com/unslothai/notebooks/blob/main/nb/Magistral_(24B)-Reasoning-Conversational.ipynb **👁️ Gemma3 Vision Support** - Fine-tune Gemma3 vision models for multimodal tasks - Gemma3 Vision notebook: https://github.com/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb ## Documentation & Guides - **Reinforcement Learning Guide**: Complete guide on RL for LLMs covering GRPO, RLHF, DPO. Check it out here: https://docs.unsloth.ai/basics/reinforcement-learning-guide - **LoRA Hyperparameters Guide**: Master optimal learning rates, epochs, LoRA rank & alpha settings, Check it out here: https://docs.unsloth.ai/get-started/fine-tuning-guide/lora-hyperparameters-guide ## What's Changed * Nightly by @danielhanchen in https://github.com/unslothai/unsloth/pull/2448 * Added k_norm & q_norm to merged Qwen3 layers by @cblomert in https://github.com/unslothai/unsloth/pull/2452 * MoE Kernel by @jeromeku in https://github.com/unslothai/unsloth/pull/2465 * Blackwell Support by @johnnynunez in https://github.com/unslothai/unsloth/pull/2458 * Added missing code of conduct by @rolandtannous in https://github.com/unslothai/unsloth/pull/2416 * Fix readme example by @yuanzhedong in https://github.com/unslothai/unsloth/pull/2492 * the pixtral vision notebook fails during inference by @mmathew23 in https://github.com/unslothai/unsloth/pull/2466 * [1/N] Enable intel GPU for unsloth by @leizhenyuan in https://github.com/unslothai/unsloth/pull/2350 * [2/N] Enable intel GPU for unsloth by @leizhenyuan in https://github.com/unslothai/unsloth/pull/2388 * vLLM Windows CUDA support [tested] by @fenglui in https://github.com/unslothai/unsloth/pull/2158 * Add Sesame CSM by @mmathew23 in https://github.com/unslothai/unsloth/pull/2527 * Add Qwen-3 chat template and Ollama template support by @kiankyars in https://github.com/unslothai/unsloth/pull/2537 * Fix typos by @omahs in https://github.com/unslothai/unsloth/pull/2540 * Add use_rslora reference to LoraConfig inititalisation by @jkumz in https://github.com/unslothai/unsloth/pull/2539 * TTS by @danielhanchen in https://github.com/unslothai/unsloth/pull/2545 * Quick fix on the CompileConfig error by @Erland366 in https://github.com/unslothai/unsloth/pull/2554 * Fix trust remote code by @Etherll in https://github.com/unslothai/unsloth/pull/2357 * fix issue with qwen3 template double quote escapes by @davedgd in https://github.com/unslothai/unsloth/pull/2563 * Display the model name in RoPE scaling unsupported error by @emmanuel-ferdman in https://github.com/unslothai/unsloth/pull/2564 * Fix Whisper, ModernBERT by @danielhanchen in https://github.com/unslothai/unsloth/pull/2565 * fix: improved error handling when llama.cpp build fails #2358 by @Hansehart in https://github.com/unslothai/unsloth/pull/2603 * Remove `dataset_text_field` from `SFTConfig` by @qgallouedec in https://github.com/unslothai/unsloth/pull/2609 * Upgrade trl fix by @Datta0 in https://github.com/unslothai/unsloth/pull/2544 * Check the `skip_prepare_dataset` before accessing dataset fields. #2496 by @Premik in https://github.com/unslothai/unsloth/pull/2633 * Llama4 MoE Grouped GEMM by @jeromeku in https://github.com/unslothai/unsloth/pull/2639 * Latest TRL, GRPO + Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/2645 * Fix SFTtraining for new trl by @mmathew23 in https://github.com/unslothai/unsloth/pull/2647 * Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/2651 * Fix quant model param fetch regex by @Datta0 in https://github.com/unslothai/unsloth/pull/2662 * Fix batched generation for prompts of different lengths by @RunFMe in https://github.com/unslothai/unsloth/pull/2216 * reroute merge logic language models + comprehensive tests + eval kits by @rolandtannous in https://github.com/unslothai/unsloth/pull/2673 * unsloth checkpointing fix for latest transformers==4.52.x by @mmathew23 in https://github.com/unslothai/unsloth/pull/2674 * patch sft_trainer to favor max_seq_length over max_length in config by @mmathew23 in https://github.com/unslothai/unsloth/pull/2669 * Update prepare 4d causal attention call by @mmathew23 in https://github.com/unslothai/unsloth/pull/2678 * Ignore None Values when building vllm subprocess_command by @Salpingopharyngeus in https://github.com/unslothai/unsloth/pull/2680 * add support for torch270 with Intel GPU by @leizhenyuan in https://github.com/unslothai/unsloth/pull/2709 * Making protobuf version more flexible by @user799595 in https://github.com/unslothai/unsloth/pull/2637 * tests for additional merge fix unsloth zoo pr 163 by @rolandtannous in https://github.com/unslothai/unsloth/pull/2719 * Reward modeling update (There seems to be another patch) by @pluesclues in https://github.com/unslothai/unsloth/pull/2710 * Fix Typos in Documentation and Comments by @leopardracer in https://github.com/unslothai/unsloth/pull/2721 * Fix renaming on other model than Llama by @Erland366 in https://github.com/unslothai/unsloth/pull/2762 * Enable vLLM to share memory space by @Datta0 in https://github.com/unslothai/unsloth/pull/2712 * Fix TRL 1.8.2 by @marcandrelarochelle in https://github.com/unslothai/unsloth/pull/2774 * Fix AttributeError in GRPO trainer for models without llm attribute by @rolandtannous in https://github.com/unslothai/unsloth/pull/2780 * Additional tests for unsloth-zoo PR#174 by @rolandtannous in https://github.com/unslothai/unsloth/pull/2779 * Update pyproject.toml by @amrothemich in https://github.com/unslothai/unsloth/pull/2778 * Fix for grpo_compute_loss_slow by @simpissa in https://github.com/unslothai/unsloth/pull/2702 * Fix GRPO by @danielhanchen in https://github.com/unslothai/unsloth/pull/2787 * Docs: Fix typo and improve MoE docstrings by @kilavvy in https://github.com/unslothai/unsloth/pull/2784 * [5/N] Enable intel GPU for unsloth by @leizhenyuan in https://github.com/unslothai/unsloth/pull/2768 * Sequence Classification Bug Fixes by @pluesclues in https://github.com/unslothai/unsloth/pull/2793 * intel 5/N fix patch by @mmathew23 in https://github.com/unslothai/unsloth/pull/2792 * [3/N] Enable intel GPU for unsloth by @leizhenyuan in https://github.com/unslothai/unsloth/pull/2620 * [4/N] Enable intel GPU for unsloth by @mmathew23 in https://github.com/unslothai/unsloth/pull/2801 * [intel] use DeviceProperties instead of torch.xxx.deviceproperties by @leizhenyuan in https://github.com/unslothai/unsloth/pull/2803 * Fix grpo sleep regex and indentation by @Datta0 in https://github.com/unslothai/unsloth/pull/2804 * Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/2805 * Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/2807 ## New Contributors * @cblomert made their first contribution in https://github.com/unslothai/unsloth/pull/2452 * @johnnynunez made their first contribution in https://github.com/unslothai/unsloth/pull/2458 * @rolandtannous made their first contribution in https://github.com/unslothai/unsloth/pull/2416 * @yuanzhedong made their first contribution in https://github.com/unslothai/unsloth/pull/2492 * @mmathew23 made their first contribution in https://github.com/unslothai/unsloth/pull/2466 * @leizhenyuan made their first contribution in https://github.com/unslothai/unsloth/pull/2350 * @fenglui made their first contribution in https://github.com/unslothai/unsloth/pull/2158 * @kiankyars made their first contribution in https://github.com/unslothai/unsloth/pull/2537 * @omahs made their first contribution in https://github.com/unslothai/unsloth/pull/2540 * @jkumz made their first contribution in https://github.com/unslothai/unsloth/pull/2539 * @davedgd made their first contribution in https://github.com/unslothai/unsloth/pull/2563 * @emmanuel-ferdman made their first contribution in https://github.com/unslothai/unsloth/pull/2564 * @qgallouedec made their first contribution in https://github.com/unslothai/unsloth/pull/2609 * @Premik made their first contribution in https://github.com/unslothai/unsloth/pull/2633 * @RunFMe made their first contribution in https://github.com/unslothai/unsloth/pull/2216 * @Salpingopharyngeus made their first contribution in https://github.com/unslothai/unsloth/pull/2680 * @user799595 made their first contribution in https://github.com/unslothai/unsloth/pull/2637 * @pluesclues made their first contribution in https://github.com/unslothai/unsloth/pull/2710 * @leopardracer made their first contribution in https://github.com/unslothai/unsloth/pull/2721 * @marcandrelarochelle made their first contribution in https://github.com/unslothai/unsloth/pull/2774 * @amrothemich made their first contribution in https://github.com/unslothai/unsloth/pull/2778 * @simpissa made their first contribution in https://github.com/unslothai/unsloth/pull/2702 * @kilavvy made their first contribution in https://github.com/unslothai/unsloth/pull/2784 **Full Changelog**: https://github.com/unslothai/unsloth/compare/May-2025...June-2025

## Qwen 3 support + bug fixes Please update Unsloth via `pip install --upgrade --force-reinstall unsloth unsloth_zoo` Qwen 3 notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb  There are also many bug fixes in this release! The 30B MoE is also fine-tunable in Unsloth! ```python from unsloth import FastModel import torch model, tokenizer = FastModel.from_pretrained( model_name = "unsloth/Qwen3-30B-A3B", max_seq_length = 2048, # Choose any for long context! load_in_4bit = True, # 4 bit quantization to reduce memory load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory full_finetuning = False, # [NEW!] We have full finetuning now! # token = "hf_...", # use one if using gated models ) ``` ## What's Changed * GGUF saving by @danielhanchen in https://github.com/unslothai/unsloth/pull/2017 * Gemma 3 readme by @danielhanchen in https://github.com/unslothai/unsloth/pull/2019 * Update README.md by @danielhanchen in https://github.com/unslothai/unsloth/pull/2028 * bug fix #2008 - load_in_4bit = True + fast_inference = True by @void-mckenzie in https://github.com/unslothai/unsloth/pull/2039 * unsloth_fast_generate model is not defined fix by @KareemMusleh in https://github.com/unslothai/unsloth/pull/2051 * Ensure trust_remote_code propagates down to unsloth_compile_transformers by @CuppaXanax in https://github.com/unslothai/unsloth/pull/2075 * Show `peft_error` by @IsaacBreen in https://github.com/unslothai/unsloth/pull/2080 * Add generation prompt error message change by @KareemMusleh in https://github.com/unslothai/unsloth/pull/2046 * Many bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/2087 * fix: config.torch_dtype in LlamaModel_fast_forward_inference by @lurf21 in https://github.com/unslothai/unsloth/pull/2091 * Updating new FFT 8bit support by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/2110 * Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/2113 * Small fix by @danielhanchen in https://github.com/unslothai/unsloth/pull/2114 * fix(utils): add missing importlib import to fix NameError by @naliazheli in https://github.com/unslothai/unsloth/pull/2134 * Add QLoRA Train and Merge16bit Test by @jeromeku in https://github.com/unslothai/unsloth/pull/2130 * Fix Transformers 4.45 by @danielhanchen in https://github.com/unslothai/unsloth/pull/2151 * Bug Fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/2197 * Issues templates by @jeromeku in https://github.com/unslothai/unsloth/pull/2242 * Fix feature_request ISSUE_TEMPLATE by @jeromeku in https://github.com/unslothai/unsloth/pull/2250 * Registry refactor by @jeromeku in https://github.com/unslothai/unsloth/pull/2255 * Update README.md by @Kimizhao in https://github.com/unslothai/unsloth/pull/2267 * Update README.md by @jackswl in https://github.com/unslothai/unsloth/pull/2119 * Update bug_report.md by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/2323 * feat: Support custom `auto_model` for wider model compatibility (Whisper, Bert,etc) & `attn_implementation` support by @Etherll in https://github.com/unslothai/unsloth/pull/2263 * fix: improved error handling when llama.cpp build fails by @Hansehart in https://github.com/unslothai/unsloth/pull/2358 * Revert "fix: improved error handling when llama.cpp build fails" by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/2375 * Fix saving 4bit for VLM by @Erland366 in https://github.com/unslothai/unsloth/pull/2381 * [WIP] Initial support for Qwen3. Will udpate when the model is released by @Datta0 in https://github.com/unslothai/unsloth/pull/2211 * Fixup qwen3 by @Datta0 in https://github.com/unslothai/unsloth/pull/2423 * Fixup qwen3 qk norm by @Datta0 in https://github.com/unslothai/unsloth/pull/2427 * Qwen3 inference fixes by @Datta0 in https://github.com/unslothai/unsloth/pull/2436 * Update mapper.py to add Qwen3 base by @Etherll in https://github.com/unslothai/unsloth/pull/2439 * Qwen 3, Bug Fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/2445 ## New Contributors * @void-mckenzie made their first contribution in https://github.com/unslothai/unsloth/pull/2039 * @CuppaXanax made their first contribution in https://github.com/unslothai/unsloth/pull/2075 * @IsaacBreen made their first contribution in https://github.com/unslothai/unsloth/pull/2080 * @lurf21 made their first contribution in https://github.com/unslothai/unsloth/pull/2091 * @naliazheli made their first contribution in https://github.com/unslothai/unsloth/pull/2134 * @jeromeku made their first contribution in https://github.com/unslothai/unsloth/pull/2130 * @Kimizhao made their first contribution in https://github.com/unslothai/unsloth/pull/2267 * @jackswl made their first contribution in https://github.com/unslothai/unsloth/pull/2119 * @Etherll made their first contribution in https://github.com/unslothai/unsloth/pull/2263 * @Hansehart made their first contribution in https://github.com/unslothai/unsloth/pull/2358 **Full Changelog**: https://github.com/unslothai/unsloth/compare/2025-03...May-2025

# 90% less memory usage GRPO ## Update Unsloth via `pip install --upgrade --no-cache-dir unsloth unsloth_zoo` More details in blog post: https://unsloth.ai/blog/grpo Llama 3.1 8B GRPO Colab: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb Metric | Unsloth | TRL + FA2 -- | -- | -- Training Memory Cost (GB) | 42GB | 414GB GRPO Memory Cost (GB) | 9.8GB | 78.3GB Inference Cost (GB) | 0GB | 16GB Inference KV Cache for 20K context (GB) | 2.5GB | 2.5GB Total Memory Usage | 54.3GB (90% less) | 510.8GB You automatically get 90% less memory usage! Also all reward logs for individual reward functions will show up.  Script to run GRPO: ```python !pip install unsloth vllm from unsloth import FastLanguageModel import torch max_seq_length = 1024 # Can increase for longer reasoning traces lora_rank = 32 # Larger rank = smarter, but slower model, tokenizer = FastLanguageModel.from_pretrained( model_name = "meta-llama/meta-Llama-3.1-8B-Instruct", max_seq_length = max_seq_length, load_in_4bit = True, # False for LoRA 16bit fast_inference = True, # Enable vLLM fast inference max_lora_rank = lora_rank, gpu_memory_utilization = 0.6, # Reduce if out of memory ) model = FastLanguageModel.get_peft_model( model, r = lora_rank, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], # Remove QKVO if out of memory lora_alpha = lora_rank, use_gradient_checkpointing = "unsloth", # Enable long context finetuning random_state = 3407, ) import re from datasets import load_dataset, Dataset global COUNTER COUNTER = 0 global PRINT_EVERY PRINT_EVERY = 20 # Load and prep dataset SYSTEM_PROMPT = """ Respond in the following format: <reasoning> ... </reasoning> <answer> ... </answer> """ XML_COT_FORMAT = """\ <reasoning> {reasoning} </reasoning> <answer> {answer} </answer> """ def extract_xml_answer(text: str) -> str: answer = text.split("<answer>")[-1] answer = answer.split("</answer>")[0] return answer.strip() def extract_hash_answer(text: str) -> str | None: if "####" not in text: return None return text.split("####")[1].strip() # uncomment middle messages for 1-shot prompting def get_gsm8k_questions(split = "train") -> Dataset: data = load_dataset('openai/gsm8k', 'main')[split] # type: ignore data = data.map(lambda x: { # type: ignore 'prompt': [ {'role': 'system', 'content': SYSTEM_PROMPT}, {'role': 'user', 'content': x['question']} ], 'answer': extract_hash_answer(x['answer']) }) # type: ignore return data # type: ignore dataset = get_gsm8k_questions() # Reward functions def correctness_reward_func(prompts, completions, answer, **kwargs) -> list[float]: responses = [completion[0]['content'] for completion in completions] q = prompts[0][-1]['content'] extracted_responses = [extract_xml_answer(r) for r in responses] global COUNTER if COUNTER % PRINT_EVERY == 0: print('-'*20, f"Question:\n{q}", f"\nAnswer:\n{answer[0]}", f"\nResponse:\n{responses[0]}", f"\nExtracted:\n{extracted_responses[0]}") COUNTER += 1 return [2.0 if r == a else 0.0 for r, a in zip(extracted_responses, answer)] def int_reward_func(completions, **kwargs) -> list[float]: responses = [completion[0]['content'] for completion in completions] extracted_responses = [extract_xml_answer(r) for r in responses] return [0.5 if r.isdigit() else 0.0 for r in extracted_responses] def strict_format_reward_func(completions, **kwargs) -> list[float]: """Reward function that checks if the completion has a specific format.""" pattern = r"^<reasoning>\n.*?\n</reasoning>\n<answer>\n.*?\n</answer>\n$" responses = [completion[0]["content"] for completion in completions] matches = [re.match(pattern, r) for r in responses] return [0.5 if match else 0.0 for match in matches] def soft_format_reward_func(completions, **kwargs) -> list[float]: """Reward function that checks if the completion has a specific format.""" pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>" responses = [completion[0]["content"] for completion in completions] matches = [re.match(pattern, r) for r in responses] return [0.5 if match else 0.0 for match in matches] def count_xml(text) -> float: count = 0.0 if text.count("<reasoning>\n") == 1: count += 0.125 if text.count("\n</reasoning>\n") == 1: count += 0.125 if text.count("\n<answer>\n") == 1: count += 0.125 count -= len(text.split("\n</answer>\n")[-1])*0.001 if text.count("\n</answer>") == 1: count += 0.125 count -= (len(text.split("\n</answer>")[-1]) - 1)*0.001 return count def xmlcount_reward_func(completions, **kwargs) -> list[float]: contents = [completion[0]["content"] for completion in completions] return [count_xml(c) for c in contents] max_prompt_length = 256 from trl import GRPOConfig, GRPOTrainer # Optional extra params for vLLM from unsloth import vLLMSamplingParams vllm_sampling_params = vLLMSamplingParams( min_p = 0.01, seed = 3407, ) training_args = GRPOConfig( learning_rate = 5e-6, warmup_ratio = 0.1, lr_scheduler_type = "cosine", optim = "adamw_8bit", per_device_train_batch_size = 1, gradient_accumulation_steps = 1, # Increase to 4 for smoother training num_generations = 6, # Decrease if out of memory max_prompt_length = max_prompt_length, max_completion_length = max_seq_length - max_prompt_length, # num_train_epochs = 1, # Set to 1 for a full training run max_steps = 250, report_to = "none", # Can use Weights & Biases vllm_sampling_params = vllm_sampling_params, # Optional temperature = 1.0, ) trainer = GRPOTrainer( model = model, processing_class = tokenizer, reward_funcs = [ xmlcount_reward_func, soft_format_reward_func, strict_format_reward_func, int_reward_func, correctness_reward_func, ], args = training_args, train_dataset = dataset, ) trainer.train() ``` ## What's Changed * GRPO Bug fixes by @danielhanchen in https://github.com/unslothai/unsloth/pull/1623 * Fixes Triton url in README.md by @DiogoNeves in https://github.com/unslothai/unsloth/pull/1607 * Update README.md by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/1654 * Update README.md by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/1688 * Fix bugs by @danielhanchen in https://github.com/unslothai/unsloth/pull/1701 * Fix bugs by @danielhanchen in https://github.com/unslothai/unsloth/pull/1706 * Memory efficient GRPO, DPO etc by @danielhanchen in https://github.com/unslothai/unsloth/pull/1716 * Add GRPO metrics by @danielhanchen in https://github.com/unslothai/unsloth/pull/1718 * llama-quantize on WINDOWS WSL error fix - edit save.py (gguf saving breaks) by @everythingisc00l in https://github.com/unslothai/unsloth/pull/1649 * Update rl_replacements.py by @SethHWeidman in https://github.com/unslothai/unsloth/pull/1754 * Update README.md by @danielhanchen in https://github.com/unslothai/unsloth/pull/1768 * fix an import error by @NinoRisteski in https://github.com/unslothai/unsloth/pull/1767 * Gemma Mask convert to float by @Erland366 in https://github.com/unslothai/unsloth/pull/1762 * [Windows Support] Add latest `xformers` wheels to pyproject.toml by @versipellis in https://github.com/unslothai/unsloth/pull/1753 * Memory Efficient GRPO by @danielhanchen in https://github.com/unslothai/unsloth/pull/1773 ## New Contributors * @DiogoNeves made their first contribution in https://github.com/unslothai/unsloth/pull/1607 * @everythingisc00l made their first contribution in https://github.com/unslothai/unsloth/pull/1649 * @SethHWeidman made their first contribution in https://github.com/unslothai/unsloth/pull/1754 * @versipellis made their first contribution in https://github.com/unslothai/unsloth/pull/1753 **Full Changelog**: https://github.com/unslothai/unsloth/compare/2025-02...2025-02-v2

## GRPO is in Unsloth! - Experience the **"aha moment"** from DeepSeek R1's paper now with Unsloth! - LoRA (16bit) / QLoRA (4bit) actually work for GRPO now! - Unsloth can do GRPO for Phi-4 14B Llama-3.1 8B in a free 15GB Colab GPU! - Unsloth now has native fast inference (**20x more throughput**) via vLLM! Use it via `model.fast_generate` after setting `FastLanguageModel.from_pretrained(..., fast_inference = True)` and installing vLLM via `pip install vllm` - Llama 3.3 70B QLoRA GRPO should fit in 1x 48GB (best 1x 80GB) - Update unsloth via `pip install --upgrade --no-cache-dir --force-reinstall unsloth_zoo unsloth vllm`  Minimal GRPO example (courtesy of [Will Brown](https://gist.github.com/willccbb/4676755236bb08cab5f4e54a0475d6fb)] ```python !pip install unsloth vllm !pip install git+https://github.com/huggingface/trl.git from unsloth import FastLanguageModel, PatchFastRL PatchFastRL("GRPO", FastLanguageModel) from unsloth import is_bfloat16_supported import torch max_seq_length = 512 lora_rank = 32 model, tokenizer = FastLanguageModel.from_pretrained( model_name = "meta-llama/meta-Llama-3.1-8B-Instruct", max_seq_length = max_seq_length, load_in_4bit = True, fast_inference = True, max_lora_rank = lora_rank, gpu_memory_utilization = 0.6, ) model = FastLanguageModel.get_peft_model( model, r = lora_rank, lora_alpha = lora_rank, ) import re from datasets import load_dataset, Dataset # Load and prep dataset SYSTEM_PROMPT = """ Respond in the following format: <reasoning> ... </reasoning> <answer> ... </answer> """ XML_COT_FORMAT = """\ <reasoning> {reasoning} </reasoning> <answer> {answer} </answer> """ def extract_xml_answer(text: str) -> str: answer = text.split("<answer>")[-1] answer = answer.split("</answer>")[0] return answer.strip() def extract_hash_answer(text: str) -> str | None: if "####" not in text: return None return text.split("####")[1].strip() # uncomment middle messages for 1-shot prompting def get_gsm8k_questions(split = "train") -> Dataset: data = load_dataset('openai/gsm8k', 'main')[split] # type: ignore data = data.map(lambda x: { # type: ignore 'prompt': [ {'role': 'system', 'content': SYSTEM_PROMPT}, {'role': 'user', 'content': x['question']} ], 'answer': extract_hash_answer(x['answer']) }) # type: ignore return data # type: ignore dataset = get_gsm8k_questions() # Reward functions def correctness_reward_func(prompts, completions, answer, **kwargs) -> list[float]: responses = [completion[0]['content'] for completion in completions] q = prompts[0][-1]['content'] extracted_responses = [extract_xml_answer(r) for r in responses] print('-'*20, f"Question:\n{q}", f"\nAnswer:\n{answer[0]}", f"\nResponse:\n{responses[0]}", f"\nExtracted:\n{extracted_responses[0]}") return [2.0 if r == a else 0.0 for r, a in zip(extracted_responses, answer)] def int_reward_func(completions, **kwargs) -> list[float]: responses = [completion[0]['content'] for completion in completions] extracted_responses = [extract_xml_answer(r) for r in responses] return [0.5 if r.isdigit() else 0.0 for r in extracted_responses] def strict_format_reward_func(completions, **kwargs) -> list[float]: """Reward function that checks if the completion has a specific format.""" pattern = r"^<reasoning>\n.*?\n</reasoning>\n<answer>\n.*?\n</answer>\n$" responses = [completion[0]["content"] for completion in completions] matches = [re.match(pattern, r) for r in responses] return [0.5 if match else 0.0 for match in matches] def soft_format_reward_func(completions, **kwargs) -> list[float]: """Reward function that checks if the completion has a specific format.""" pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>" responses = [completion[0]["content"] for completion in completions] matches = [re.match(pattern, r) for r in responses] return [0.5 if match else 0.0 for match in matches] def count_xml(text) -> float: count = 0.0 if text.count("<reasoning>\n") == 1: count += 0.125 if text.count("\n</reasoning>\n") == 1: count += 0.125 if text.count("\n<answer>\n") == 1: count += 0.125 count -= len(text.split("\n</answer>\n")[-1])*0.001 if text.count("\n</answer>") == 1: count += 0.125 count -= (len(text.split("\n</answer>")[-1]) - 1)*0.001 return count def xmlcount_reward_func(completions, **kwargs) -> list[float]: contents = [completion[0]["content"] for completion in completions] return [count_xml(c) for c in contents] from trl import GRPOConfig, GRPOTrainer training_args = GRPOConfig( use_vllm = True, # use vLLM for fast inference! learning_rate = 5e-6, adam_beta1 = 0.9, adam_beta2 = 0.99, weight_decay = 0.1, warmup_ratio = 0.1, lr_scheduler_type = "cosine", optim = "paged_adamw_8bit", logging_steps = 1, bf16 = is_bfloat16_supported(), fp16 = not is_bfloat16_supported(), per_device_train_batch_size = 1, gradient_accumulation_steps = 1, num_generations = 6, max_prompt_length = 256, max_completion_length = 200, # num_train_epochs = 1, max_steps = 250, save_steps = 250, max_grad_norm = 0.1, report_to = "none", output_dir = "outputs", ) trainer = GRPOTrainer( model = model, processing_class = tokenizer, reward_funcs = [ xmlcount_reward_func, soft_format_reward_func, strict_format_reward_func, int_reward_func, correctness_reward_func, ], args = training_args, train_dataset = dataset, ) trainer.train() ``` ## Bug Fixes - Gemma 2 should be fixed now - Mistral base mapping should be fixed - Some syntax warning issue fixes - And many many more bug fixes! ## What's Changed * Add use_exact_model_name option to prevent automatic model name modification by @niryuu in https://github.com/unslothai/unsloth/pull/1339 * Improve debugging experience by @Erland366 in https://github.com/unslothai/unsloth/pull/1512 * changing model to base_model if peft model is already used by @mosama1994 in https://github.com/unslothai/unsloth/pull/1509 * All attention refactor fix by @KareemMusleh in https://github.com/unslothai/unsloth/pull/1491 * Update granite to work with latest post_patch methods by @Datta0 in https://github.com/unslothai/unsloth/pull/1502 * Minor fixes for granite models by @CoffeeVampir3 in https://github.com/unslothai/unsloth/pull/1503 * support modelscope models and datasets by @tastelikefeet in https://github.com/unslothai/unsloth/pull/1481 * Update README.md by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/1529 * Update bug_report.md by @danielhanchen in https://github.com/unslothai/unsloth/pull/1538 * Update README.md by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/1542 * Torch.Cuda Is Available Condition and Warning by @aminwhat in https://github.com/unslothai/unsloth/pull/1545 * Add dropout to granite to match HF's implementation by @Datta0 in https://github.com/unslothai/unsloth/pull/1557 * fix: flash_attn_detection_error by @Zzhiter in https://github.com/unslothai/unsloth/pull/1556 * Fix Mistral, Qwen by @danielhanchen in https://github.com/unslothai/unsloth/pull/1565 * Update README.md by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/1569 * Update README.md by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/1580 * Update README.md by @shimmyshimmer in https://github.com/unslothai/unsloth/pull/1595 * Mistral 24B, Qwen 2.5 VL support by @danielhanchen in https://github.com/unslothai/unsloth/pull/1598 * GRPO, vLLM, Bug Fixes, Reinforcement Learning by @danielhanchen in https://github.com/unslothai/unsloth/pull/1620 ## New Contributors * @niryuu made their first contribution in https://github.com/unslothai/unsloth/pull/1339 * @mosama1994 made their first contribution in https://github.com/unslothai/unsloth/pull/1509 * @KareemMusleh made their first contribution in https://github.com/unslothai/unsloth/pull/1491 * @tastelikefeet made their first contribution in https://github.com/unslothai/unsloth/pull/1481 * @aminwhat made their first contribution in https://github.com/unslothai/unsloth/pull/1545 * @Zzhiter made their first contribution in https://github.com/unslothai/unsloth/pull/1556 **Full Changelog**: https://github.com/unslothai/unsloth/compare/2025-01...2025-02