Special thanks to @dxqb from OneTrainer with the idea of reducing the text encoder cache size for ideogram 4. ## What's Changed * sanitise logger output of secrets for accelerate launch cmd by @bghira in https://github.com/bghira/SimpleTuner/pull/2752 * disable colours when logging via coglet by @bghira in https://github.com/bghira/SimpleTuner/pull/2753 * zlab i1 support: TREAD, TwinFlow, LayerSync, CFG modifications and more by @bghira in https://github.com/bghira/SimpleTuner/pull/2754 * ideogram4 text embed size reduction with pre-normalised cache by @bghira in https://github.com/bghira/SimpleTuner/pull/2755 * merge by @bghira in https://github.com/bghira/SimpleTuner/pull/2756 **Full Changelog**: https://github.com/bghira/SimpleTuner/compare/v4.3.3...v4.3.4

## What's Changed * (#2678) update skrample and pin to 0.6.0 by @bghira in https://github.com/bghira/SimpleTuner/pull/2679 **Full Changelog**: https://github.com/bghira/SimpleTuner/compare/v4.2.0...v4.2.1

## Breaking change The minor version is bumped to **4.1.0** for a breaking change. - LTX-2 audio configuration is now required to be set manually in order to have audio dataset auto-creation enabled. - The LTXVIDEO2 quickstart is updated with this information. - Qwen Image no longer uses attention mask when the sequence is full (small speed-up for very very long captions) ## Features - CREPA can now be enabled and used with image model training, even on video models - U-REPA is implemented with documentation for SDXL, SD1x, and Kolors - T-LoRA via LyCORIS implementation (note: missing ortho init, to be included in follow-up) - Wan now uses attention dispatcher, can use flash-attn, cudnn, other options ## Bugfixes - Musubi block swap validation speed fixed for Flux2 - LoKr init norm works with torchao quant - Huggingface model card epoch + step count alignment - Flux2 + Ramtorch validation error resolved ## What's Changed * U-REPA: SDXL, SD1x, Kolors by @bghira in https://github.com/bghira/SimpleTuner/pull/2563 * update docs for installing cuda13 variant of torch by @bghira in https://github.com/bghira/SimpleTuner/pull/2596 * wan: switch to attn dispatcher so backends can be changed; improve context-parallel performance by @bghira in https://github.com/bghira/SimpleTuner/pull/2599 * flux2: fix ramtorch validation by checking device location correctly by @bghira in https://github.com/bghira/SimpleTuner/pull/2611 * fix flux2 block swap validation performance by @bghira in https://github.com/bghira/SimpleTuner/pull/2614 * (#2574) add eval dataset type for lookup in vaecache by @bghira in https://github.com/bghira/SimpleTuner/pull/2590 * expand CREPA coverage to image models by @bghira in https://github.com/bghira/SimpleTuner/pull/2562 * qwen image and qwen edit do not need attn masking when no padding by @bghira in https://github.com/bghira/SimpleTuner/pull/2598 * LyCORIS T-LoRA by @bghira in https://github.com/bghira/SimpleTuner/pull/2609 * (#2602) hooks for two-stage model pipeline capability by @bghira in https://github.com/bghira/SimpleTuner/pull/2608 * test coverage for multi-stage hook by @bghira in https://github.com/bghira/SimpleTuner/pull/2616 * add documentation for using CREPA for images by @bghira in https://github.com/bghira/SimpleTuner/pull/2617 * (#2573) step number consistency in async upload by @bghira in https://github.com/bghira/SimpleTuner/pull/2618 * (#2572) add another just in case setting for batch size by @bghira in https://github.com/bghira/SimpleTuner/pull/2619 * (#2612) fix --init_lokr_norm with torchao quant by @bghira in https://github.com/bghira/SimpleTuner/pull/2620 * s2v/ltx-2 audio auto split should be enabled more intelligently by @bghira in https://github.com/bghira/SimpleTuner/pull/2621 **Full Changelog**: https://github.com/bghira/SimpleTuner/compare/v4.0.6...v4.1.0

## What's Changed * (#2484) fix use of spread operator on ES6 object with getters by @bghira in https://github.com/bghira/SimpleTuner/pull/2486 * environment creation wizard size constraints for smaller (1920x1080) viewports under 4k by @bghira in https://github.com/bghira/SimpleTuner/pull/2487 * (#2479) add TEXT_JSON field type for complex data types in a simple text field input by @bghira in https://github.com/bghira/SimpleTuner/pull/2488 * use TEXT_JSON field type for TREAD by @bghira in https://github.com/bghira/SimpleTuner/pull/2489 * (#2480) adjust num_frames automatically by limit instead of throwing error by @bghira in https://github.com/bghira/SimpleTuner/pull/2490 * (#2475) bypass batch size for eval dataset by @bghira in https://github.com/bghira/SimpleTuner/pull/2491 * add max_num_samples per-dataset by @bghira in https://github.com/bghira/SimpleTuner/pull/2492 * (#2477) GPU circuit breaker by @bghira in https://github.com/bghira/SimpleTuner/pull/2493 * (#2474) surface processing statistics in webui; store count of too_small etc image count in dataset metadata files by @bghira in https://github.com/bghira/SimpleTuner/pull/2494 * (#2483) validation epoch tracking should simulate dataset scheduling by @bghira in https://github.com/bghira/SimpleTuner/pull/2495 * (#2274) add end_step / end_epoch scheduling for datasets by @bghira in https://github.com/bghira/SimpleTuner/pull/2496 * (#2470) multi-aspect input conditioning for kontext, flux2 and qwen edit by @bghira in https://github.com/bghira/SimpleTuner/pull/2497 * (#1812) i2v validation using image datasets and documentation updates by @bghira in https://github.com/bghira/SimpleTuner/pull/2499 * ss_tag_frequency should contain only terms in more than 50% of all captions by @bghira in https://github.com/bghira/SimpleTuner/pull/2500 * mkDocs: move to Zensical instead, and fix the theme by @bghira in https://github.com/bghira/SimpleTuner/pull/2501 * GPU circuit-breaker should treat thermal events as warning only, and display GPU thermal throttling in UI by @bghira in https://github.com/bghira/SimpleTuner/pull/2502 * avoid reusing stale job pid by canceling *local* running jobs at startup by @bghira in https://github.com/bghira/SimpleTuner/pull/2503 * LTX-2: IC-LoRA training with reference videos by @bghira in https://github.com/bghira/SimpleTuner/pull/2498 * z image (base) by @bghira in https://github.com/bghira/SimpleTuner/pull/2505 * (#2509) end-to-end JSON field handling fix for CLI launched training job by @bghira in https://github.com/bghira/SimpleTuner/pull/2511 * (#2507) eval dataset should have effective_batch_size of 1 by @bghira in https://github.com/bghira/SimpleTuner/pull/2512 * (#2508) calculate and sum all epoch stats as we receive them instead of incorrectly only counting the prev by @bghira in https://github.com/bghira/SimpleTuner/pull/2513 * face detection fixes for TrainingSample with PIL fallback by @bghira in https://github.com/bghira/SimpleTuner/pull/2515 * webui/webhooks: error reporting refactor by @bghira in https://github.com/bghira/SimpleTuner/pull/2516 * UI event system should rely on SSE manager by @bghira in https://github.com/bghira/SimpleTuner/pull/2517 * merge by @bghira in https://github.com/bghira/SimpleTuner/pull/2518 **Full Changelog**: https://github.com/bghira/SimpleTuner/compare/v4.0.2...v4.0.3

This release introduces flux2 klein 4b and 9b, a `disable_multiline_split` option for disabling multi-caption split on newlines; new options for customizing text encoder layers in FLUX.2 models, enhancements for model metadata, expanded validation strategies using datasets, and detailed CREPA regularization scheduling controls. Data Loader Options: - Added disable_multiline_split option to dataloader documentation in English (DATALOADER.md), Spanish (DATALOADER.es.md), Portuguese (DATALOADER.pt-BR.md), Hindi (DATALOADER.hi.md), Japanese (DATALOADER.ja.md), and Chinese (DATALOADER.zh.md). This option prevents splitting captions by newlines, useful for preserving intentional line breaks. Updated example configs to include this option. [[1]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-832fa306c74a75dd82d3f3ea4991ef8ae01d9e5fe676fa48064ad900650fbedbR215-R220) [[2]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-832fa306c74a75dd82d3f3ea4991ef8ae01d9e5fe676fa48064ad900650fbedbR705) [[3]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-53020b5b852ec497768f106a35a23f9643d17154aede1a376f3d0ea2823a1c77R181-R186) [[4]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-53020b5b852ec497768f106a35a23f9643d17154aede1a376f3d0ea2823a1c77R671) [[5]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-91821919af7cefc854d5b74caff07f8a098fbb95d543840e88be1699d6ad86f4R181-R186) [[6]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-91821919af7cefc854d5b74caff07f8a098fbb95d543840e88be1699d6ad86f4R671) [[7]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-40cc304678d5de3b894191892493b735faf57669a9639eb6f263b6743cbb2017R181-R186) [[8]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-40cc304678d5de3b894191892493b735faf57669a9639eb6f263b6743cbb2017R671) [[9]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-1ed4a7dac98fba8d8bccea736baf8b209b326025ae3755421b7ffaa7badf1bc1R181-R186) [[10]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-1ed4a7dac98fba8d8bccea736baf8b209b326025ae3755421b7ffaa7badf1bc1R672) [[11]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-0f61518874320948c7f36ca9c2070b7d11b5cd55df24a4ed6586a864b8fb96cbR181-R186) [[12]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-0f61518874320948c7f36ca9c2070b7d11b5cd55df24a4ed6586a864b8fb96cbR672) Model Training Options: - Added --custom_text_encoder_intermediary_layers option to Spanish (OPTIONS.es.md) and Hindi (OPTIONS.hi.md) documentation, allowing users to override which hidden state layers are extracted from the text encoder for FLUX.2 models. Includes format, defaults, usage notes, and warnings about cache invalidation. [[1]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-3f0cd39e3a7f176d81408ae40dc4f8c6271347f75b1cee5cf71c4e701d7db561R174-R183) [[2]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-39b5789a705b2e70400106934436b619d10413ca5e37bd7a2b5fa7aa0c0fb8cfR174-R183) - Added --modelspec_comment option to Spanish (OPTIONS.es.md) documentation, enabling embedding custom comments into model metadata, visible in external viewers. Supports environment variable substitution and multiple lines. Updated CLI usage and options reference. [[1]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-3f0cd39e3a7f176d81408ae40dc4f8c6271347f75b1cee5cf71c4e701d7db561R406-R429) [[2]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-3f0cd39e3a7f176d81408ae40dc4f8c6271347f75b1cee5cf71c4e701d7db561R1439) [[3]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-3f0cd39e3a7f176d81408ae40dc4f8c6271347f75b1cee5cf71c4e701d7db561R2110-R2112) Validation and Conditioning: - Documented new validation strategies in Spanish (OPTIONS.es.md): --validation_using_datasets for img2img validation using training dataset images, and --eval_dataset_id for selecting a specific dataset for evaluation. Includes detailed explanations of conditioning modes, dataset types, and how these options interact. CREPA Regularization Scheduling: - Expanded documentation for CREPA regularization in Spanish (OPTIONS.es.md) with new options: --crepa_scheduler, --crepa_warmup_steps, --crepa_decay_steps, --crepa_lambda_end, --crepa_power, --crepa_cutoff_step, --crepa_similarity_threshold, --crepa_similarity_ema_decay, and --crepa_threshold_mode. Includes configuration examples and usage notes for advanced scheduling and stopping criteria. [[1]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-3f0cd39e3a7f176d81408ae40dc4f8c6271347f75b1cee5cf71c4e701d7db561R1040-R1094) [[2]](https://github.com/bghira/SimpleTuner/pull/2434/files#diff-3f0cd39e3a7f176d81408ae40dc4f8c6271347f75b1cee5cf71c4e701d7db561R1112-R1120)

## Features - [SDNQ](https://github.com/Disty0/SDNQ) quantisation engine for **weights** and **optimisers** - Musubi block swap expanded to cover **auraflow, chroma, longcat-image, lumina2, omnigen, hidream, sana, sd3, and z-image** - Kandinsky5 memory-efficient VAE now used instead of Diffusers' HunyuanVideo implementation (runs on consumer hw) - `resolution_frames` bucket strategy for video training so that multi-length dataset is possible with just a single config entry - WebUI: Training configuration wizard now allows filling in the number of checkpoints to keep - metadata will be written to the model / LoRA checkpoint for ComfyUI LoRA Auto Trigger Words node to make use of - OmniGen & Lumina2: TREAD, TwinFlow, and LayerSync - Qwen Image: experimental tiled attention support that avoids OOM in attention calc (disabled, have to enter the code to enable it for now) ## Bugfixes - RamTorch - Now applies to text encoders properly (incl CLIP) - Extended to support Conv2D and Embedding layers (eg. SDXL offload) - Compatibility with Quanto (tested with int2, int4, int8-quanto) - System memory use reduction by not calculating gradients when `requires_grad=False` - Text encoder memory not unloading fixed for Qwen Image - No more quantize_via pipeline error when no quantisation is enabled - Qwen Image batch size > 1 training fixed (padded) - ROCm: bypass PyTorch bug for building kernels, enabling full Quanto compatibility (int2, int4, int8, fp8) ## What's Changed * add metadata for ComfyUI-Lora-Auto-Trigger-Words node by @bghira in https://github.com/bghira/SimpleTuner/pull/2222 * auraflow: implement musubi block swap by @bghira in https://github.com/bghira/SimpleTuner/pull/2227 * chroma: implement musubi block swap by @bghira in https://github.com/bghira/SimpleTuner/pull/2228 * longcat image: implement musubi block swap by @bghira in https://github.com/bghira/SimpleTuner/pull/2230 * modernise lumina2 implementation with TREAD, block swapping, twinflow and layersync by @bghira in https://github.com/bghira/SimpleTuner/pull/2231 * modernise omnigen implementation with TREAD, block swapping, twinflow and layersync by @bghira in https://github.com/bghira/SimpleTuner/pull/2232 * pixart: implement musubi block swap by @bghira in https://github.com/bghira/SimpleTuner/pull/2233 * add qwen-edit-2511 support, and an edit-v2+ flavour which enables 2511 features on 2509 by @bghira in https://github.com/bghira/SimpleTuner/pull/2223 * hidream: implement musubi block swap by @bghira in https://github.com/bghira/SimpleTuner/pull/2234 * sana & sanavideo: implement musubi block swap by @bghira in https://github.com/bghira/SimpleTuner/pull/2235 * sd3: implement musubi block swap by @bghira in https://github.com/bghira/SimpleTuner/pull/2236 * z-image turbo & omni: implement musubi block swap by @bghira in https://github.com/bghira/SimpleTuner/pull/2237 * use kandinsky5 optimised VAE with added temporal roll and chunked conv3d by @bghira in https://github.com/bghira/SimpleTuner/pull/2229 * when preparing model with offload enabled, do not move to accelerator by @bghira in https://github.com/bghira/SimpleTuner/pull/2238 * docs: document SIMPLETUNER_JOB_ID env var for webhook job_id by @rafstahelin in https://github.com/bghira/SimpleTuner/pull/2239 * sdnq quant engine by @bghira in https://github.com/bghira/SimpleTuner/pull/2225 * fix error str vs int comparison by @bghira in https://github.com/bghira/SimpleTuner/pull/2241 * fix error when quantize_via=pipeline but no_change level was provided by @bghira in https://github.com/bghira/SimpleTuner/pull/2242 * ramtorch: when using it for text encoders, do not move to gpu by @bghira in https://github.com/bghira/SimpleTuner/pull/2244 * add resolution_frames bucket strategy for video datasets so that different lengths can exist in one dataset by @bghira in https://github.com/bghira/SimpleTuner/pull/2240 * add checkpoints total limit to wizard by @bghira in https://github.com/bghira/SimpleTuner/pull/2243 * qwen image: fix padding for text embeds by @bghira in https://github.com/bghira/SimpleTuner/pull/2246 * quanto: fix ROCm compiler error for int2-quanto; fix for RamTorch compatibility by @bghira in https://github.com/bghira/SimpleTuner/pull/2248 * qwen image: tiled attention fallback when we hit OOM by @bghira in https://github.com/bghira/SimpleTuner/pull/2249 * ramtorch: fix for gradient memory ballooning; fix text encoder application; extend for Conv2D and Embedding offload by @bghira in https://github.com/bghira/SimpleTuner/pull/2250 * merge by @bghira in https://github.com/bghira/SimpleTuner/pull/2251 ## New Contributors * @rafstahelin made their first contribution in https://github.com/bghira/SimpleTuner/pull/2239 **Full Changelog**: https://github.com/bghira/SimpleTuner/compare/v3.3.2...v3.3.3

## Features - Better diffusion loss tracking when using LayerSync + CREPA - WebUI easy memory optimisation config for light/medium/aggressive configs - TUI `simpletuner configure` also able to apply optimisation presets to existing configs <img width="1048" height="940" alt="image" src="https://github.com/user-attachments/assets/805bc145-3bb6-4dff-9a08-99100ec2ba6a" /> <img width="835" height="282" alt="image" src="https://github.com/user-attachments/assets/ca334069-e56b-42d7-92e7-960d450d4f88" /> ## Bugfixes - ComfyUI will now automatically enable v-prediction and ztsnr for relevant checkpoints - LongCat batched training now works correctly - LongCat edit fixed - ControlNet demo dataset repeats boosted - Chroma indent issue fixed, now trains again - Example configs fixed, populate in UI correctly - Example configs no longer use constant LR scheduler with warmup steps incorrectly - SDXL hidden state buffer arg removed - TinyGemm device mismatch - Examples no longer suggest `validation_torch_compile` or lion optimiser for video models (degrades) ## What's Changed * add pure diffusion loss term pre-augmentation when aux loss is enabled by @bghira in https://github.com/bghira/SimpleTuner/pull/2201 * switch video training example configs from Lion to AdamW BF16 by @bghira in https://github.com/bghira/SimpleTuner/pull/2206 * remove validation torch compile option from examples by @bghira in https://github.com/bghira/SimpleTuner/pull/2207 * (#2175) move scale_shift to _data device by @bghira in https://github.com/bghira/SimpleTuner/pull/2202 * when example uses lr warmup, use constant_with_warmup by @bghira in https://github.com/bghira/SimpleTuner/pull/2208 * Fixup crepa states extraction for K5 by @kabachuha in https://github.com/bghira/SimpleTuner/pull/2209 * fix: remove unsupported hidden_states_buffer from SDXL model_predict by @joeqzzuo in https://github.com/bghira/SimpleTuner/pull/2213 * fix config syntax by @bghira in https://github.com/bghira/SimpleTuner/pull/2214 * (#2211) fix Chroma indent issue and resolve validation and training noise by @bghira in https://github.com/bghira/SimpleTuner/pull/2215 * use repeats of 4 by default on demo CN datasets by @bghira in https://github.com/bghira/SimpleTuner/pull/2218 * add lycoris example for longcat edit by @bghira in https://github.com/bghira/SimpleTuner/pull/2217 * longcat image: fix text encoder padding on inputs and initialisation of text processor by @bghira in https://github.com/bghira/SimpleTuner/pull/2216 * (#1822) add --delete_model_after_load to remove files from disk after they're loaded into memory by @bghira in https://github.com/bghira/SimpleTuner/pull/2210 * comfyui: ztsnr and vpred compatibility by @bghira in https://github.com/bghira/SimpleTuner/pull/2220 * easy memory optimisation presets by @bghira in https://github.com/bghira/SimpleTuner/pull/2221 * merge by @bghira in https://github.com/bghira/SimpleTuner/pull/2219 ## New Contributors * @kabachuha made their first contribution in https://github.com/bghira/SimpleTuner/pull/2209 * @joeqzzuo made their first contribution in https://github.com/bghira/SimpleTuner/pull/2213 **Full Changelog**: https://github.com/bghira/SimpleTuner/compare/v3.3.1...v3.3.2

## What's Changed * flux2: do not bypass the special model loader by @bghira in https://github.com/bghira/SimpleTuner/pull/2170 * (#2030) scheduled dataset sampling by @bghira in https://github.com/bghira/SimpleTuner/pull/2167 * GLANCE: better code example by @bghira in https://github.com/bghira/SimpleTuner/pull/2171 * TwinFlow: do not initialise neg time embed when disabled by @bghira in https://github.com/bghira/SimpleTuner/pull/2174 * UI (datasets): remove ControlNet conditioning option from selections when CN is disabled; select reference_strict by default otherwise by @bghira in https://github.com/bghira/SimpleTuner/pull/2177 * add missing LayerSync support to kandinsky5 video by @bghira in https://github.com/bghira/SimpleTuner/pull/2179 * qwen-edit: fix text embed cache generation with image context; disable image embeddings for multi-conditioning input by @bghira in https://github.com/bghira/SimpleTuner/pull/2176 * chroma 4d text embed fix by @bghira in https://github.com/bghira/SimpleTuner/pull/2181 * ensure edit-v2 either uses 1:1 or 0 image embeds by @bghira in https://github.com/bghira/SimpleTuner/pull/2186 * upload zip: preserve subdirs by @bghira in https://github.com/bghira/SimpleTuner/pull/2189 * allow `simpletuner server env=...` to auto-start training after webUI launches by @bghira in https://github.com/bghira/SimpleTuner/pull/2191 * add more indicators to dataset page when conditioning parameters are not set by @bghira in https://github.com/bghira/SimpleTuner/pull/2192 * Git-based configuration sync across SimpleTuner nodes (wip) by @bghira in https://github.com/bghira/SimpleTuner/pull/2172 * Z-Image-Omni with optional SigLIP conditioning support, TREAD, LayerSync, CFG layer skip, fp16 clamping, and TwinFlow by @bghira in https://github.com/bghira/SimpleTuner/pull/2183 * (#2182) add --peft_lora_target_modules for arbitrary layer definition by @bghira in https://github.com/bghira/SimpleTuner/pull/2193 * (#2190) add webUI onboarding config to "simpletuner configure" by @bghira in https://github.com/bghira/SimpleTuner/pull/2194 * merge by @bghira in https://github.com/bghira/SimpleTuner/pull/2196 * (#2173) remove early check for CREPA since we are using LayerSync features with certain configs by @bghira in https://github.com/bghira/SimpleTuner/pull/2195 * (#2187) better image resizing for validation inputs when validation resolution != training resolution by @bghira in https://github.com/bghira/SimpleTuner/pull/2197 * adjust default resolution on dataset page to equal --resolution, and ensure min/max/target down sample size are equal by @bghira in https://github.com/bghira/SimpleTuner/pull/2198 * merge by @bghira in https://github.com/bghira/SimpleTuner/pull/2199 **Full Changelog**: https://github.com/bghira/SimpleTuner/compare/v3.3.0...v3.3.1