ITADN

external-script validation crashes before first checkpoint exists

#2680Closedrafstahelin 创建于 2026-04-14
R
rafstahelincommented
## Bug When using `validation_method: "external-script"` with a script template containing `{local_checkpoint_path}`, training crashes at: - Step 0 (base-model benchmark) if `disable_benchmark: false` - First validation interval (e.g. step 100) even with `disable_benchmark: true`, because no checkpoint has been saved yet ## Traceback ``` File "simpletuner/helpers/training/validation.py", line 1348, in _resolve_latest_checkpoint_path raise ValueError( ValueError: validation_external_script requires {local_checkpoint_path}, but no checkpoints exist in output_dir. ``` ## Expected behavior When `{local_checkpoint_path}` is in the script template and no checkpoint exists yet, validation should be silently skipped for that interval (log a warning, continue training). The first external validation would then run after the first checkpoint save. ## Workaround Set `disable_benchmark: true` and accept that validation is blind until the first checkpoint lands. Training itself continues fine — it's only the validation call that crashes. ## Config snippet ```json "validation_method": "external-script", "validation_external_script": "my_script.py --local {local_checkpoint_path} --global_step {global_step}", "validation_external_background": true ``` ## Environment - SimpleTuner installed from main (post-4.1.2) - Qwen Image 2512 - RTX PRO 6000 Blackwell
关闭于 2026-04-15 2 条评论