ITADN

[Bug] Oumi wrapper fail to parse SLURM nodelist when submitting multiple nodes jobs

#2015Closedmonnetb 创建于 2025-11-17
bugtriage
M
monnetbcommented
### What happened? When launching distributed training on allocated nodes through SLURM the job invocation with is failing to parse the nodelist and extract the first as master `srun --ntasks=2 --ntasks-per-node=1 --nodes=2 oumi distributed torchrun --log-level DEBUG -m oumi train -c <my-yaml-file>` The distributed_run.py fail to parse the nodelist and report and shows the command line as: ### Steps to reproduce the bug Steps to reproduce: 1. Allocate 2 or mode nodes with Slurm -> `salloc -N2 -n16` 2. Launch the oumi distributed training job -> `srun --ntasks=2 --ntasks-per-node=1 --nodes=2 oumi distributed torchrun --log-level DEBUG -m oumi train -c <my-yaml-file>` 3. Check the command output in DEBUG mode. it shows the master-addr with wrong parsing. The output shows `Command: ['torchrun', '--nnodes=1', '--node-rank=0', '--nproc-per-node=8', '--master-addr=g100n[040-041]', '--master-port=8007', '-m', 'oumi', 'train', '-c', '/nfs/bruno/APPLICATIONS/LLM/oumi/hpe-scripts/train-llama3.1-8b-h200.yaml']` when the SLURM_NODE_LIST is `SLURM_NODELIST=g100n[040-041]`. ### System Info ```shell ____ _ _ __ __ _____ / __ \| | | | \/ |_ _| | | | | | | | \ / | | | | | | | | | | |\/| | | | | |__| | |__| | | | |_| |_ \____/ \____/|_| |_|_____| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Oumi environment information: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┌────────────────┬──────────────────────────────────────────────┐ │ Oumi version │ 0.4.3.dev19+gf986e0287 │ │ Python version │ 3.12.3 │ │ Platform │ Linux-6.8.0-84-generic-x86_64-with-glibc2.39 │ └────────────────┴──────────────────────────────────────────────┘ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Installed dependencies: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ PACKAGE ┃ VERSION ┃ ┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ accelerate │ 1.11.0 │ │ aiohttp │ 3.13.2 │ │ bitsandbytes │ <not installed> │ │ datasets │ 3.2.0 │ │ diffusers │ <not installed> │ │ einops │ <not installed> │ │ jsonlines │ 4.0.0 │ │ liger-kernel │ <not installed> │ │ llama-cpp-python │ <not installed> │ │ lm-eval │ 0.4.9.1 │ │ mlflow │ 3.6.0 │ │ numpy │ 2.3.5 │ │ nvidia-ml-py │ <not installed> │ │ omegaconf │ 2.4.0.dev3 │ │ open_clip_torch │ <not installed> │ │ pandas │ 2.3.3 │ │ peft │ 0.17.1 │ │ pexpect │ 4.9.0 │ │ pillow │ 11.3.0 │ │ pydantic │ 2.11.10 │ │ responses │ 0.25.8 │ │ sglang │ <not installed> │ │ skypilot │ 0.10.5 │ │ tensorboard │ 2.20.0 │ │ timm │ <not installed> │ │ torch │ 2.8.0 │ │ torchdata │ 0.9.0 │ │ torchvision │ 0.23.0 │ │ tqdm │ 4.67.1 │ │ transformers │ 4.57.1 │ │ trl │ 0.25.1 │ │ typer │ 0.20.0 │ │ vllm │ <not installed> │ │ wandb │ 0.21.4 │ └──────────────────┴─────────────────┘ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Environment variables: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ VARIABLE ┃ VALUE ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ ACCELERATE_DYNAMO_BACKEND │ <not set> │ │ ACCELERATE_DYNAMO_MODE │ <not set> │ │ ACCELERATE_DYNAMO_USE_DYNAMIC │ <not set> │ │ ACCELERATE_DYNAMO_USE_FULLGRAPH │ <not set> │ │ ACCELERATE_USE_FSDP │ <not set> │ │ CUDA_VISIBLE_DEVICES │ <not set> │ │ LOCAL_RANK │ <not set> │ │ LOCAL_WORLD_SIZE │ <not set> │ │ OUMI_EXTRA_DEPS_FILE │ <not set> │ │ OUMI_FORCE_EDITABLE_INSTALL │ <not set> │ │ OUMI_SLURM_CONNECTIONS │ <not set> │ │ OUMI_USE_SPOT_VM │ <not set> │ │ RANK │ <not set> │ │ WORLD_SIZE │ <not set> │ └─────────────────────────────────┴───────────┘ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ PyTorch information: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┌────────────────┬───────────────────────┐ │ CUDA available │ True │ │ CUDA version │ 12.8 │ │ cuDNN version │ 91.0.2 │ │ Number of GPUs │ 8 │ │ GPU type │ NVIDIA H100 80GB HBM3 │ │ GPU memory │ 79.2GB │ └────────────────┴───────────────────────┘ slurm is 25.11.0 ```
关闭于 2025-11-17 3 条评论