scaled fp8 mm breaks ideogram on 5090 system

Not reproducible on L40S, but: ``` [RANK 0] 2026-06-10 15:48:44,160 [INFO] Precomputing the negative prompt embed for validations: blurry, cropped, ugly [RANK 0] 2026-06-10 15:48:44,195 [INFO] Completed validation prompt gathering. [RANK 0] 2026-06-10 15:48:44,709 [INFO] Text encoder 1: memory before unload: 8.38 GB [RANK 0] 2026-06-10 15:48:44,715 [INFO] Text encoder 1: 265 params and 855 buffers on CUDA before move [RANK 0] 2026-06-10 15:48:44,725 [INFO] Text encoder 1 successfully moved to meta device [RANK 0] 2026-06-10 15:48:44,725 [INFO] Text encoder 1: memory after unload: 0.20 GB (freed 8.18 GB) (…)ion_pytorch_model.safetensors.index.json: 50.4kB [00:00, 92.2MB/s] transformer/diffusion_pytorch_model.safe(…): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.29G/9.29G [01:32<00:00, 100MB/s] [RANK 0] 2026-06-10 15:50:20,393 [INFO] Unloading all model components... [RANK 0] 2026-06-10 15:50:20,755 [INFO] Model components unloaded successfully. Traceback (most recent call last): File "/nvme/home/mikaelh/Stable_Diffusion/bghira/SimpleTunerv4/simpletuner/train.py", line 203, in <module> trainer.init_load_base_model() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/nvme/home/mikaelh/Stable_Diffusion/bghira/SimpleTunerv4/simpletuner/helpers/training/trainer.py", line 2249, in init_load_base_model self.model.load_model(move_to_device=False) ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ File "/nvme/home/mikaelh/Stable_Diffusion/bghira/SimpleTunerv4/simpletuner/helpers/models/ideogram/model.py", line 77, in load_model self.model = _build_transformer( ~~~~~~~~~~~~~~~~~~^ Ideogram4Config(), ^^^^^^^^^^^^^^^^^^ ...<2 lines>... self.config.weight_dtype, ^^^^^^^^^^^^^^^^^^^^^^^^^ ) ```