Krishna Chaitanya b53b10ea61 Fix Train LoRA crash when training_dtype is "none" with bfloat16 LoRA weights (#13145) ...

When training_dtype is set to "none" and the model's native dtype is
float16, GradScaler was unconditionally enabled. However, GradScaler
does not support bfloat16 gradients (only float16/float32), causing a
NotImplementedError when lora_dtype is "bf16" (the default).

Fix by only enabling GradScaler when LoRA parameters are not in
bfloat16, since bfloat16 has the same exponent range as float32 and
does not need gradient scaling to avoid underflow.

Fixes #13124

2026-03-24 23:53:44 -04:00

52 KiB

Raw Blame History

View Raw