# Corrupted deepspeed checkpoint **URL:** **Author:** ![abilek](https://avatars.discourse-cdn.com/v4/letter/a/a9a28c/32.png) [@abilek](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabilek)%5C **Post date:** [March 12, 2025, 5:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrupted-deepspeed-checkpoint%2F145438%2F1 "2025-03-12T17:46:10Z") Hello. I configured my training run to use deepspeed zero stage2 with the huggingface trainer with the following settings: ```auto DEEPSPEED_CONFIG = { "optimizer": { "type": "AdamW", "params": {"lr": "auto", "betas": "auto", "eps": "auto", "weight_decay": "auto"}, }, "scheduler": { "type": "WarmupLR", "params": {"warmup_min_lr": "auto", "warmup_max_lr": "auto", "warmup_num_steps": "auto"}, }, "zero_optimization": { "stage": 2, "offload_optimizer": {"device": "cpu", "pin_memory": True}, "allgather_partitions": True, "allgather_bucket_size": 2e8, "overlap_comm": False, "reduce_scatter": True, "reduce_bucket_size": 2e8, "contiguous_gradients": True, }, "gradient_accumulation_steps": "auto", "gradient_clipping": "auto", "train_batch_size": "auto", "train_micro_batch_size_per_gpu": "auto", } ``` During training I use the trainer to create checkpoints and Sagemaker checkpointing configuration in the Huggingface estimator to create checkpoints and upload to s3. I then create zero\_to\_fp32.py (from the deepspeed checkpoint) to create a pytorch\_model.bin file. When I try to load that pytorch\_model.bin to resume training, I get the following error: ```auto RuntimeError: linalg.vector_norm: Expected a floating point or complex tensor as input. Got Long ``` When I load the pytorch\_model.bin model to perform an evaluation, I also notice that loss values on the same Dev dataset as training are near infinite compared to the small loss values observed at the same step in training. The training behavior is as expected. Losses reported during training at different steps are L2 losses and normal for the problem. The underlying transformer model is a Falcom model with a custom configuration. Has anyone observed similar problems trying to resume training or run inference from deepspeed checkpoints? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [March 13, 2025, 6:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrupted-deepspeed-checkpoint%2F145438%2F2 "2025-03-13T06:46:32Z")
It seems like a troublesome problem… In some cases, it seems that you can avoid it by omitting the **torch\_dtype** specification when loading. > > > \*\*Describe the bug\*\* > I have been able to run my model successfully in Zero Stag…e 3 without any problems. However, when I attempt to run the same model in Zero Stage 2, I encounter an error: > 10.223.17.15: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.223.13.141: Traceback (most recent call last): > 10.78.107.139: self.\_take\_model\_step(lr\_kwargs) > 10.67.196.141: self.deepspeed\_engine\_wrapped.backward(loss, \*\*kwargs) > 10.78.121.13: > 10.223.17.15: main() > 10.223.13.141: File "train.py", line 706, in \ > 10.78.107.139: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2075, in \_take\_model\_step > 10.67.196.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/accelerate/utils/deepspeed.py", line 176, in backward > 10.78.121.13: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2075, in \_take\_model\_step > 10.223.17.15: File "train.py", line 587, in main > 10.67.196.141: self.\_take\_model\_step(lr\_kwargs) > 10.78.107.139: self.deepspeed\_engine\_wrapped.backward(loss, \*\*kwargs) > 10.223.13.141: scaled\_global\_grad\_norm = self.scaled\_global\_norm() > 10.78.121.13: self.\_take\_model\_step(lr\_kwargs) > 10.223.17.15: return torch.norm(torch.stack(norm\_groups), p=norm\_type) > 10.67.196.141: self.optimizer.step() > 10.78.107.139: File "/opt/conda/envs/progen/lib/python3.8/site-packages/accelerate/utils/deepspeed.py", line 176, in backward > 10.223.13.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1789, in scaled\_global\_norm > 10.223.13.141: self.optimizer.step() > 10.78.121.13: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2075, in \_take\_model\_step > 10.223.17.15: File "/opt/conda/envs/progen/lib/python3.8/site-packages/torch/functional.py", line 1626, in norm > 10.78.107.139: self.optimizer.step()self.optimizer.step() > 10.67.196.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2075, in \_take\_model\_step > 10.223.13.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.223.17.15: self.\_take\_model\_step(lr\_kwargs) > 10.78.107.139: > 10.78.107.139: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.67.196.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.78.121.13: self.optimizer.step()Traceback (most recent call last): > 10.223.13.141: main() > 10.223.17.15: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2075, in \_take\_model\_step > 10.78.107.139: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.67.196.141: main() > 10.223.17.15: accelerator.backward(loss) > 10.78.107.139: self.engine.step() > 10.223.13.141: File "train.py", line 587, in main > 10.78.121.13: > 10.67.196.141: File "train.py", line 587, in main > 10.223.17.15: File "/opt/conda/envs/progen/lib/python3.8/site-packages/accelerate/accelerator.py", line 1960, in backward > 10.223.17.15: main() > 10.78.107.139: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2169, in step > 10.223.13.141: self.optimizer.step()self.\_take\_model\_step(lr\_kwargs) > 10.78.121.13: self.optimizer.step() File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.78.121.13: > 10.67.196.141: self.engine.step() > 10.223.17.15: File "train.py", line 587, in main > 10.78.107.139: self.\_take\_model\_step(lr\_kwargs) > 10.223.13.141: > 10.78.121.13: File "train.py", line 706, in \ > 10.67.196.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2169, in step > 10.223.17.15: scaled\_global\_grad\_norm = self.scaled\_global\_norm() > 10.78.107.139: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2075, in \_take\_model\_step > 10.223.13.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2075, in \_take\_model\_step > 10.78.121.13: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.223.13.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.78.107.139: main() > 10.67.196.141: scaled\_global\_grad\_norm = self.scaled\_global\_norm() > 10.223.17.15: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1789, in scaled\_global\_norm > 10.78.121.13: main() > 10.223.13.141: self.\_take\_model\_step(lr\_kwargs) > 10.78.107.139: File "train.py", line 587, in main > 10.67.196.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1789, in scaled\_global\_norm > 10.223.17.15: return torch.linalg.vector\_norm(input, \_p, \_dim, keepdim, dtype=dtype) > 10.78.121.13: self.optimizer.step() > 10.223.13.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2075, in \_take\_model\_step > 10.78.107.139: Traceback (most recent call last): > 10.223.13.141: accelerator.backward(loss) > 10.223.17.15: RuntimeError: linalg.vector\_norm: Expected a floating point or complex tensor as input. Got Long > 10.78.121.13: File "train.py", line 587, in main > 10.223.17.15: self.optimizer.step() > 10.67.196.141: scaled\_global\_grad\_norm = self.scaled\_global\_norm() > 10.78.107.139: scaled\_global\_grad\_norm = self.scaled\_global\_norm() > 10.223.13.141: self.\_take\_model\_step(lr\_kwargs) > 10.78.121.13: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.78.121.13: self.optimizer.step() > 10.78.107.139: File "train.py", line 706, in \ > 10.223.17.15: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.223.13.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/accelerate/accelerator.py", line 1960, in backward > 10.67.196.141: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1789, in scaled\_global\_norm > 10.78.121.13: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1842, in step > 10.78.107.139: File "/opt/conda/envs/progen/lib/python3.8/site-packages/deepspeed/runtime/zero/stage\_1\_and\_2.py", line 1789, in scaled\_global\_norm > > \*\*To Reproduce\*\* > I am using a cluster of 6 machines, each equipped with 40GB A100 GPUs. > > launch cmd: > accelerate launch --config\_file config/yaml/zero.yaml train.py --with\_tracking --report\_to tensorboard --output\_dir tblog --project\_name test --checkpointing\_steps epoch --per\_device\_train\_batch\_size 8 --num\_train\_epochs 50 --learning\_rate 5e-5 --seed 42 --precision bf16 > > Here is my yaml: > compute\_environment: LOCAL\_MACHINE > deepspeed\_config: > deepspeed\_config\_file: ./config/zero/zero\_stage2\_bf16.json > deepspeed\_multinode\_launcher: pdsh > deepspeed\_hostfile: ./config/yaml/hostfile > zero3\_init\_flag: true > distributed\_type: DEEPSPEED > fsdp\_config: {} > machine\_rank: 0 > main\_process\_ip: 10.223.17.15 > main\_process\_port: 36769 > main\_training\_function: main > num\_machines: 6 > num\_processes: 48 > use\_cpu: false > > zero\_stage2\_bf16.json: > { > "bf16": { > "enabled": true > }, > "optimizer": { > "type": "AdamW", > "params": { > "lr": "auto", > "weight\_decay": "auto", > "torch\_adam": true, > "adam\_w\_mode": true > } > }, > "scheduler": { > "type": "WarmupDecayLR", > "params": { > "warmup\_min\_lr": "auto", > "warmup\_max\_lr": "auto", > "warmup\_num\_steps": "auto", > "total\_num\_steps": 533770 > } > }, > "zero\_optimization": { > "stage": 2, > "offload\_optimizer": { > "device": "cpu", > "pin\_memory": true > }, > "offload\_param": { > "device": "cpu", > "pin\_memory": true > }, > "allgather\_partitions": true, > "allgather\_bucket\_size": 2e8, > "overlap\_comm": true, > "reduce\_scatter": true, > "reduce\_bucket\_size": "auto", > "contiguous\_gradients": true > }, > "gradient\_accumulation\_steps": 1, > "gradient\_clipping": "auto", > "steps\_per\_print": 2000, > "train\_batch\_size": "auto", > "train\_micro\_batch\_size\_per\_gpu": "auto", > "wall\_clock\_breakdown": false > } > > > \### 🐛 Describe the bug > > Setting an \`int\` tensor to \[linalg.norm()\](https://pytor…ch.org/docs/stable/generated/torch.linalg.norm.html) gets the error message as shown below: > > \`\`\`python > import torch > from torch import linalg > > my\_tensor = torch.tensor(\[8, -3, 0, 1\]) > > linalg.norm(input=my\_tensor) # Error > \`\`\` > > \> RuntimeError: linalg.vector\_norm: Expected a floating point or complex tensor as input. Got Long > > But, setting a \`complex\` tensor to \`linalg.norm()\` returns a \`float\` tensor as shown below: > > \`\`\`python > import torch > from torch import linalg > > my\_tensor = torch.tensor(\[8.+0.j, -3.+0.j, 0.+0.j, 1.+0.j\]) > > linalg.norm(input=my\_tensor) > \# tensor(8.6023) > > linalg.norm(input=my\_tensor).dtype > \# torch.float32 > \`\`\` > > So, I set \`dtype=torch.complex64\` to \`linalg.norm()\` but it still returns a \`float\` tensor as shown below: > > \`\`\`python > import torch > from torch import linalg > > my\_tensor = torch.tensor(\[8.+0.j, -3.+0.j, 0.+0.j, 1.+0.j\]) > # ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ > linalg.norm(input=my\_tensor, dtype=torch.complex64) > \# tensor(8.6023) > > linalg.norm(input=my\_tensor, dtype=torch.complex64).dtype > \# torch.float32 > \`\`\` > > \### Versions > > \`\`\`python > import torch > > torch.\_\_version\_\_ # '2.3.0' > \`\`\` > > cc @jianyuh @nikitaved @pearu @mruberry @walterddr @xwang233 @Lezcano