# DeepSpeed Further Training Issue
**URL:**
**Author:**  [@JackBAI](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJackBAI)%5C
**Post date:** [October 13, 2023, 7:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeepspeed-further-training-issue%2F58496%2F1 "2023-10-13T19:50:42Z")
@stas Excuse me, I’m using deepspeed for multi-node pre-traininig. My training script:
```
python -m torch.distributed.launch \
--nproc_per_node=8 \
--nnodes=4 \
--node_rank=... \
--master_addr=... \
--master_port=19500 \
--use_env \
transformers/examples/pytorch/language-modeling/run_mlm.py \
--resume_from_checkpoint $CKPT_TO_RESUME \
--config_name $MODEL \
--tokenizer_name $TOKENIZER \
--dataset_name $PT_DATASET \
--max_steps $MAX_STEPS \
...
--deepspeed config.json
```
After training I get these files in my ckpt dir:
config.json rng\_state\_10.pth rng\_state\_17.pth rng\_state\_23.pth rng\_state\_2.pth rng\_state\_7.pth trainer\_state.json
global\_step1500 rng\_state\_11.pth rng\_state\_18.pth rng\_state\_24.pth rng\_state\_30.pth rng\_state\_8.pth training\_args.bin
latest rng\_state\_12.pth rng\_state\_19.pth rng\_state\_25.pth rng\_state\_31.pth rng\_state\_9.pth vocab.json
merges.txt rng\_state\_13.pth rng\_state\_1.pth rng\_state\_26.pth rng\_state\_3.pth scheduler.pt zero\_to\_fp32.py
optimizer.pt rng\_state\_14.pth rng\_state\_20.pth rng\_state\_27.pth rng\_state\_4.pth special\_tokens\_map.json
pytorch\_model.bin rng\_state\_15.pth rng\_state\_21.pth rng\_state\_28.pth rng\_state\_5.pth tokenizer\_config.json
rng\_state\_0.pth rng\_state\_16.pth rng\_state\_22.pth rng\_state\_29.pth rng\_state\_6.pth tokenizer.json
Then I run `run_glue.sh` on this ckpt, but the scores are just equal to before pretraining. Is there anything wrong I’m doing? Many thanks for any help!
---
**Author:**  [@stas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstas)%5C
**Post date:** [November 11, 2023, 2:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeepspeed-further-training-issue%2F58496%2F2 "2023-11-11T02:56:22Z")
Apologies for not replying sooner, since I left HF my account is now stuck with a disabled emall at [hf.co](http://hf.co) that I can’t change so I get no notifications about someone tagging me.
It looks that perhaps it’s not loading the custom checkpoint and loads the default model instead?
Enabling a higher logging level like INFO is likely to log which model path is getting loaded.
But it’s also possible that your training, I assume finetuning didn’t improve the original baseline which is also a possibility.
If you’re still stuck probably the best to ask at the transformers Issues and provide more information like I suggested above - as what you shared here is insufficient to come to any conclusions?
---
**Author:**  [@JackBAI](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJackBAI)%5C
**Post date:** [November 25, 2023, 2:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeepspeed-further-training-issue%2F58496%2F3 "2023-11-25T14:38:32Z")
I tested it out and it seems like a problem with deepspeed config. After adjusting the config and solve the conflicts with the launching script config, it works. Thanks so much for your reply!