# Trainer using Checkpoint makes TPU crash
**URL:**
**Author:**  [@Kamel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKamel)%5C
**Post date:** [October 15, 2021, 9:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-using-checkpoint-makes-tpu-crash%2F10794%2F1 "2021-10-15T09:21:32Z")
Hi,
I am running a Bert from scratch on google cloud, and it is working.
But when I am doing the training using a saved Checkpoint with “model\_name\_or\_path” it makes the TPU crash with SIGKILL error (memory issue I guess).
I don’t understand this behavior, since if I rerun it without the checkpoint it works with no problems.
Running with checkpoint works only if I use just 1 core (num\_cores=1) which is not convenient for me (takes a much larger time).
Anyone have an idea to help me?
Thanks
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 15, 2021, 12:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-using-checkpoint-makes-tpu-crash%2F10794%2F2 "2021-10-15T12:15:47Z")
You might need more RAM to be able to resume from a checkpoint. The core of the issue is that the optimizer state is loaded on each TPU before being transferred to the XLA device (it can’t be directly loaded on the XLA device sadly) but since you have 8 processes loading it, it’s loaded 8 times on CPU.
---
**Author:**  [@Kamel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKamel)%5C
**Post date:** [October 15, 2021, 1:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-using-checkpoint-makes-tpu-crash%2F10794%2F3 "2021-10-15T13:16:47Z")
Thanks a lot @sgugger. I am not very used to these issues, so I am now on a 30GB memory vm instance, how can I know to how much memory should I upgrade it?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [October 15, 2021, 1:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-using-checkpoint-makes-tpu-crash%2F10794%2F4 "2021-10-15T13:31:28Z")
It depends on the model you are using. The size of the optimizer state is twice the size of the model if you’re using the default Adam, and this will need to be multiplied by 8.
---
**Author:**  [@Kamel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKamel)%5C
**Post date:** [October 15, 2021, 6:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-using-checkpoint-makes-tpu-crash%2F10794%2F5 "2021-10-15T18:33:42Z")
Ok I see. My optimizer.pt is 1.43GB so I need at least 12GB more ram.
Thanks again