# Trainers.train() with accelerate
**URL:**
**Author:**  [@zsaladin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzsaladin)%5C
**Post date:** [March 7, 2023, 3:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainers-train-with-accelerate%2F33209%2F1 "2023-03-07T03:05:55Z")
[Official document](https://huggingface.co/docs/transformers/accelerate) describes how to train with accelerate. But using accelerate with [Trainer.train()](https://huggingface.co/docs/transformers/v4.26.1/en/main_classes/trainer#transformers.Trainer.train) aren’t explained.
Is is possible to do it?
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [June 28, 2023, 10:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainers-train-with-accelerate%2F33209%2F2 "2023-06-28T22:19:28Z")
try this:
```auto
from transformers import GPT2LMHeadModel, GPT2TokenizerFast, AdamW
from accelerate import Accelerator
from datasets import load_dataset
import torch
# Initialize accelerator
accelerator = Accelerator()
# Load a dataset
dataset = load_dataset('text', data_files={'train': 'train.txt', 'test': 'test.txt'})
# Tokenization
tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
def tokenize_function(examples):
# We are doing causal (unidirectional) masking
return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=512)
dataset = dataset.map(tokenize_function, batched=True)
dataset.set_format("torch", columns=["input_ids", "attention_mask"])
# Split the dataset into train and test
train_dataset = dataset["train"]
test_dataset = dataset["test"]
# Initialize model
model = GPT2LMHeadModel.from_pretrained("gpt2")
optimizer = AdamW(model.parameters())
# Prepare everything with our `accelerator`.
model, optimizer, train_dataset, test_dataset = accelerator.prepare(model, optimizer, train_dataset, test_dataset)
# Now let's define our training loop
device = accelerator.device
model.train()
for epoch in range(3):
for step, batch in enumerate(train_dataset):
inputs = {k: v.to(device) for k, v in batch.items()}
outputs = model(**inputs)
loss = outputs.loss
accelerator.backward(loss)
optimizer.step()
optimizer.zero_grad()
# Evaluation logic
model.eval()
eval_loss = 0.0
eval_steps = 0
for batch in test_dataset:
with torch.no_grad():
inputs = {k: v.to(device) for k, v in batch.items()}
outputs = model(**inputs)
eval_loss += outputs.loss.item()
eval_steps += 1
eval_loss = eval_loss / eval_steps
print(f'Evaluation loss: {eval_loss}')
model.train()
```
let me know if it works.
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [June 28, 2023, 11:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainers-train-with-accelerate%2F33209%2F3 "2023-06-28T23:54:17Z")
> [@zsaladin](#):
>
> Is is possible to do it?
Trainer uses accelerate now under the hood, so there is nothing needed to be done or added or changed