# Is the trainer's seed reset at every model\_init?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-the-trainers-seed-reset-at-every-model-init%2F16161%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 28, 2022, 12:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-the-trainers-seed-reset-at-every-model-init%2F16161 "2022-03-28T12:33:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [March 28, 2022, 12:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-the-trainers-seed-reset-at-every-model-init%2F16161%2F1 "2022-03-28T12:33:31Z")

</div>

Because `seed` is a TrainingArgument, we can use it as in hyperparameter optimization if we’d want to, e.g.

```auto
def hparams_ray(trial):
    from ray import tune

    return {
        "learning_rate": tune.loguniform(1e-6, 1e-3),
        "per_device_train_batch_size": tune.choice([4, 8, 16, 32]),
         "seed": tune.choice(range(1, 43)),
    }

```

But I am looking for clarification what happens when we do not include `seed` here. Is the seed (or any training argument for that matter), reset at each trial? This is important: if it is not reset, then every trial has a different starting seed and hyperparameter search is not independent.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 28, 2022, 2:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-the-trainers-seed-reset-at-every-model-init%2F16161%2F2 "2022-03-28T14:22:31Z")

</div>

Yes, it’s done at [that line](https://github.com/huggingface/transformers/blob/7ca46335553609e4852dcb018c73cd5215e6e25a/src/transformers/trainer.py#L303) specifically.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [March 28, 2022, 3:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-the-trainers-seed-reset-at-every-model-init%2F16161%2F3 "2022-03-28T15:08:49Z")

</div>

I guess I misunderstand how Ray (or Optuna) interacts with the trainer then.

I thought that the Trainer is never copied/reinitialized but only the model is reinitialized for every trial. You seem to suggest that a new trainer is created for every trial (and in the trainer init, set\_seed is called), is that correct?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 28, 2022, 4:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-the-trainers-seed-reset-at-every-model-init%2F16161%2F4 "2022-03-28T16:52:59Z")

</div>

Pasted the wrong link, sorry. It is reinitialized (with the seed set) at each new call to `train` [here](https://github.com/huggingface/transformers/blob/7ca46335553609e4852dcb018c73cd5215e6e25a/src/transformers/trainer.py#L1155)

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [March 28, 2022, 4:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-the-trainers-seed-reset-at-every-model-init%2F16161%2F5 "2022-03-28T16:56:47Z")

</div>

Great, that was what I was looking for. Thanks!
