# AdamW Pytorch vs Huggingface

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadamw-pytorch-vs-huggingface%2F30562%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 27, 2023, 1:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadamw-pytorch-vs-huggingface%2F30562 "2023-01-27T13:16:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JAugusto97](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@JAugusto97](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJAugusto97)%5C
**Post date:** [January 27, 2023, 1:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadamw-pytorch-vs-huggingface%2F30562%2F1 "2023-01-27T13:16:57Z")

</div>

I noticed that the default weight decay parameter differs between pytorch’s implementation and huggingface’s (0 on transformers, 1e-2 on pytorch). Also, when finetuning bert we usually set the weight decay directly on the layer parameters, which makes me think the correct value for the optimizer should be 0, otherwise you would be adding weight decay twice (?).

Example:

```auto
no_decay = ["bias", "LayerNorm.weight", "LayerNorm.bias"]
optimizer_grouped_parameters = [
            {
                "params": [
                    p
                    for n, p in model.named_parameters()
                    if not any(nd in n for nd in no_decay)
                ],
                "weight_decay": self.hparams.weight_decay,
            },
            {
                "params": [
                    p
                    for n, p in model.named_parameters()
                    if any(nd in n for nd in no_decay)
                ],
                "weight_decay": 0.0,
            },
        ]
optimizer = AdamW(
            optimizer_grouped_parameters,
            lr=self.hparams.learning_rate,
            eps=self.hparams.adam_epsilon,
            weight_decay=?
)

```

I think transformers encourages you to use pytorch’s implementation using a deprecation warning, so that makes it even more confusing.

AdamW pytorch: [AdamW — PyTorch 1.13 documentation](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html)  
AdamW transformers: [Optimization](https://huggingface.co/docs/transformers/main_classes/optimizer_schedules#transformers.AdamW)
