# Do you train all layers when fine-tuning T5?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%3E%5C
**Category:** Beginners\
**Created:** [September 8, 2020, 4:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034 "2020-09-08T04:04:19Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![hyura](https://avatars.discourse-cdn.com/v4/letter/h/8797f3/32.png) [@hyura](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhyura)%5C
**Post date:** [September 8, 2020, 4:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%2F1 "2020-09-08T04:04:19Z")

</div>

From my beginner-level understanding, when it comes to BERT, sometimes people will train just the last layer, or sometimes they’ll train all layers a couple epochs and then the last layer a few more epochs.

Does T5 have any similar practices? Or is it normal to just train the whole thing when fine-tuning?

And very tangentially related: to fine-tune T5, we just do `loss.backward()` on the result of the `forward()` call of the T5 model right (the `loss` key in the returned dict)? So there’s no need to calculate any loss on our own?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 8, 2020, 6:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%2F2 "2020-09-08T06:04:39Z")

</div>

I haven’t seen much experiments for this, but IMO it’s better to fine-tune the whole model.

Also when you pass `labels` argument to `T5ForConditionalGeneration`'s `forward` method then it calculates the loss for you and returns it as the first value in the returned `tuple` .

And you can use the `finetune.py` script [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) to fine-tuning T5 and other seq2seq models

See this thread [T5 Finetuning Tips](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-finetuning-tips%2F684)

---

<div class="post-metadata">

**Author:** ![ivankrstev7](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ivankrstev7/32/4201_2.png) [@ivankrstev7](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fivankrstev7)%5C
**Post date:** [September 29, 2021, 12:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%2F3 "2021-09-29T12:23:34Z")

</div>

Hi,  
I am trying to fine tune T5 for conditional text generation, and it works perfectly. But, I would like to have more control over the layers I am training i.e. freeze and unfreeze certain layers. For example, I am also using GPT2 for the same use case and I do something like this:

```auto
# - Freeze selective layers:
# - Freeze all layers except last n:
for parameter in model.parameters():
    parameter.requires_grad = False

for i, m in enumerate(model.transformer.h):        
    #Only un-freeze the last n transformer blocks
    if i+1 > 12 - config['UNFREEZE_LAST_N']:
        for parameter in m.parameters():
            parameter.requires_grad = True 

for parameter in model.transformer.ln_f.parameters():        
    parameter.requires_grad = True

for parameter in model.lm_head.parameters():        
    parameter.requires_grad = True

```

Now, I am able to freeze all the layers in T5, but I have trouble unfreezing the last n layers, since T5ForConditionalGeneration has no attribute “transformer”. Does anyone has an idea on how to achieve this, because I couldn’t find anything about this issue?

---

<div class="post-metadata">

**Author:** ![TheLongSentance](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thelongsentance/32/2470_2.png) [@TheLongSentance](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTheLongSentance)%5C
**Post date:** [September 30, 2021, 3:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%2F4 "2021-09-30T15:34:17Z")

</div>

Maybe start this as a fresh topic with a more specific title to your issue - likely to stand more chance of a response? (sorry can’t help, just my experience of this forum!).

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [October 1, 2021, 7:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%2F5 "2021-10-01T07:58:00Z")

</div>

You can print out any PyTorch model to see its architecture, or you can simply look at the source code. On the highest level you’ll find shared embeddings `shared`, an encoder `encoder`, a deocder `decoder` and a LM head `lm_head`. You can use any of those. So for instance, unfreezing the decoder transformer layers:

```auto
for i, m in enumerate(model.decoder.block):        
    #Only un-freeze the last n transformer blocks in the decoder
    if i+1 > 12 - config['UNFREEZE_LAST_N']:
        for parameter in m.parameters():
            parameter.requires_grad = True 

```

It is `block` because the `decoder` is a `T5Stack` which has its transformer layers as a ModuleList called `block`.

> <https://github.com/huggingface/transformers/blob/8bbb53e20b7873ba7f63be70d4d798e0c3568bfa/src/transformers/models/t5/modeling_t5.py#L811>

So it isn’t hard at all, you just have to go look for the right name in the code.

---

<div class="post-metadata">

**Author:** ![ivankrstev7](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ivankrstev7/32/4201_2.png) [@ivankrstev7](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fivankrstev7)%5C
**Post date:** [October 1, 2021, 11:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%2F6 "2021-10-01T11:56:15Z")

</div>

Thank you, this was actually very helpful 🙂

---

<div class="post-metadata">

**Author:** ![VGan](https://avatars.discourse-cdn.com/v4/letter/v/ea666f/32.png) [@VGan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVGan)%5C
**Post date:** [September 26, 2023, 7:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%2F7 "2023-09-26T07:48:18Z")

</div>

Do you guys know what the default behavior is? Does finetuning unfreeze all layers and weights? Or does it just unfreeze the LM head (linear projection)?

---

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdblakely)%5C
**Post date:** [September 26, 2023, 10:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-you-train-all-layers-when-fine-tuning-t5%2F1034%2F8 "2023-09-26T10:45:43Z")

</div>

The default behavior in Huggingface is to fine-tune the whole model, there’s no freezing. If you want to freeze layers, you actually have to write some code yourself to do that.
