# Where does causal mask get generated for T5 decoder?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-does-causal-mask-get-generated-for-t5-decoder%2F68439%3E%5C
**Category:** Beginners\
**Created:** [January 9, 2024, 12:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-does-causal-mask-get-generated-for-t5-decoder%2F68439 "2024-01-09T00:04:02Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rranjan1](https://avatars.discourse-cdn.com/v4/letter/r/df788c/32.png) [@rranjan1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frranjan1)%5C
**Post date:** [January 9, 2024, 12:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-does-causal-mask-get-generated-for-t5-decoder%2F68439%2F1 "2024-01-09T00:04:02Z")

</div>

I am trying to figure out the causal mask implementation for T5 encoder-decoder model. The docstring for decoder\_attention\_mask says that “Causal mask will also be used by default”. However, I do not find any code in modeling\_t5.py that is generating the causal mask. Can somebody point me to the place where causal mask is generated for T5 decoder?

---

<div class="post-metadata">

**Author:** ![rranjan1](https://avatars.discourse-cdn.com/v4/letter/r/df788c/32.png) [@rranjan1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frranjan1)%5C
**Post date:** [January 9, 2024, 12:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-does-causal-mask-get-generated-for-t5-decoder%2F68439%2F2 "2024-01-09T00:30:12Z")

</div>

Found it. The causal mask gets created in get\_extended\_attention\_mask here: [transformers/src/transformers/modeling\_utils.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L979)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [January 9, 2024, 12:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-does-causal-mask-get-generated-for-t5-decoder%2F68439%2F3 "2024-01-09T12:30:39Z")

</div>

Closed on January 9, 2024, 12:30pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.
