# Roberta hidden\_states\[0\] == Bert pooler\_output?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Froberta-hidden-states-0-bert-pooler-output%2F20817%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 25, 2022, 7:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Froberta-hidden-states-0-bert-pooler-output%2F20817 "2022-07-25T07:42:38Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![81Yuji](https://avatars.discourse-cdn.com/v4/letter/8/47e85d/32.png) [@81Yuji](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2F81Yuji)%5C
**Post date:** [July 25, 2022, 7:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Froberta-hidden-states-0-bert-pooler-output%2F20817%2F1 "2022-07-25T07:42:38Z")

</div>

I want to feed the last layer hidden state which is generated by RoberTa.

out = pretrained\_roberta(dummy\_input[“input\_ids”],  
dummy\_input[“attention\_mask”], output\_hidden\_states=True)

out = out.hidden\_states[0]  
out = nn.Dense(features=3)(out)

Is that equivalent to pooler\_output in Bert?

pooler\_output (torch.FloatTensor of shape (batch\_size, hidden\_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task.