# BertForSequenceClassification Index Error

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertforsequenceclassification-index-error%2F343%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 17, 2020, 6:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertforsequenceclassification-index-error%2F343 "2020-07-17T06:28:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![iamchirag](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iamchirag/32/168_2.png) [@iamchirag](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fiamchirag)%5C
**Post date:** [July 17, 2020, 6:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertforsequenceclassification-index-error%2F343%2F1 "2020-07-17T06:28:57Z")

</div>

Hey Beginner Here, I was using BertForSequenceClassification using this code:

```auto
def train(model, optimizer, critertion=nn.BCELoss(),train_loader=train_iter,valid_loader=valid_iter,num_epochs=5
                   ,eval_every = len(train_iter) // 2,file_path = "",best_valid_loss = float("Inf")):
    # initialize running values
    running_loss = 0.0
    valid_running_loss = 0.0
    global_step = 0
    train_loss_list = []
    valid_loss_list = []
    global_steps_list = []
    
    model.train()
    for epoch in range(num_epochs):
        for (labels, title, text, titletext), _ in train_loader:
            labels = labels.type(torch.LongTensor) 
            labels = labels.to(device)
            
            titletext = titletext.type(torch.LongTensor)  
            titletext = titletext.to(device)
            print(labels.shape)
            print(titletext.shape)
            
            output = model(titletext, labels)
            loss, _ = output
            
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            
            running_loss += loss.item()
            global_step += 1
#removed other part of the code which was for validation and testing. Error is generated in train loop.

```

but when the run the code it shows me the following error:

```auto
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-63-e4474bff9c36> in <module>
      2 optimizer = optim.Adam(model.parameters(), lr=2e-5)
      3 
----> 4 train(model=model, optimizer=optimizer)

<ipython-input-62-e6359dc8788e> in train(model, optimizer, critertion, train_loader, valid_loader, num_epochs, eval_every, file_path, best_valid_loss)
     20 print(titletext.shape)
     21 
---> 22 output = model(titletext, labels)
     23 loss, _ = output
     24 

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__ (self, *input, **kwargs)
    548 result = self._slow_forward(*input, **kwargs)
    549 else:
--> 550 result = self.forward(*input, **kwargs)
    551 for hook in self._forward_hooks.values():
    552 hook_result = hook(self, input, result)

<ipython-input-59-3d3782128a40> in forward(self, text, label)
      7 
      8 def forward(self, text, label):
----> 9 loss, text_fea = self.encoder(text, labels=label)[:2]
     10 
     11 return loss, text_fea

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__ (self, *input, **kwargs)
    548 result = self._slow_forward(*input, **kwargs)
    549 else:
--> 550 result = self.forward(*input, **kwargs)
    551 for hook in self._forward_hooks.values():
    552 hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels)
   1158 else:
   1159 loss_fct = CrossEntropyLoss()
-> 1160 loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
   1161 outputs = (loss,) + outputs
   1162 

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__ (self, *input, **kwargs)
    548 result = self._slow_forward(*input, **kwargs)
    549 else:
--> 550 result = self.forward(*input, **kwargs)
    551 for hook in self._forward_hooks.values():
    552 hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    930 def forward(self, input, target):
    931 return F.cross_entropy(input, target, weight=self.weight,
--> 932 ignore_index=self.ignore_index, reduction=self.reduction)
    933 
    934 

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2315 if size_average is not None or reduce is not None:
   2316 reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2317 return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2318 
   2319 

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2113 .format(input.size(0), target.size(0)))
   2114 if dim == 2:
-> 2115 ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2116 elif dim == 4:
   2117 ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target 5213 is out of bounds.

```

meanwhile, the shape of **label** and **titletext** is **[16]** and **[16,128]** respectively.

- I tried to unsqueeze label too but that didn’t help.
- I tried to whether if the target index is missing or the label and target are not of same length but it didn’t help as well.

What can be done to fix this?

Full Code + Dataset : [Click here](https://www.kaggle.com/iamchiragsharma/bert-fine-tuning-fake-news-classification?scriptVersionId=38928838)

---

<div class="post-metadata">

**Author:** ![stas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stas/32/19289_2.png) [@stas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstas)%5C
**Post date:** [July 19, 2020, 5:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertforsequenceclassification-index-error%2F343%2F2 "2020-07-19T05:48:13Z")

</div>

OK, I had a look at your code - your input data that you feed to the model is very borked. Your labels instead of being {0,1} are row ids converted to float. Always dump at least the first row/batch of your inputs to see that what you feed is what you expect it to be. In your case, for labels in the first batch you get something like:

```auto
x = next(iter(train_iter))
x.label

```

```auto
tensor([.169, .254, .512, ...

```

definitely not 2 categories.

The 2 main errors are that you (1) save the row IDs in the csv files and `TabularDataset.splits` gives labels the row ids (2) you don’t convert FAKE/REAL strings to {0,1}.

So before you do `X_train.to_csv(...`, you need to:

```auto
news['label'] = news['label'].astype('category').cat.codes

```

and also you need to drop the row IDs, so the correct code is:

```auto
X_train.to_csv("./real-and-fake-news-dataset/train.csv", index=False)
X_test.to_csv("./real-and-fake-news-dataset/test.csv", index=False)
X_valid.to_csv("./real-and-fake-news-dataset/valid.csv", index=False)

```

and later the fields need to be corrected too (label moved to the end):

```auto
fields = [('title', text_field), ('text', text_field), ('titletext', text_field), ('label', label_field),]

```

alternatively you could keep the row ids and then adjust your train loop to ignore the first field.

* * *

I don’t think `fix_length=MAX_SEQ_LEN` does what you think it does - I think the description of that field is confusing and misleading - as it’s related to padding and not truncating - you get millions of warnings during `TabularDataset.splits` call.

```auto
Token indices sequence length is longer than the specified maximum sequence length for this model (1129 > 512). Running this sequence through the model will result in indexing errors

```

So I added:

```auto
news['titletext'] = news['titletext'].str.slice(0,128)
news['title'] = news['title'].str.slice(0,128)
news['text'] = news['text'].str.slice(0,128)

```

you have other bugs in your code, but you will sort those out.

I uploaded the partially corrected nb [here](https://gist.github.com/stas00/2c711f43422f8469f188bf302914bfa5).

You can see the debug prints I added in the code.

Your csv files were also saved in the wrong dir (not the one it was read from). So I adjusted the dirs too.

You will need to remove the initial dataset truncation I added to make the dev cycle fast (and note that trick for your own future development process).
