# LM finetuning on domain specific unlabelled data

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433%3E%5C
**Category:** Beginners\
**Created:** [April 12, 2021, 11:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433 "2021-04-12T11:54:39Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sML](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sML](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FsML)%5C
**Post date:** [April 12, 2021, 11:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433%2F1 "2021-04-12T11:54:39Z")

</div>

Hello Team,  
Thanks a lot for the great work!  
Can you please tell me how to finetune a(any) MLM model on domain specific corpus ? I am following this [link](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) obtained from the huggingface documentation. Is this the procedure I should be following ? if this is how it is done, how will this update the vocabulary to adapt to new tokens of my domain specific corpus ?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![sML](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sML](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FsML)%5C
**Post date:** [April 17, 2021, 8:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433%2F2 "2021-04-17T08:18:07Z")

</div>

Can anyone help ?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [April 17, 2021, 4:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433%2F3 "2021-04-17T16:18:33Z")

</div>

hey @sML, in general fine-tuning will _not_ update the vocabulary of the tokenizer as the vocabulary is specific to the corpus that the model was pretrained on.

if your domain is not too different from the corpus used to pretrain the model, then i would just try fine-tuning the LM (following the tutorial you lined to) to see what kind of results you get.

one alternative would be to manually add new tokens to the tokenizer’s vocabulary, e.g.

```auto
tokenizer = ...
model = ...

new_toks = [tok1, tok2, ..., tokN]
# docs: https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=add_tokens#transformers.tokenization_utils_base.SpecialTokensMixin.add_tokens
added_toks = tokenizer.add_tokens(new_toks)
# you need to resize the dimension of the token embeddings to match the new vocab size
model.resize_token_embeddings(len(tokenizer))

```

you can find a more sophisticated approach described in this paper: [https://www.aclweb.org/anthology/2020.findings-emnlp.129.pdf](https://www.aclweb.org/anthology/2020.findings-emnlp.129.pdf)

hth!

---

<div class="post-metadata">

**Author:** ![sML](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sML](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FsML)%5C
**Post date:** [April 17, 2021, 4:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433%2F4 "2021-04-17T16:36:47Z")

</div>

Thanks a lot for the clarification. Apparently my domain is completely different from the actual corpus. Adding tokens manually also doesn’t seems feasible. I will go through the paper provided. Thank you once again.

---

<div class="post-metadata">

**Author:** ![clin](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@clin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclin)%5C
**Post date:** [April 21, 2021, 3:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433%2F5 "2021-04-21T15:56:35Z")

</div>

If your domain vocabulary is totally different, then you probably do not want to adapt an existing, pretrained LM model to your domain-specific corpus. In this case, it probably makes more sense to train an LM from scratch.

You can check how different your domain is from an existing pretrained vocabulary by running the corresponding pretrained tokenizer against a sample of your domain’s text and counting the number of unknown tokens that show up.

In my experience, the pretrained tokenizers like BPE will not return any unknown tokens if your domain-specific corpus is in the same language (e.g., English).

---

<div class="post-metadata">

**Author:** ![sML](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sML](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FsML)%5C
**Post date:** [April 21, 2021, 4:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433%2F6 "2021-04-21T16:36:21Z")

</div>

Thankyou for the response!  
Okay. My corpus is in English itself, but very much specific to a particular domain. Would it be worth trying MLM fine-tuning with BPE tokenizer, may be like Roberta?

---

<div class="post-metadata">

**Author:** ![clin](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@clin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclin)%5C
**Post date:** [April 21, 2021, 5:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-finetuning-on-domain-specific-unlabelled-data%2F5433%2F7 "2021-04-21T17:54:38Z")

</div>

“Vocabulary” as it is used in the context of tokenizers is not the same as the common English definition of “vocabulary”, i.e., a tokenizer doesn’t necessarily break up a text into intelligible English words. You may want to read up on how tokenizers work:

> **[Summary of the tokenizers](https://huggingface.co/docs/transformers/main/tokenizer_summary)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
