# PReTrain RoBERTa from scratch in Norwegian

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-norwegian%2F7093%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 23, 2021, 11:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-norwegian%2F7093 "2021-06-23T11:43:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 23, 2021, 11:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-norwegian%2F7093%2F1 "2021-06-23T11:43:30Z")

</div>

# RoBERTa/BERT for Norwegian

Currently, there is only a very limited amount of BERT-like models for Thai on the hub: [Models – Hugging Face](https://huggingface.co/models?filter=bert,no) . For this project, the goal is to create a RoBERTa/BERT model for just the Norwegian language.

## Model

A randomly initialized RoBERTa/BERT model

## Datasets

One can make use [OSCAR](https://oscar-corpus.com/) the dataset is also available through the `datasets` library here: [Oscar (Oscar Kjell)](https://huggingface.co/datasets/oscar).

## Available training scripts

A masked language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py). It can be used pretty much without any required code changes.

## (Optional) Desired project outcome

The desired project output is a strong RoBERTa/BERT model in Norwegian.

## (Optional) Challenges

The OSCAR dataset might be too small (it has \< 5GB of data for Thai). Also it might be important  
to find datasets the BERT-like model can be evaluated on after pretraining in Norwegian. Having found a dataset to fine-tune the pretrained BERT-like model on, one can make use of the text-classification script [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py)

## (Optional) Links to read upon

The most important read would be the following colab:

- [Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/masked_language_modeling_flax.ipynb)

---

<div class="post-metadata">

**Author:** ![versae](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/versae/32/2153_2.png) [@versae](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fversae)%5C
**Post date:** [June 28, 2021, 12:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-norwegian%2F7093%2F2 "2021-06-28T12:10:08Z")

</div>

I am also really interested in this one, or maybe even ALBERT. But the available dataset on OSCAR might not be enough. Using mC4 might add almost 100GB of clean-ish text. I have worked on [Norwegian BERT-based models before](https://www.aclweb.org/anthology/2021.nodalida-main.3/), and we (the National Library of Norway) have a big corpus that unfortunately we are not allowed to share. We could maybe share parts of it to complement mC4/OSCAR, but not all it. Not sure if there is a way to train on a private corpus in this event, I guess not, but worth asking.

There are also datasets for extrinsic evaluation on Norwegian for POS, NER, and sentiment (the sentiment one I feel was kinda designed for word2vec-like models) from the UiO LTG group, another on political speech party affiliation we built from the Talks of Norway, and a recent one on hate speech we could use. Not sure if evaluating on WikiANN could be an option.

---

<div class="post-metadata">

**Author:** ![saattrupdan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saattrupdan/32/28869_2.png) [@saattrupdan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaattrupdan)%5C
**Post date:** [June 28, 2021, 5:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-norwegian%2F7093%2F3 "2021-06-28T17:39:36Z")

</div>

It could make sense to pretrain a common Scandinavian model instead, as our languages are so similar. I’ve made a project suggestion here: [Scandinavian RoBERTa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscandinavian-roberta%2F7533).
