Model uploaded

README.md

@@ -1,3 +1,218 @@
 ---
+language:
+
+- ca
+
 license: apache-2.0
+
+tags:
+
+- "catalan"
+
+- "masked-lm"
+
+- "RoBERTa-large-ca"
+
+- "CaText"
+
+- "Catalan Textual Corpus"
+
+widget:
+- text: "El Català és una llengua molt <mask>."
+- text: "Salvador Dalí va viure a <mask>."
+- text: "La Costa Brava té les millors <mask> d'Espanya."
+- text: "El cacaolat és un batut de <mask>."
+- text: "<mask> és la capital de la Garrotxa."
+- text: "Vaig al <mask> a buscar bolets."
+- text: "Antoni Gaudí vas ser un <mask> molt important per la ciutat."
+- text: "Catalunya és una referència en <mask> a nivell europeu."
+
 ---
+
+# Catalan BERTa (roberta-large-ca) large model
+
+## Table of Contents
+<details>
+<summary>Click to expand</summary>
+
+- [Model Description](#model-description)
+- [Intended Uses and Limitations](#intended-uses-and-limitations)
+- [How to Use](#how-to-use)
+- [Training](#training)
+  - [Training Data](#training-data)
+  - [Training Procedure](#training-procedure)
+- [Evaluation](#evaluation)
+   - [CLUB Benchmark](#club-benchmark)
+   - [Evaluation Results](#evaluation-results)
+- [Licensing Information](#licensing-information)
+- [Citation Information](#citation-information)
+- [Funding](#funding)
+- [Contributions](#contributions)
+
+</details>
+
+## Model description
+
+The **roberta-large-ca** is a transformer-based masked language model for the Catalan language. 
+It is based on the [RoBERTA](https://github.com/pytorch/fairseq/tree/master/examples/roberta) large model 
+and has been trained on a medium-size corpus collected from publicly available corpora and crawlers.
+
+## Intended Uses and Limitations
+
+**roberta-large-ca** model is ready-to-use only for masked language modeling to perform the Fill Mask task (try the inference API or read the next section).
+However, it is intended to be fine-tuned on non-generative downstream tasks such as Question Answering, Text Classification, or Named Entity Recognition.
+
+## How to Use
+
+Here is how to use this model:
+
+```python
+from transformers import AutoModelForMaskedLM
+from transformers import AutoTokenizer, FillMaskPipeline
+from pprint import pprint
+tokenizer_hf = AutoTokenizer.from_pretrained('projecte-aina/roberta-large-ca')
+model = AutoModelForMaskedLM.from_pretrained('projecte-aina/roberta-large-ca')
+model.eval()
+pipeline = FillMaskPipeline(model, tokenizer_hf)
+text = f"Em dic <mask>."
+res_hf = pipeline(text)
+pprint([r['token_str'] for r in res_hf])
+```
+
+## Training
+
+### Training data
+
+The training corpus consists of several corpora gathered from web crawling and public corpora.
+
+
+| Corpus                  | Size in GB |
+|-------------------------|------------|
+| Catalan Crawling        | 13.00      |
+| Wikipedia               | 1.10       |
+| DOGC                    | 0.78       |
+| Catalan Open Subtitles  | 0.02       |
+| Catalan Oscar           | 4.00       |
+| CaWaC                   | 3.60       |
+| Cat. General Crawling   | 2.50       |
+| Cat. Goverment Crawling | 0.24       |
+| ACN                     | 0.42       |
+| Padicat                 | 0.63       |
+| RacoCatalá              | 8.10       |
+| Nació Digital           | 0.42       |
+| Vilaweb                 | 0.06       |
+| Tweets                  | 0.02       |
+
+### Training Procedure
+
+The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2)
+used in the original [RoBERTA](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model with a vocabulary size of 52,000 tokens. 
+The RoBERTa-large pretraining consists of a masked language model training that follows the approach employed for the RoBERTa large model
+with the same hyperparameters as in the original work.
+The training lasted a total of 96 hours with 32 NVIDIA V100 GPUs of 16GB DDRAM.
+
+
+## Evaluation
+
+### CLUB Benchmark
+
+The BERTa-large model has been fine-tuned on the downstream tasks of the Catalan Language Understanding Evaluation benchmark (CLUB),
+that has been created along with the model.
+
+It contains the following tasks and their related datasets:
+
+ 1. Named Entity Recognition (NER)
+
+    
+    **[NER (AnCora)](https://zenodo.org/record/4762031#.YKaFjqGxWUk)**: extracted named entities from the original [Ancora](https://doi.org/10.5281/zenodo.4762030) version,
+    filtering out some unconventional ones, like book titles, and transcribed them into a standard CONLL-IOB format
+
+
+ 2. Part-of-Speech Tagging (POS)
+    
+    **[POS (AnCora)](https://zenodo.org/record/4762031#.YKaFjqGxWUk)**: from the [Universal Dependencies treebank](https://github.com/UniversalDependencies/UD_Catalan-AnCora) of the well-known Ancora corpus.
+
+ 3. Text Classification (TC)
+     
+    **[TeCla](https://huggingface.co/datasets/projecte-aina/tecla)**: consisting of 137k news pieces from the Catalan News Agency ([ACN](https://www.acn.cat/)) corpus, with 30 labels.
+
+ 4. Textual Entailment (TE)
+     
+    **[TE-ca](https://huggingface.co/datasets/projecte-aina/teca)**: consisting of 21,163 pairs of premises and hypotheses, annotated according to the inference relation they have (implication, contradiction, or neutral), extracted from the [Catalan Textual Corpus](https://huggingface.co/datasets/projecte-aina/catalan_textual_corpus).
+
+ 5. Semantic Textual Similarity (STS)
+    
+    **[STS-ca](https://huggingface.co/datasets/projecte-aina/sts-ca)**: consisting of more than 3000 sentence pairs, annotated with the semantic similarity between them, scraped from the [Catalan Textual Corpus](https://huggingface.co/datasets/projecte-aina/catalan_textual_corpus).
+
+ 6. Question Answering (QA):
+    
+    **[VilaQuAD](https://huggingface.co/datasets/projecte-aina/vilaquad)**: contains 6,282 pairs of questions and answers, outsourced from 2095 Catalan language articles from VilaWeb newswire text.
+    
+    **[ViquiQuAD](https://huggingface.co/datasets/projecte-aina/viquiquad)**: consisting of more than 15,000 questions outsourced from Catalan Wikipedia randomly chosen from a set of 596 articles that were originally written in Catalan.
+   
+    **[CatalanQA](https://huggingface.co/datasets/projecte-aina/catalanqa)**: an aggregation of 2 previous datasets (VilaQuAD and ViquiQuAD), 21,427 pairs of Q/A balanced by type of question, containing one question and one answer per context, although the contexts can repeat multiple times.
+    
+    **[XQuAD-ca](https://huggingface.co/datasets/projecte-aina/xquad-ca)**: the Catalan translation of XQuAD, a multilingual collection of manual translations of 1,190 question-answer pairs from English Wikipedia used only as a _test set_.
+    
+Here are the train/dev/test splits of the datasets:
+
+| Task (Dataset) | Total | Train | Dev  | Test |
+|:--|:--|:--|:--|:--|
+| NER (Ancora)  |13,581 | 10,628 | 1,427 | 1,526 |
+| POS (Ancora)| 16,678 | 13,123 | 1,709 | 1,846 |
+| STS (STS-ca)         | 3,073 | 2,073 | 500 | 500 |
+| TC (TeCla) |  137,775 | 110,203 | 13,786 |  13,786|
+| TE (TE-ca) |  21,163 | 16,930 | 2,116 | 2,117
+| QA (VilaQuAD) | 6,282  | 3,882  | 1,200  | 1,200 |
+| QA (ViquiQuAD) | 14,239  | 11,255  | 1,492  | 1,429 |
+| QA (CatalanQA) | 21,427  | 17,135  | 2,157  | 2,135 |
+
+### Evaluation Results
+
+| Task        | NER (F1)      | POS (F1)   | STS-ca (Comb)   | TeCla (Acc.) | TEca (Acc.) | VilaQuAD (F1/EM)| ViquiQuAD (F1/EM) | CatalanQA (F1/EM) | XQuAD-ca <sup>1</sup> (F1/EM) | 
+| ------------|:-------------:| -----:|:------|:------|:-------|:------|:----|:----|:----|
+| RoBERTa-large-ca        | **89.82** | **99.02** | **83.41** | **75.46** | **83.61** | **89.34**/75.50 | **89.20**/75.77 | **90.72/79.06** | **73.79**/55.34 |
+| RoBERTa-base-ca-v2      | 89.29 | 98.96 | 79.07 | 74.26 | 83.14 | 87.74/72.58 | 88.72/**75.91** | 89.50/76.63 | 73.64/**55.42** |
+| BERTa                   | 89.76 | 98.96 | 80.19 | 73.65 | 79.26 | 85.93/70.58 | 87.12/73.11 | 89.17/77.14 | 69.20/51.47 |
+| mBERT                   | 86.87 | 98.83 | 74.26 | 69.90 | 74.63 | 82.78/67.33 | 86.89/73.53 | 86.90/74.19 | 68.79/50.80 |
+| XLM-RoBERTa             | 86.31 | 98.89 | 61.61 | 70.14 | 33.30 | 86.29/71.83 | 86.88/73.11 | 88.17/75.93 | 72.55/54.16 |
+
+<sup>1</sup> : Trained on CatalanQA, tested on XQuAD-ca.
+
+## Licensing Information
+
+[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+## Citation Information 
+
+If you use any of these resources (datasets or models) in your work, please cite our latest paper:
+```bibtex
+@inproceedings{armengol-estape-etal-2021-multilingual,
+    title = "Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? {A} Comprehensive Assessment for {C}atalan",
+    author = "Armengol-Estap{\'e}, Jordi  and
+      Carrino, Casimiro Pio  and
+      Rodriguez-Penagos, Carlos  and
+      de Gibert Bonet, Ona  and
+      Armentano-Oller, Carme  and
+      Gonzalez-Agirre, Aitor  and
+      Melero, Maite  and
+      Villegas, Marta",
+    booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",
+    month = aug,
+    year = "2021",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.findings-acl.437",
+    doi = "10.18653/v1/2021.findings-acl.437",
+    pages = "4933--4946",
+}
+```
+
+### Funding
+
+This work was funded by the [Departament de la Vicepresidència i de Polítiques Digitals i Territori de la Generalitat de Catalunya](https://politiquesdigitals.gencat.cat/ca/inici/index.html#googtrans(ca|en) within the framework of [Projecte AINA](https://politiquesdigitals.gencat.cat/ca/economia/catalonia-ai/aina).
+
+## Contributions
+
+[N/A]
+

config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.20.1",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 50262
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb19948b086142bf513a7ad2d818c42e57edb5bfd8032c832130f5837584abfb
+size 1421767851

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": true, "errors": "replace", "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "max_len": 512, "special_tokens_map_file": null, "name_or_path": "/gpfs/projects/bsc88/tools/corpus-utils-lm/17-06-2021-python/output/jsc_ca_output/roberta-2022-03-21-1502-3a6a-69ad/train_tokenizer_output_fix/train-tokenizer-2022-03-21-1502-3a6a-0e8b"}