vietnamese-gpt2

This model is a fine-tuned version of nmthien/vietnamese-gpt2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.0338

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.00015
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• training_steps: 18300
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
3.2310	0.0546	1000	3.1417
3.2352	0.1093	2000	3.1292
3.2325	0.1639	3000	3.1168
3.2855	0.2186	4000	3.1082
3.1793	0.2732	5000	3.0997
3.2052	0.3279	6000	3.0917
3.2149	0.3825	7000	3.0852
3.1383	0.4372	8000	3.0775
3.1691	0.4918	9000	3.0718
3.2341	0.5464	10000	3.0647
3.2229	0.6011	11000	3.0598
3.1826	0.6557	12000	3.0539
3.1602	0.7104	13000	3.0485
3.2108	0.7650	14000	3.0441
3.1858	0.8197	15000	3.0407
3.1510	0.8743	16000	3.0375
3.1331	0.9290	17000	3.0355
3.1221	0.9836	18000	3.0338

Framework versions

• Transformers 5.0.0
• Pytorch 2.10.0+cu128
• Datasets 5.0.0
• Tokenizers 0.22.2