flexitok/mod-tokenizers flexitok/mod-tokenizers-bpe_numeric_268 Updated Feb 25 flexitok/mod-tokenizers-bpe_numeric_1 Updated Feb 25
tokenization_robustness gsaltintas/social_media_informal_text Viewer • Updated Aug 18, 2025 • 106 • 276 gsaltintas/mathematical_scientific_notation Viewer • Updated Aug 18, 2025 • 47 • 44 gsaltintas/code_technical_content Viewer • Updated Aug 18, 2025 • 94 • 104 gsaltintas/named_entities Viewer • Updated Aug 18, 2025 • 3 • 17
supertoken gsaltintas/supertoken_models-llama_google-gemma-2-2b 2B • Updated Aug 11, 2025 • 3 gsaltintas/supertoken_models_gpt4o Text Generation • 2B • Updated Jun 27, 2025 gsaltintas/supertoken_models-llama_gpt2 1B • Updated Jul 23, 2025 • 3
Fineweb2-HQ Tokenizers flexitok/bpe_dan_Latn_16000 Updated Apr 10 flexitok/bpe_swe_Latn_16000 Updated Apr 10 flexitok/bpe_fas_Arab_16000 Updated Apr 10 flexitok/bpe_hun_Latn_16000 Updated Apr 10
toksuit gsaltintas/gsa-supertoken-gpt-4o 2B • Updated Aug 12, 2025 • 3 gsaltintas/supertoken_models-llama_google-gemma-2-2b-100b 2B • Updated Aug 20, 2025 • 34 gsaltintas/supertoken_models-llama_google-gemma-2-2b 2B • Updated Aug 11, 2025 • 3
flexitok/mod-tokenizers flexitok/mod-tokenizers-bpe_numeric_268 Updated Feb 25 flexitok/mod-tokenizers-bpe_numeric_1 Updated Feb 25
Fineweb2-HQ Tokenizers flexitok/bpe_dan_Latn_16000 Updated Apr 10 flexitok/bpe_swe_Latn_16000 Updated Apr 10 flexitok/bpe_fas_Arab_16000 Updated Apr 10 flexitok/bpe_hun_Latn_16000 Updated Apr 10
tokenization_robustness gsaltintas/social_media_informal_text Viewer • Updated Aug 18, 2025 • 106 • 276 gsaltintas/mathematical_scientific_notation Viewer • Updated Aug 18, 2025 • 47 • 44 gsaltintas/code_technical_content Viewer • Updated Aug 18, 2025 • 94 • 104 gsaltintas/named_entities Viewer • Updated Aug 18, 2025 • 3 • 17
toksuit gsaltintas/gsa-supertoken-gpt-4o 2B • Updated Aug 12, 2025 • 3 gsaltintas/supertoken_models-llama_google-gemma-2-2b-100b 2B • Updated Aug 20, 2025 • 34 gsaltintas/supertoken_models-llama_google-gemma-2-2b 2B • Updated Aug 11, 2025 • 3
supertoken gsaltintas/supertoken_models-llama_google-gemma-2-2b 2B • Updated Aug 11, 2025 • 3 gsaltintas/supertoken_models_gpt4o Text Generation • 2B • Updated Jun 27, 2025 gsaltintas/supertoken_models-llama_gpt2 1B • Updated Jul 23, 2025 • 3
