Awesome Open Source
Awesome Open Source


HerBERT is a series of BERT-based language models trained for Polish language understanding.

All three HerBERT models are summarized below:

Model Tokenizer Vocab Size Batch Size Train Steps KLEJ Score
herbert-klej-cased-v1 BPE 50K 570 180k 80.5
herbert-base-cased BPE-Dropout 50K 2560 50k 86.3
herbert-large-cased BPE-Dropout 50K 2560 60k 88.4

Full KLEJ Benchmark leaderboard is available here.

For more details about model architecture, training process, used corpora and evaluation please refer to:


Example of how to load the model:

from transformers import AutoTokenizer, AutoModel

model_names = {
    "herbert-klej-cased-v1": {
        "tokenizer": "allegro/herbert-klej-cased-tokenizer-v1", 
        "model": "allegro/herbert-klej-cased-v1",
    "herbert-base-cased": {
        "tokenizer": "allegro/herbert-base-cased", 
        "model": "allegro/herbert-base-cased",
    "herbert-large-cased": {
        "tokenizer": "allegro/herbert-large-cased", 
        "model": "allegro/herbert-large-cased",

tokenizer = AutoTokenizer.from_pretrained(model_names["allegro/herbert-base-cased"]["tokenizer"])
model = AutoModel.from_pretrained(model_names["allegro/herbert-base-cased"]["model"])

And how to use the model:

output = model(
                "A potem szedł środkiem drogi w kurzawie, bo zamiatał nogami, ślepy dziad prowadzony przez tłustego kundla na sznurku.",
                "A potem leciał od lasu chłopak z butelką, ale ten ujrzawszy księdza przy drodze okrążył go z dala i biegł na przełaj pól do karczmy."


CC BY 4.0


If you use this model, please cite the following papers:

The herbert-klej-cased-v1 version of the model:

    title = "{KLEJ}: Comprehensive Benchmark for Polish Language Understanding",
    author = "Rybak, Piotr and Mroczkowski, Robert and Tracz, Janusz and Gawlik, Ireneusz",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "",
    pages = "1191--1201",

The herbert-base-cased or herbert-large-cased version of the model:

    title = "{H}er{BERT}: Efficiently Pretrained Transformer-based Language Model for {P}olish",
    author = "Mroczkowski, Robert  and
      Rybak, Piotr  and
      Wr{\'o}blewska, Alina  and
      Gawlik, Ireneusz",
    booktitle = "Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing",
    month = apr,
    year = "2021",
    address = "Kiyv, Ukraine",
    publisher = "Association for Computational Linguistics",
    url = "",
    pages = "1--10",


You can contact us at: [email protected]

Related Awesome Lists
Top Programming Languages
Top Projects

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Corpus (5,353
Tokenizer (1,291
Language Model (1,034