Awesome Open Source

Programming Languages

Search results for language model tokenizer

language-model x

15 search results found

Tokenizers ⭐ 8,056

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Kobert ⭐ 1,035

Korean BERT pre-trained cased (KoBERT)

Ckip Transformers ⭐ 439

CKIP Transformers

Korean GPT-2 pretrained cased (KoGPT2)

Transformer Lm ⭐ 155

Transformer language model (GPT-2) with sentencepiece tokenizer

package lingo provides the data structures and algorithms required for natural language processing

Vietnamese Electra ⭐ 59

Electra pre-trained model using Vietnamese corpus

Fastai Projects ⭐ 55

Jupyter notebooks that use the Fastai library

Smart Language Model

Transformers Embedder ⭐ 34

A Word Level Transformer layer based on PyTorch and 🤗 Transformers.

HerBERT is a BERT-based Language Model trained on Polish Corpora using only MLM objective with dynamic masking of whole words.

Nim Tokenizer ⭐ 18

Implementation of a simple BPE tokenizer, but in Nim

German small and large versions of GPT2.

Thailmcut ⭐ 15

Bytepiece Rs ⭐ 12

The Bytepiece Tokenizer Implemented in Rust.

Related Searches

Python Language Model (540)

Python Tokenizer (341)

1-15 of 15 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.