Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for bpe
bpe
x
21 search results found
Subword Nmt
⭐
1,937
Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
Youtokentome
⭐
943
Unsupervised text tokenizer focused on computational efficiency
Gpt Tokenizer
⭐
309
JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
Tokenizer
⭐
224
Fast and customizable text tokenization library with BPE and SentencePiece support
Nlp_made_easy
⭐
217
Explains nlp building blocks in a simple manner.
Python Bpe
⭐
178
Byte Pair Encoding for Python!
Tiktoken Rs
⭐
163
Ready-made tokenizer library for working with GPT and tiktoken
Nfelib
⭐
114
nfelib - bindings Python para e ler e gerir XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e
Go Gpt 3 Encoder
⭐
74
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
Subwordencoding Cws
⭐
43
Subword Encoding in Lattice LSTM for Chinese Word Segmentation
Youtokentome Ruby
⭐
20
High performance unsupervised text tokenization for Ruby
Piecelearn
⭐
18
Learning BPE embeddings by first learning a segmentation model and then training word2vec
Gpt3 Tokenizer
⭐
15
GPT3 encoder & decoder tool written in Swift
Tokenizers.bpe
⭐
13
R package for Byte Pair Encoding based on YouTokenToMe
Phishytics Machine Learning For Phishing
⭐
12
Machine Learning for Phishing Website Detection
Submrc
⭐
11
Subword-augmented Embedding for Cloze Reading Comprehension (COLING 2018)
Tokenization Scorer
⭐
10
Simple-to-use scoring function for arbitrarily tokenized texts.
Llmsharp Tokenizers
⭐
9
BPE Tokenizer implementations in C# for Anthropic, OpenAI LLM offerings
Vietnamese Stock Article Classification
⭐
5
Sentiment-based classification for stock article title using PhoBert
Nlcodec
⭐
5
Natural Language EnCoder-Decoder: word, char, bpe etc
Ktoken
⭐
5
Kotlin multiplatform BPE tokenizer library for OpenAI models
Related Searches
Python Bpe (12)
Tokenizer Bpe (11)
Bpe Byte Pair Encoding (8)
Natural Language Processing Bpe (7)
Tokenization Bpe (6)
Openai Bpe (5)
Word Segmentation Bpe (4)
Encoding Bpe (4)
Segmentation Bpe (4)
Gpt 2 Bpe (4)
1-21 of 21 search results
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.