Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Sense2vec | 1,486 |  | 6 | 7 | a year ago | 24 | April 19, 2021 | 20 | mit | Python |
| 🦆 Contextually-keyed word vectors | ||||||||||
| Stringi | 292 | 4 months ago | 42 | other | C++ | |||||
| Fast and portable character string processing in R (with the Unicode ICU) | ||||||||||
| Tokenizer | 224 | | 15 | 5 | 9 months ago | 68 | January 11, 2023 | 2 | mit | C++ |
| Fast and customizable text tokenization library with BPE and SentencePiece support | ||||||||||
| Rouge 2.0 | 145 | 4 years ago | 1 | apache-2.0 | Java | |||||
| ROUGE automatic summarization evaluation toolkit. Support for ROUGE-[N, L, S, SU], stemming and stopwords in different languages, unicode text evaluation, CSV output. | ||||||||||
| Guide To Swift Strings Sample Code | 124 | 5 years ago | 1 | Swift | ||||||
| Xcode Playground Sample Code for the Flight School Guide to Swift Strings | ||||||||||
| Unihandecode | 71 | 2 years ago | 17 | July 23, 2020 | 1 | gpl-3.0 | Python | |||
| unihandecode is a transliteration library to convert all characters/words in Unicode into ASCII alphabet that aware with Language preference priorities | ||||||||||
| Bengali Alphabet | 51 | 5 months ago | 1 | mit | JavaScript | |||||
| ✍️ Bengali alphabet (বাংলা বর্ণমালা) | ||||||||||
| Uax29 | 35 | 6 | 6 months ago | 40 | May 26, 2023 | 1 | mit | Go | ||
| A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and graphemes. | ||||||||||
| Stringx | 25 | 5 months ago | 9 | other | HTML | |||||
| Drop-in replacements for base R string functions powered by stringi | ||||||||||
| Urdu Characters | 18 | 3 years ago | mit | Python | ||||||
| 📄 Complete collection of Urdu language characters & unicode code points. | ||||||||||
Alternatives To TokenizerSelect To Compare
Alternative Project Comparisons
Popular Natural Language Processing Projects
Popular Unicode Projects
Popular Machine Learning Categories
Related Searches
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
C Plus Plus
Natural Language Processing
Unicode
Tokenizer
Machine Translation
Icu