Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

Tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
Stars
399
License
mit
Open Issues
6
Most Recent Commit
6 months ago
Programming Language
Go
Total Releases
15
Latest Release
September 05, 2023
Categories
Repo
Alternatives To Tokenmonster
Select To Compare
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34most recent commit 3 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85most recent commit 3 months agopypi tokenizers} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141most recent commit 4 months agonpm file-type} Downloads
Tntsearch ⭐ 3,004
A fully featured full text search engine written in PHP
dependent packages 27total releases 63most recent commit 4 months agopackagist teamtnt/tntsearch} Downloads
Chevrotain ⭐ 2,381
Parser Building Toolkit for JavaScript
dependent packages 272total releases 170most recent commit 22 days agonpm chevrotain} Downloads
Text ⭐ 1,172
Making text a first-class citizen in TensorFlow.
dependent packages 134total releases 67most recent commit 4 months agopypi tensorflow-text} Downloads
Hazm ⭐ 1,107
Persian NLP Toolkit
dependent packages 13total releases 20most recent commit 16 days agopypi hazm} Downloads


Suggest An Alternative To tokenmonster
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Jit
Ghc
Ida
Related Searches
Golang Tokenizer
Tokenizer Tokenization
Golang Tokenization
Tokenizer Tokenize
Golang Vocabulary
Tokenization Tokenize

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Golang
Tokenizer
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.