Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Tokenizer Open Source Projects

Open source projects categorized as Tokenizer
Categories > Compilers > Tokenizer
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85most recent commit a year agopypi tokenizers} Downloads
Gpt2 Chinese ⭐ 7,249
Chinese version of GPT2 training code, using BERT tokenizer.
most recent commit 2 years ago
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36most recent commit 4 years agopackagist phpunit/php-token-stream} Downloads
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8most recent commit 2 years agopackagist theseer/tokenizer} Downloads
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141most recent commit 2 years agonpm file-type} Downloads
Tntsearch ⭐ 3,004
A fully featured full text search engine written in PHP
dependent packages 27total releases 63most recent commit 2 years agopackagist teamtnt/tntsearch} Downloads
Hazm ⭐ 1,296
Persian NLP Toolkit
dependent packages 13total releases 20most recent commit a year agopypi hazm} Downloads
Kobert ⭐ 1,035
Korean BERT pre-trained cased (KoBERT)
most recent commit 2 years ago
Nlp With Ruby ⭐ 1,002
Curated List: Practical Natural Language Processing done in Ruby
most recent commit 2 years ago
Autophrase ⭐ 978
AutoPhrase: Automated Phrase Mining from Massive Text Corpora
total releases 3most recent commit 3 years ago
Moo ⭐ 850
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
dependent packages 263total releases 15most recent commit 2 years agonpm moo} Downloads
Superpower ⭐ 848
A C# parser construction toolkit with high-quality error reporting
dependent packages 44total releases 99most recent commit 3 years agonuget Superpower} Downloads
Soynlp ⭐ 801
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
dependent packages 9total releases 33most recent commit 3 years agopypi soynlp} Downloads
Kagome ⭐ 769
Self-contained Japanese Morphological Analyzer written in pure Go
dependent packages 27total releases 74most recent commit a year ago
Parsimmon ⭐ 714
Parsimmon is a wee linguistics toolkit for iOS written in Swift.
total releases 10most recent commit 8 years ago
Mustard ⭐ 686
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
most recent commit 7 years ago
React Typeahead ⭐ 660
Pure react-based typeahead and typeahead-tokenizer
dependent packages 17total releases 28most recent commit 5 years agonpm react-typeahead} Downloads
Wordless ⭐ 649
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
most recent commit 2 years ago
Ru_transformers ⭐ 627
most recent commit 5 years ago
Laracms ⭐ 598
LaraCMS 是在学习 laravel （ web 开发实战进阶 + 实战构架 API 服务器） 过程中产生的一个业余作品，试图通过简单的方式，快速构建一套基本的企业站同时保留很灵活的扩展能力和优雅 也是一个学习Laravel 不错的参考示例。
total releases 4most recent commit 2 years ago
Php Parser ⭐ 500
🌿 NodeJS PHP Parser - extract AST or tokens
dependent packages 100total releases 74most recent commit 2 years agonpm php-parser} Downloads
Sacremoses ⭐ 495
Python port of Moses tokenizer, truecaser and normalizer
dependent packages 164total releases 53most recent commit a year agopypi sacremoses} Downloads
Gretel Synthetics ⭐ 490
Synthetic data generators for structured and unstructured text, featuring differentially private learning.
dependent packages 1total releases 53most recent commit 2 years agopypi gretel-synthetics} Downloads
The 10 Most Depended On Tokenizer Open Source Projects
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141latest release November 11, 2023most recent commit 2 years agonpm file-type} Downloads
Js Tokens ⭐ 410
Tiny JavaScript tokenizer.
dependent packages 1,870total releases 21latest release September 09, 2023most recent commit 2 years agonpm js-tokens} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit a year agopypi tokenizers} Downloads
Refmt ⭐ 45
Object mapping for golang.
dependent packages 315total releases 11latest release December 10, 2022most recent commit 2 years ago
Moo ⭐ 850
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
dependent packages 263total releases 15latest release October 03, 2022most recent commit 2 years agonpm moo} Downloads
Css Selector Tokenizer ⭐ 78
Parses and stringifies CSS selectors.
dependent packages 244total releases 18latest release June 01, 2021most recent commit 4 years agonpm css-selector-tokenizer} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 4 years agopackagist phpunit/php-token-stream} Downloads
Sacremoses ⭐ 495
Python port of Moses tokenizer, truecaser and normalizer
dependent packages 164total releases 53latest release October 30, 2023most recent commit a year agopypi sacremoses} Downloads
Php Parser ⭐ 500
:herb: NodeJS PHP Parser - extract AST or tokens
dependent packages 100total releases 74latest release June 07, 2023most recent commit 2 years agonpm php-parser} Downloads
Tokenizer ⭐ 138
[DISCONTINUED] Source code tokenizer
dependent packages 97total releases 9latest release February 09, 2022most recent commit 3 years agopackagist nette/tokenizer} Downloads
The 10 Latest Releases In Tokenizer Open Source Projects
Dt Sql Parser ⭐ 228
SQL Parsers for BigData, built with antlr4.
dependent packages 9total releases 50latest release December 13, 2023most recent commit 2 years agonpm dt-sql-parser} Downloads
Tokenizer ⭐ 8
Tokenizer package as used by Hippo.
dependent packages 1latest release December 02, 2023most recent commit 8 years agopackagist hippophp/tokenizer} Downloads
Cereja ⭐ 21
Cereja is a bundle of useful functions we don't want to rewrite and .. just pure fun!
dependent packages 4total releases 125latest release December 01, 2023most recent commit 2 years agopypi cereja} Downloads
Gretel Synthetics ⭐ 490
Synthetic data generators for structured and unstructured text, featuring differentially private learning.
dependent packages 1total releases 53latest release November 28, 2023most recent commit 2 years agopypi gretel-synthetics} Downloads
Dadmatools ⭐ 142
DadmaTools is a Persian NLP tools developed by Dadmatech Co.
total releases 66latest release November 25, 2023most recent commit 2 years ago
Zearch ⭐ 9
ragecoded code search with json endpoint
latest release November 23, 2023most recent commit 8 years ago
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8latest release November 20, 2023most recent commit 2 years agopackagist theseer/tokenizer} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit a year agopypi tokenizers} Downloads
Tiktoken Rs ⭐ 163
Ready-made tokenizer library for working with GPT and tiktoken
dependent packages 24total releases 26latest release November 14, 2023most recent commit 2 years agocargo tiktoken-rs} Downloads
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141latest release November 11, 2023most recent commit 2 years agonpm file-type} Downloads
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Tokenizer
Dex
Jit
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2025 Awesome Open Source.  All rights reserved.