Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Tokenizer Open Source Projects

Open source projects categorized as Tokenizer
Categories > Compilers > Tokenizer
Sentencepiece ⭐ 8,211
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 581total releases 34most recent commit 13 days agopypi sentencepiece} Downloads
Tokenizers ⭐ 7,583
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 244total releases 76most recent commit 10 days agopypi tokenizers} Downloads
Gpt2 Chinese ⭐ 6,983
Chinese version of GPT2 training code, using BERT tokenizer.
most recent commit 4 months ago
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 186total releases 36most recent commit 2 years agopackagist phpunit/php-token-stream} Downloads
Tokenizer ⭐ 5,069
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 7most recent commit a month agopackagist theseer/tokenizer} Downloads
File Type ⭐ 3,229
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,724total releases 139most recent commit 25 days agonpm file-type} Downloads
Tntsearch ⭐ 2,968
A fully featured full text search engine written in PHP
dependent packages 27total releases 63most recent commit a month agopackagist teamtnt/tntsearch} Downloads
Chevrotain ⭐ 2,249
Parser Building Toolkit for JavaScript
dependent packages 233total releases 170most recent commit 3 days agonpm chevrotain} Downloads
Blingfire ⭐ 1,702
A lightning fast Finite State machine and REgular expression manipulation library.
dependent packages 6total releases 19most recent commit 3 months agopypi blingfire} Downloads
Text ⭐ 1,143
Making text a first-class citizen in TensorFlow.
dependent packages 112total releases 64most recent commit 19 days agopypi tensorflow-text} Downloads
Natasha ⭐ 1,085
Solves basic Russian NLP tasks, API for lower level Natasha projects
dependent packages 9total releases 19most recent commit 2 days agopypi natasha} Downloads
Kobert ⭐ 1,035
Korean BERT pre-trained cased (KoBERT)
most recent commit 8 months ago
Nlp With Ruby ⭐ 1,002
Curated List: Practical Natural Language Processing done in Ruby
most recent commit 3 months ago
Autophrase ⭐ 978
AutoPhrase: Automated Phrase Mining from Massive Text Corpora
total releases 3most recent commit 2 years ago
Superpower ⭐ 848
A C# parser construction toolkit with high-quality error reporting
dependent packages 31total releases 99most recent commit a year agonuget Superpower} Downloads
Soynlp ⭐ 801
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
dependent packages 5total releases 33most recent commit a year agopypi soynlp} Downloads
Libinjection ⭐ 759
SQL / SQLI tokenizer parser analyzer
total releases 27most recent commit 3 years ago
Kagome ⭐ 753
Self-contained Japanese Morphological Analyzer written in pure Go
dependent packages 17total releases 67most recent commit 4 days ago
Moo ⭐ 749
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
dependent packages 173total releases 14most recent commit 4 months agonpm moo} Downloads
Parsimmon ⭐ 714
Parsimmon is a wee linguistics toolkit for iOS written in Swift.
total releases 10most recent commit 7 years ago
Elasticsearch Inquisitor ⭐ 712
Site plugin for Elasticsearch to help understand and debug queries.
most recent commit 6 years ago
Mustard ⭐ 686
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
most recent commit 5 years ago
React Typeahead ⭐ 660
Pure react-based typeahead and typeahead-tokenizer
dependent packages 14total releases 28most recent commit 3 years agonpm react-typeahead} Downloads
The 10 Most Depended On Tokenizer Open Source Projects
File Type ⭐ 3,229
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,724total releases 139latest release June 04, 2023most recent commit 25 days agonpm file-type} Downloads
Js Tokens ⭐ 406
Tiny JavaScript tokenizer.
dependent packages 702total releases 19latest release May 15, 2022most recent commit 22 days agonpm js-tokens} Downloads
Sentencepiece ⭐ 8,211
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 581total releases 34latest release May 02, 2023most recent commit 13 days agopypi sentencepiece} Downloads
Tokenizers ⭐ 7,583
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 244total releases 76latest release April 05, 2023most recent commit 10 days agopypi tokenizers} Downloads
Chevrotain ⭐ 2,249
Parser Building Toolkit for JavaScript
dependent packages 233total releases 170latest release August 14, 2023most recent commit 3 days agonpm chevrotain} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 186total releases 36latest release November 30, 2020most recent commit 2 years agopackagist phpunit/php-token-stream} Downloads
Moo ⭐ 749
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
dependent packages 173total releases 14latest release September 29, 2019most recent commit 4 months agonpm moo} Downloads
Text ⭐ 1,143
Making text a first-class citizen in TensorFlow.
dependent packages 112total releases 64latest release August 18, 2023most recent commit 19 days agopypi tensorflow-text} Downloads
Php Parser ⭐ 500
:herb: NodeJS PHP Parser - extract AST or tokens
dependent packages 73total releases 70latest release September 13, 2022most recent commit 20 days agonpm php-parser} Downloads
Sentences ⭐ 378
A multilingual command line sentence tokenizer in Golang
dependent packages 71total releases 7latest release May 26, 2021most recent commit a month ago
The 10 Latest Releases In Tokenizer Open Source Projects
Text ⭐ 1,143
Making text a first-class citizen in TensorFlow.
dependent packages 112total releases 64latest release August 18, 2023most recent commit 19 days agopypi tensorflow-text} Downloads
Chevrotain ⭐ 2,249
Parser Building Toolkit for JavaScript
dependent packages 233total releases 170latest release August 14, 2023most recent commit 3 days agonpm chevrotain} Downloads
Natasha ⭐ 1,085
Solves basic Russian NLP tasks, API for lower level Natasha projects
dependent packages 9total releases 19latest release July 24, 2023most recent commit 2 days agopypi natasha} Downloads
Tntsearch ⭐ 2,968
A fully featured full text search engine written in PHP
dependent packages 27total releases 63latest release July 19, 2023most recent commit a month agopackagist teamtnt/tntsearch} Downloads
File Type ⭐ 3,229
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,724total releases 139latest release June 04, 2023most recent commit 25 days agonpm file-type} Downloads
Sentencepiece ⭐ 8,211
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 581total releases 34latest release May 02, 2023most recent commit 13 days agopypi sentencepiece} Downloads
Tokenizers ⭐ 7,583
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 244total releases 76latest release April 05, 2023most recent commit 10 days agopypi tokenizers} Downloads
Peast ⭐ 154
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
dependent packages 6total releases 36latest release January 21, 2023most recent commit a month agopackagist mck89/peast} Downloads
Dt Sql Parser ⭐ 193
SQL Parsers for BigData, built with antlr4.
dependent packages 5total releases 39latest release January 09, 2023most recent commit 4 days agonpm dt-sql-parser} Downloads
Lindera ⭐ 298
A morphological analysis library.
dependent packages 17total releases 40latest release December 27, 2022most recent commit 19 days agocargo lindera-core} Downloads
Popular Compilers Categories
Gcc
Abstract Syntax Tree
Llvm
Clang
Lexer
Wsl
Emscripten
Ghc
Tokenizer
Linker
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2023 Awesome Open Source.  All rights reserved.