Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Tokenizer Open Source Projects

Open source projects categorized as Tokenizer
Categories > Compilers > Tokenizer
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34most recent commit 3 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85most recent commit 3 months agopypi tokenizers} Downloads
Gpt2 Chinese ⭐ 7,249
Chinese version of GPT2 training code, using BERT tokenizer.
most recent commit 3 months ago
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8most recent commit 4 months agopackagist theseer/tokenizer} Downloads
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141most recent commit 3 months agonpm file-type} Downloads
Tntsearch ⭐ 3,004
A fully featured full text search engine written in PHP
dependent packages 27total releases 63most recent commit 3 months agopackagist teamtnt/tntsearch} Downloads
Chevrotain ⭐ 2,381
Parser Building Toolkit for JavaScript
dependent packages 272total releases 170most recent commit 7 days agonpm chevrotain} Downloads
Text ⭐ 1,172
Making text a first-class citizen in TensorFlow.
dependent packages 134total releases 67most recent commit 3 months agopypi tensorflow-text} Downloads
Hazm ⭐ 1,096
Persian NLP Toolkit
dependent packages 13total releases 20most recent commit a day agopypi hazm} Downloads
Natasha ⭐ 1,085
Solves basic Russian NLP tasks, API for lower level Natasha projects
dependent packages 9total releases 19most recent commit 7 months agopypi natasha} Downloads
Kobert ⭐ 1,035
Korean BERT pre-trained cased (KoBERT)
most recent commit a year ago
Nlp With Ruby ⭐ 1,002
Curated List: Practical Natural Language Processing done in Ruby
most recent commit 10 months ago
Autophrase ⭐ 978
AutoPhrase: Automated Phrase Mining from Massive Text Corpora
total releases 3most recent commit 2 years ago
Superpower ⭐ 848
A C# parser construction toolkit with high-quality error reporting
dependent packages 44total releases 99most recent commit 2 years agonuget Superpower} Downloads
Soynlp ⭐ 801
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
dependent packages 9total releases 33most recent commit 2 years agopypi soynlp} Downloads
Kagome ⭐ 769
Self-contained Japanese Morphological Analyzer written in pure Go
dependent packages 27total releases 74most recent commit 3 months ago
Moo ⭐ 763
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
dependent packages 263total releases 15most recent commit a year agonpm moo} Downloads
Libinjection ⭐ 759
SQL / SQLI tokenizer parser analyzer
total releases 27most recent commit 3 years ago
Parsimmon ⭐ 714
Parsimmon is a wee linguistics toolkit for iOS written in Swift.
total releases 10most recent commit 7 years ago
Elasticsearch Inquisitor ⭐ 712
Site plugin for Elasticsearch to help understand and debug queries.
most recent commit 6 years ago
Mustard ⭐ 686
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
most recent commit 6 years ago
Goro ⭐ 673
PHP in Go
total releases 1most recent commit 3 months ago
The 10 Most Depended On Tokenizer Open Source Projects
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141latest release November 11, 2023most recent commit 3 months agonpm file-type} Downloads
Js Tokens ⭐ 410
Tiny JavaScript tokenizer.
dependent packages 1,870total releases 21latest release September 09, 2023most recent commit 6 months agonpm js-tokens} Downloads
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 3 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 3 months agopypi tokenizers} Downloads
Refmt ⭐ 45
Object mapping for golang.
dependent packages 315total releases 11latest release December 10, 2022most recent commit 5 months ago
Chevrotain ⭐ 2,381
Parser Building Toolkit for JavaScript
dependent packages 272total releases 170latest release August 14, 2023most recent commit 7 days agonpm chevrotain} Downloads
Moo ⭐ 763
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
dependent packages 263total releases 15latest release October 03, 2022most recent commit a year agonpm moo} Downloads
Css Selector Tokenizer ⭐ 78
Parses and stringifies CSS selectors.
dependent packages 244total releases 18latest release June 01, 2021most recent commit 3 years agonpm css-selector-tokenizer} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
Sacremoses ⭐ 476
Python port of Moses tokenizer, truecaser and normalizer
dependent packages 164total releases 53latest release October 30, 2023most recent commit 6 months agopypi sacremoses} Downloads
The 10 Latest Releases In Tokenizer Open Source Projects
Dt Sql Parser ⭐ 228
SQL Parsers for BigData, built with antlr4.
dependent packages 9total releases 50latest release December 13, 2023most recent commit 3 months agonpm dt-sql-parser} Downloads
Php Rtflex ⭐ 8
A pure PHP lexer / tokenizer for RTF files
latest release December 03, 2023most recent commit 5 years ago
Tokenizer ⭐ 8
Tokenizer package as used by Hippo.
dependent packages 1latest release December 02, 2023most recent commit 7 years agopackagist hippophp/tokenizer} Downloads
Cereja ⭐ 21
Cereja is a bundle of useful functions we don't want to rewrite and .. just pure fun!
dependent packages 4total releases 125latest release December 01, 2023most recent commit 4 months agopypi cereja} Downloads
Gretel Synthetics ⭐ 490
Synthetic data generators for structured and unstructured text, featuring differentially private learning.
dependent packages 1total releases 53latest release November 28, 2023most recent commit 3 months agopypi gretel-synthetics} Downloads
Dadmatools ⭐ 142
DadmaTools is a Persian NLP tools developed by Dadmatech Co.
total releases 66latest release November 25, 2023most recent commit 3 months ago
Zearch ⭐ 9
ragecoded code search with json endpoint
latest release November 23, 2023most recent commit 6 years ago
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8latest release November 20, 2023most recent commit 4 months agopackagist theseer/tokenizer} Downloads
Text ⭐ 1,172
Making text a first-class citizen in TensorFlow.
dependent packages 134total releases 67latest release November 15, 2023most recent commit 3 months agopypi tensorflow-text} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 3 months agopypi tokenizers} Downloads
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Jit
Ghc
Ida
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.