The Top 1,301 Tokenizer Open Source Projects

Awesome Open Source

Share On Twitter

Combined Topics

tokenizer x

The Top 1,301 Tokenizer Open Source Projects

Categories > Compilers > Tokenizer

Php Token Stream ⭐ 6,457

Wrapper around PHP's tokenizer extension.

dependent packages 186total releases 36most recent commit a year ago

packagist phpunit/php-token-stream} Downloads

Gpt Neo ⭐ 6,160

An implementation of model parallel GPT-2 and GPT-3-style models using the mesh-tensorflow library.

most recent commit 6 months ago

Sentencepiece ⭐ 6,111

Unsupervised text tokenizer for Neural Network-based text generation.

dependent packages 268total releases 31most recent commit 14 hours ago

Tokenizers ⭐ 5,804

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

dependent packages 117total releases 71most recent commit 7 days ago

Tokenizer ⭐ 4,863

A small library for converting tokenized PHP source code into XML (and potentially other formats)

dependent packages 11total releases 7most recent commit 2 months ago

Gpt2 Chinese ⭐ 4,794

Chinese version of GPT2 training code, using BERT tokenizer.

most recent commit 5 months ago

Tntsearch ⭐ 2,825

A fully featured full text search engine written in PHP

dependent packages 27total releases 52most recent commit 13 days ago

File Type ⭐ 2,643

Detect the file type of a Buffer/Uint8Array/ArrayBuffer

dependent packages 1,086total releases 127most recent commit 12 days ago

Chevrotain ⭐ 1,973

Parser Building Toolkit for JavaScript

dependent packages 136total releases 160most recent commit a day ago

Blingfire ⭐ 1,636

A lightning fast Finite State machine and REgular expression manipulation library.

dependent packages 2total releases 19most recent commit a month ago

Autophrase ⭐ 978

AutoPhrase: Automated Phrase Mining from Massive Text Corpora

total releases 3most recent commit 7 months ago

Text ⭐ 975

Making text a first-class citizen in TensorFlow.

dependent packages 81total releases 54most recent commit 3 days ago

Kobert ⭐ 930

Korean BERT pre-trained cased (KoBERT)

most recent commit 5 days ago

Nlp With Ruby ⭐ 910

Curated List: Practical Natural Language Processing done in Ruby

most recent commit a year ago

Superpower ⭐ 848

A C# parser construction toolkit with high-quality error reporting

dependent packages 31total releases 99most recent commit 2 months ago

Natasha ⭐ 846

Solves basic Russian NLP tasks, API for lower level Natasha projects

dependent packages 3total releases 17most recent commit 10 months ago

Libinjection ⭐ 759

SQL / SQLI tokenizer parser analyzer

total releases 27most recent commit a year ago

Parsimmon ⭐ 714

Parsimmon is a wee linguistics toolkit for iOS written in Swift.

total releases 10most recent commit 5 years ago

Elasticsearch Inquisitor ⭐ 712

Site plugin for Elasticsearch to help understand and debug queries.

most recent commit 5 years ago

Mustard ⭐ 686

🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.

most recent commit 4 years ago

Kagome ⭐ 660

Self-contained Japanese Morphological Analyzer written in pure Go

dependent packages 17total releases 67most recent commit a month ago

React Typeahead ⭐ 660

Pure react-based typeahead and typeahead-tokenizer

dependent packages 14total releases 28most recent commit 2 years ago

Goro ⭐ 658

PHP in Go

most recent commit 4 years ago

Moo ⭐ 639

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

dependent packages 173total releases 14most recent commit 3 months ago

Ru_transformers ⭐ 627

most recent commit 2 years ago

Soynlp ⭐ 615

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

dependent packages 5total releases 33most recent commit a year ago

Ekphrasis ⭐ 554

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

total releases 54most recent commit 3 months ago

Laracms ⭐ 539

LaraCMS 是在学习 laravel （ web 开发实战进阶 + 实战构架 API 服务器）过程中产生的一个业余作品，试图通过简单的方式，快速构建一套基本的企业站同时保留很灵活的扩展能力和优雅也是一个学习Laravel 不错的参考示例。

total releases 4most recent commit 2 years ago

Wordless ⭐ 538

An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation

most recent commit 2 days ago

Smartcode ⭐ 508

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!

dependent packages 5total releases 31most recent commit 2 months ago

Parsekit ⭐ 503

Objective-C Tokenizer and Parser Generator. Supports Grammars.

total releases 5most recent commit 5 years ago

Bayes ⭐ 494

Naive-Bayes Classifier for node.js

dependent packages 8total releases 8most recent commit 2 years ago

Jflex ⭐ 468

The fast scanner generator for Java™ with full Unicode support

dependent packages 32total releases 12most recent commit 2 months ago

Php Text Analysis ⭐ 452

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

dependent packages 6total releases 39most recent commit 13 days ago

packagist yooper/php-text-analysis} Downloads

Php Parser ⭐ 451

🌿 NodeJS PHP Parser - extract AST or tokens

dependent packages 62total releases 68most recent commit 4 days ago

Cogcomp Nlp ⭐ 448

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.

most recent commit a month ago

Open Korean Text ⭐ 435

Open Korean Text Processor - An Open-source Korean Text Processor

dependent packages 6total releases 14most recent commit a year ago

Bert Japanese ⭐ 415

BERT with SentencePiece for Japanese text.

most recent commit 2 years ago

Sacremoses ⭐ 404

Python port of Moses tokenizer, truecaser and normalizer

dependent packages 68total releases 50most recent commit 19 days ago

Mmseg4j Solr ⭐ 403

mmseg4j for lucene or solr analyzer

dependent packages 2total releases 9most recent commit 3 years ago

Elasticsearch Analysis Vietnamese ⭐ 391

Vietnamese Analysis Plugin for Elasticsearch

most recent commit 3 months ago

Node Question Answering ⭐ 382

Fast and production-ready question answering in Node.js

total releases 16most recent commit 5 months ago

Lexmachine ⭐ 370

Lex machinary for go.

dependent packages 35total releases 6most recent commit a month ago

Spyglass ⭐ 362

A library for mentions on Android

total releases 16most recent commit 5 months ago

Php Short Array Syntax Converter ⭐ 352

Command-line script to convert PHP's array() syntax to PHP 5.4's short array syntax []

most recent commit 3 years ago

Structural Probes ⭐ 340

Codebase for testing whether hidden states of neural networks encode discrete structures.

most recent commit 3 months ago

Friso ⭐ 331

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

most recent commit a year ago

Sentences ⭐ 325

A multilingual command line sentence tokenizer in Golang

dependent packages 64total releases 7most recent commit 4 days ago

Smoothnlp ⭐ 320

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

total releases 21most recent commit 2 years ago

Kobart ⭐ 320

Korean BART

most recent commit 3 months ago

Deepcut ⭐ 319

A Thai word tokenization library using Deep Neural Network

dependent packages 3total releases 30most recent commit 2 years ago

Kldns ⭐ 316

快乐二级域名分发系统

most recent commit 6 months ago

Transformers.jl ⭐ 316

Julia Implementation of Transformer models

most recent commit 11 days ago

Nmt Chatbot ⭐ 309

NMT Chatbot

most recent commit 2 years ago

Js Tokens ⭐ 305

Tiny JavaScript tokenizer.

dependent packages 377total releases 19most recent commit 3 months ago

Ckip Transformers ⭐ 301

CKIP Transformers

most recent commit 5 days ago

Kogpt2 ⭐ 298

Korean GPT-2 pretrained cased (KoGPT2)

most recent commit 6 months ago

Elasticsearch Analysis Jieba ⭐ 296

The plugin includes the `jieba` analyzer, `jieba` tokenizer, and `jieba` token filter, and have two mode you can choose. one is `index` which means it will be used when you want to index a document. another is `search` mode which used when you want to search something.

most recent commit 5 years ago

Coccoc Tokenizer ⭐ 295

high performance tokenizer for Vietnamese language

most recent commit a year ago

Jumanpp ⭐ 291

Juman++ (a Morphological Analyzer Toolkit)

most recent commit 2 months ago

Sudachipy ⭐ 290

Python version of Sudachi, a Japanese tokenizer.

most recent commit 4 months ago

Gretel Synthetics ⭐ 287

Synthetic data generators for structured and unstructured text, featuring differentially private learning.

dependent packages 1total releases 44most recent commit a month ago

Melusine ⭐ 272

Melusine is a high-level library for emails classification and feature extraction "dédiée aux courriels français".

total releases 52most recent commit 2 months ago

Kcbert ⭐ 271

🤗 Pretrained BERT model & WordPiece tokenizer trained on Korean Comments 한국어 댓글로 프리트레이닝한 BERT 모델

most recent commit 10 months ago

Mail Parser ⭐ 263

Tokenizer for raw mails

dependent packages 13total releases 84most recent commit a year ago

Tsql Parser ⭐ 247

Library Written in C# For Parsing SQL Server T-SQL Scripts in .Net

dependent packages 10total releases 27most recent commit a month ago

Tabloid ⭐ 238

A minimal programming language inspired by clickbait headlines

most recent commit a year ago

Vscode Blockman ⭐ 228

VSCode extension to highlight nested code blocks

most recent commit 2 months ago

Opennlp ⭐ 221

Open source NLP tools (sentence splitter, tokenizer, chunker, coref, NER, parse trees, etc.) in C#

dependent packages 2total releases 27most recent commit 10 months ago

Bitextor ⭐ 221

Bitextor generates translation memories from multilingual websites

most recent commit 12 days ago

Text2text ⭐ 194

Text2Text: Cross-lingual natural language processing and generation toolkit

total releases 71most recent commit a month ago

Fugashi ⭐ 192

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

dependent packages 19total releases 62most recent commit 6 months ago

Udpipe ⭐ 191

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit

dependent packages 15total releases 21most recent commit 2 days ago

Tokenizer ⭐ 187

Fast and customizable text tokenization library with BPE and SentencePiece support

dependent packages 5total releases 62most recent commit 10 days ago

Mpmd ⭐ 181

Magento Project Mess Detector (for n98-magerun)

most recent commit 5 years ago

Konoha ⭐ 177

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.

dependent packages 1total releases 9most recent commit 21 days ago

Query Translator ⭐ 176

Query Translator is a search query translator with AST representation

dependent packages 3total releases 4most recent commit 7 months ago

packagist netgen/query-translator} Downloads

Lindera ⭐ 176

A morphological analysis library.

dependent packages 17total releases 34most recent commit a month ago

Elasticsearch Analysis Hao ⭐ 175

一个非常hao用的elasticsearch(es)中文分词器插件

most recent commit a month ago

Vaporetto ⭐ 171

🛥 Vaporetto: Very Accelerated POintwise pREdicTion based TOkenizer

dependent packages 2total releases 12most recent commit 2 days ago

Simple ⭐ 167

支持中文（简体和繁体）和拼音的 SQLite fts5 扩展｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin

most recent commit a month ago

Tokenizers ⭐ 164

Fast, Consistent Tokenization of Natural Language Text

dependent packages 19total releases 7most recent commit a year ago

Segmentit ⭐ 162

任何 JS 环境可用的中文分词包，fork from leizongmin/node-segment

dependent packages 3total releases 17most recent commit 8 months ago

Syntok ⭐ 158

Text tokenization and sentence segmentation (segtok v2)

dependent packages 8total releases 16most recent commit 5 months ago

Naive Bayes Classifier ⭐ 157

yet another general purpose naive bayesian classifier.

dependent packages 1total releases 4most recent commit 3 years ago

Vntk ⭐ 155

Vietnamese NLP Toolkit for Node

dependent packages 1total releases 15most recent commit 2 years ago

Transformer Lm ⭐ 155

Transformer language model (GPT-2) with sentencepiece tokenizer

most recent commit a year ago

Segtok ⭐ 151

Segtok v2 is here: https://github.com/fnl/syntok -- A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic features.

dependent packages 11total releases 23most recent commit 8 months ago

Nlc ⭐ 150

Neural Language Correction implemented on Tensorflow

most recent commit 6 years ago

Lex ⭐ 142

Replaced by foonathan/lexy

most recent commit 2 years ago

Ladmin ⭐ 137

EasyLms is a development management system with Laravel

most recent commit 5 years ago

Tokenizer ⭐ 134

[DISCONTINUED] Source code tokenizer

dependent packages 97total releases 9most recent commit 6 months ago

Vokenization ⭐ 133

PyTorch code for EMNLP 2020 Paper "Vokenization: Improving Language Understanding with Visual Supervision"

most recent commit a year ago

Shield ⭐ 131

Bayesian text classifier with flexible tokenizers and storage backends for Go

most recent commit 2 years ago

Works For Me ⭐ 129

Collection of developer toolkits

most recent commit 4 years ago

Ts Parsec ⭐ 128

Writing a custom parser is a fairly common need. Although there are already parser combinators in others languages, TypeScript provides a powerful and well-structured foundation for building this. Common parser combinators’ weakness are error handling and ambiguity resolving, but these are ts-parsec’s important features. Additionally, ts-parsec provides a very easy to use programming interface, that could help people to build programming-language-scale parsers in just a few hours. This technolog

dependent packages 2total releases 6most recent commit a year ago

Html5 El ⭐ 128

HTML5 and XHTML5 tools in Emacs Lisp

most recent commit 8 years ago

Peast ⭐ 126

JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification

dependent packages 6total releases 34most recent commit 3 months ago

Html5gum ⭐ 125

A WHATWG-compliant HTML5 tokenizer and tag soup parser

dependent packages 2total releases 9most recent commit a month ago

Coreblog ⭐ 124

一款优雅的博客系统

total releases 32most recent commit a year ago

1-100 of 1,301 projects

Next >