Awesome Open Source
Awesome Open Source
Combined Topics
tokenizer x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 361 Tokenizer Open Source Projects on Github

Categories > Compilers > Tokenizer
Tokenizer ⭐ 4,702
A small library for converting tokenized PHP source code into XML (and potentially other formats)
Chevrotain ⭐ 1,782
Parser Building Toolkit for JavaScript
Natasha ⭐ 846
Solves basic Russian NLP tasks, API for lower level Natasha projects
Mustard ⭐ 686
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
Soynlp ⭐ 615
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
Kagome ⭐ 590
Self-contained Japanese Morphological Analyzer written in pure Go
Moo ⭐ 510
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
Open Korean Text ⭐ 435
Open Korean Text Processor - An Open-source Korean Text Processor
Wordless ⭐ 424
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
Php Parser ⭐ 423
🌿 NodeJS PHP Parser - extract AST or tokens (PHP5 and PHP7)
Ekphrasis ⭐ 422
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
Jflex ⭐ 415
The fast scanner generator for Java™ with full Unicode support
Sacremoses ⭐ 349
Python port of Moses tokenizer, truecaser and normalizer
Friso ⭐ 331
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
Smoothnlp ⭐ 320
专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference
Sentences ⭐ 301
A multilingual command line sentence tokenizer in Golang
Lexmachine ⭐ 290
Lex machinary for go.
Js Tokens ⭐ 272
Tiny JavaScript tokenizer.
Jumanpp ⭐ 266
Juman++ (a Morphological Analyzer Toolkit)
Bitextor ⭐ 197
Bitextor generates translation memories from multilingual websites
Vscode Blockman ⭐ 192
VSCode extension to highlight nested code blocks
Fugashi ⭐ 175
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
Udpipe ⭐ 175
R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit
Tokenizers ⭐ 164
Fast, Consistent Tokenization of Natural Language Text
Tokenizer ⭐ 161
Fast and customizable text tokenization library with BPE and SentencePiece support
Query Translator ⭐ 161
Query Translator is a search query translator with AST representation
Lex ⭐ 142
Replaced by foonathan/lexy
Text2text ⭐ 140
Text2Text: Multilingual tokenization, embedding, search, translation, summarization, question generation, question answering, data augmentation, distance measurement
Works For Me ⭐ 129
Collection of developer toolkits
Tokenizer ⭐ 129
Source code tokenizer
Lindera ⭐ 123
A morphological analysis library.
Microtokenizer ⭐ 119
一个微型&算法全面的中文分词引擎 | A micro tokenizer for Chinese
Peast ⭐ 116
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
Rust Tokenizers ⭐ 102
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
Japanesetokenizers ⭐ 101
aim to use JapaneseTokenizer as easy as possible
Syntok ⭐ 101
Text tokenization and sentence segmentation (segtok v2)
Jison ⭐ 97
bison / YACC / LEX in JavaScript (LALR(1), SLR(1), etc. lexer/parser generator)
Somajo ⭐ 96
A tokenizer and sentence splitter for German and English web and social media texts.
Megamark ⭐ 96
😻 Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer
Sentence Splitter ⭐ 86
Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
Djurl ⭐ 86
Simple yet helpful library for writing Django urls by an easy, short and intuitive way.
Hunspell ⭐ 86
High-Performance Stemmer, Tokenizer, and Spell Checker for R
Jargon ⭐ 85
Tokenizers and lemmatizers for Go
Hippo ⭐ 82
PHP standards checker.
Simple ⭐ 75
支持中文（简体和繁体）和拼音的 SQLite fts5 扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
Rustfst ⭐ 75
Rust re-implementation of OpenFST - library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs).
Cols Agent Tasks ⭐ 72
Colin's ALM Corner Custom Build Tasks
Wirb ⭐ 69
Ruby Object Inspection for IRB
Neural_tokenizer ⭐ 64
Tokenize English sentences using neural networks.
Tokenize.jl ⭐ 64
Tokenization for Julia source code
Vaporetto ⭐ 64
🛥 Vaporetto: a fast and lightweight pointwise prediction based tokenizer
Ebnfparser ⭐ 56
Convenient parser generator for Python(check out https://github.com/thautwarm/RBNF for an advanced version).
String Calc ⭐ 54
PHP calculator library for mathematical terms (expressions) passed as strings
Greynir ⭐ 53
The greynir.is natural language processing website for Icelandic
Chiffon ⭐ 52
A small ECMAScript parser, tokenizer and minifier written in JavaScript.
Thot ⭐ 52
Thot toolkit for statistical machine translation
Html5gum ⭐ 49
A WHATWG-compliant HTML5 tokenizer and tag soup parser
Grasp ⭐ 49
Essential NLP & ML, short & fast pure Python code
Tokenizer ⭐ 46
Tokenize CSS according to the CSS Syntax
Farasapy ⭐ 43
A Python implementation of Farasa toolkit
Py Nltools ⭐ 42
A collection of basic python modules for spoken natural language processing
Tokenizer ⭐ 42
A simple tokenizer in Ruby for NLP tasks.
Bredon ⭐ 40
A modern CSS value compiler in JavaScript
Cang Jie ⭐ 40
Chinese tokenizer for tantivy, based on jieba-rs
Talismane ⭐ 40
NLP framework: sentence detector, tokeniser, pos-tagger and dependency parser
Roy_vntokenizer ⭐ 40
Vietnamese tokenizer (Maximum Matching and CRF)
Wink Tokenizer ⭐ 39
Multilingual tokenizer that automatically tags each token with its type
Chinese Tokenizer ⭐ 39
Tokenizes Chinese texts into words.
Refmt ⭐ 38
Object mapping for golang.
Lex ⭐ 38
Lex is an implementation of lex tool in Ruby.
Sharpmath ⭐ 36
A small .NET math library.
Alm ⭐ 33
Smart Language Model
Kortok ⭐ 33
The code and models for "An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks" (AACL-IJCNLP 2020)
Android Sqlite Fts5 Tokenizer ⭐ 33
集成了FTS5中文分词器的Sqlite3源码
Psr2r Sniffer ⭐ 32
A PSR-2-R code sniffer and code-style auto-correction-tool - including many useful additions
Kagome.ipadic ⭐ 30
Graphql Query Compress ⭐ 30
Compress GraphQL Query String
Nlp Js Tools French ⭐ 29
POS Tagger, lemmatizer and stemmer for french language in javascript
Omnicat Bayes ⭐ 29
Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)
Lfuzzer ⭐ 28
Fuzzing Parsers with Tokens
Tif ⭐ 28
Text Interchange Formats
Python Mecab ⭐ 27
A repository to bind mecab for Python 3.5+. Not using swig nor pybind. (Not Maintained Now)
Lindera Tantivy ⭐ 27
A tokenizer for Tantivy, based on Lindera.
Python Ucto ⭐ 27
This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).
Sinling ⭐ 26
A collection of NLP tools for Sinhalese (සිංහල).
Gd Tokenizer ⭐ 26
A small godot project with a tokenizer written in GDScript.
Python Vncorenlp ⭐ 26
A Python wrapper for VnCoreNLP using a bidirectional communication channel.
Tok Tok ⭐ 26
A fast, simple, multilingual tokenizer
Botok ⭐ 26
🏷 བོད་ཏོག [pʰøtɔk̚] Tibetan word tokenizer in Python
Tokenizer ⭐ 25
A tokenizer for Icelandic text
Elasticsearch Plugins ⭐ 25
Some native scoring script plugins for elasticsearch
Swilex ⭐ 24
A universal lexer library in Swift.
Lisp Esque Language ⭐ 24
💠The Lel programming language
Pck ⭐ 24
The Parser Construction Kit ("Puck"): A Parser Generator and Grammar Translator in C#
Suika ⭐ 24
Suika 🍉 is a Japanese morphological analyzer written in pure Ruby
Jpophp ⭐ 23
JSON Parser Object PHP is a library for parsing the data in JSON format.
Loretta ⭐ 23
A C# (G)Lua lexer, parser, code analysis, transformation and generation library.
Toy_lang ⭐ 21
The first language I made.
Mystem Scala ⭐ 21
Morphological analyzer `mystem` wrapper for JVM languages
Text Classification Lstms Pytorch ⭐ 20
The aim of this repository is to show a baseline model for text classification by implementing a LSTM-based model coded in PyTorch. In order to provide a better understanding of the model, it will be used a Tweets dataset provided by Kaggle.
1-100 of 361 projects

Related Projects

Python Tokenizer Projects (96)
Nlp Tokenizer Projects (82)
Parser Tokenizer Projects (80)
Lexer Tokenizer Projects (62)
Javascript Tokenizer Projects (53)
Python Nlp Tokenizer Projects (39)
Natural Language Processing Tokenizer Projects (36)
Java Tokenizer Projects (35)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
Privacy policy
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2021