Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

Tokenstream

A versatile token stream for handwritten parsers.
Stars
12
License
mit
Open Issues
5
Most Recent Commit
9 months ago
Programming Language
Python
Monthly Downloads
pypi tokenstream Downloads
Dependent Packages
5
Total Releases
41
Latest Release
August 03, 2023
Categories
Site
Repo
Alternatives To Tokenstream
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Chevrotain2,381npm chevrotain} Downloads70627210 days ago170August 14, 202350apache-2.0TypeScript
Parser Building Toolkit for JavaScript
Moo763npm moo} Downloads12,799263a year ago15October 03, 202231bsd-3-clauseJavaScript
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
Jflex52319464a year ago14March 11, 202325otherJava
The fast scanner generator for Java™ with full Unicode support
Php Parser500npm php-parser} Downloads3271007 months ago74June 07, 202334bsd-3-clauseJavaScript
:herb: NodeJS PHP Parser - extract AST or tokens
Lexmachine3701452 years ago6July 15, 20225otherGo
Lex machinary for go.
Lex142
3 years agobsl-1.0C++
Replaced by foonathan/lexy
Html5gum140cargo html5gum} Downloads24 months ago14July 26, 202313mitRust
A WHATWG-compliant HTML5 tokenizer and tag soup parser
Works For Me129
6 years agomitJavaScript
Collection of developer toolkits
Jison97npm jison-gho} Downloads500393 years ago39April 14, 2019n,ullmitJavaScript
bison / YACC / LEX in JavaScript (LALR(1), SLR(1), etc. lexer/parser generator)
Tokenize.jl97
3 months ago11otherJulia
Tokenization for Julia source code
Alternatives To Tokenstream
Select To Compare
Chevrotain ⭐ 2,381
Parser Building Toolkit for JavaScript
dependent packages 272total releases 170most recent commit 10 days agonpm chevrotain} Downloads
Moo ⭐ 763
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
dependent packages 263total releases 15most recent commit a year agonpm moo} Downloads
Jflex ⭐ 523
The fast scanner generator for Java™ with full Unicode support
dependent packages 64total releases 14most recent commit a year ago
Php Parser ⭐ 500
:herb: NodeJS PHP Parser - extract AST or tokens
dependent packages 100total releases 74most recent commit 7 months agonpm php-parser} Downloads
Lexmachine ⭐ 370
Lex machinary for go.
dependent packages 45total releases 6most recent commit 2 years ago
Lex ⭐ 142
Replaced by foonathan/lexy
most recent commit 3 years ago
Html5gum ⭐ 140
A WHATWG-compliant HTML5 tokenizer and tag soup parser
dependent packages 2total releases 14most recent commit 4 months agocargo html5gum} Downloads
Jison ⭐ 97
bison / YACC / LEX in JavaScript (LALR(1), SLR(1), etc. lexer/parser generator)
dependent packages 39total releases 39most recent commit 3 years agonpm jison-gho} Downloads


Suggest An Alternative To tokenstream
Alternative Project Comparisons
Tokenstream vs Chevrotain
Tokenstream vs Moo
Tokenstream vs Jflex
Tokenstream vs Php Parser
Tokenstream vs Lexmachine
Tokenstream vs Lex
Tokenstream vs Html5gum
Tokenstream vs Works For Me
Tokenstream vs Jison
Tokenstream vs Tokenize.jl
Popular Lexer Projects
Flex ⭐ 3,315
The Fast Lexical Analyzer - scanner generator for lexing in C and C++
dependent packages 2total releases 1latest release February 27, 2018most recent commit 3 months ago
Participle ⭐ 3,215
A parser library for Go
dependent packages 283total releases 54latest release November 30, 2023most recent commit 3 months ago
Logos ⭐ 2,414
Create ridiculously fast Lexers
dependent packages 198total releases 49latest release April 10, 2023most recent commit 3 months agocargo logos} Downloads
Csstree ⭐ 1,756
A tool set for CSS including fast detailed parser, walker, generator and lexer based on W3C specs and browser implementations
dependent packages 744total releases 55latest release December 14, 2022most recent commit 6 months agonpm css-tree} Downloads
Popular Tokenizer Projects
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 3 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 3 months agopypi tokenizers} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8latest release November 20, 2023most recent commit 5 months agopackagist theseer/tokenizer} Downloads
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Jit
Ghc
Ida
Related Searches
Python Tokenizer
Python Lexer
Lexer Tokenizer
Lexer Recursive Descent Parser
Python Recursive Descent Parser
Tokenizer Recursive Descent Parser

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Lexer
Tokenizer
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.