Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

Tok

Fast and customizable tokenization 🚤
Stars
54
License
No license specified
Open Issues
1
Most Recent Commit
5 years ago
Programming Language
Python
Monthly Downloads
pypi tok Downloads
Dependent Repos
1
Dependent Packages
1
Total Releases
9
Latest Release
July 09, 2019
Categories
Repo
Alternatives To Tok
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Moo763npm moo} Downloads12,799263a year ago15October 03, 202231bsd-3-clauseJavaScript
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
Jflex52319464a year ago14March 11, 202325otherJava
The fast scanner generator for Java™ with full Unicode support
Js Tokens410npm js-tokens} Downloads586,3601,8706 months ago21September 09, 2023mitJavaScript
Tiny JavaScript tokenizer.
Lexmachine3701452 years ago6July 15, 20225otherGo
Lex machinary for go.
Tokenizer138packagist nette/tokenizer} Downloads33397a year ago9February 09, 2022otherPHP
[DISCONTINUED] Source code tokenizer
Node Tokenizer95npm tokenizer} Downloads267417 years ago11July 11, 20139JavaScript
A tokenizer that looks like a stream for JavaScript and node.js
Djurl86
6 years ago5July 05, 2017otherPython
Simple yet helpful library for writing Django urls by an easy, short and intuitive way.
Tok54pypi tok} Downloads115 years ago9July 09, 20191Python
Fast and customizable tokenization :speedboat:
Chiffon52npm chiffon} Downloads148 years ago35April 17, 20165mitJavaScript
A small ECMAScript parser, tokenizer and minifier written in JavaScript.
Greeb16
25 years ago26January 14, 2015mitRuby
Greeb is a simple Unicode-aware regexp-based tokenizer.
Alternatives To Tok
Select To Compare
Moo ⭐ 763
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
dependent packages 263total releases 15most recent commit a year agonpm moo} Downloads
Jflex ⭐ 523
The fast scanner generator for Java™ with full Unicode support
dependent packages 64total releases 14most recent commit a year ago
Js Tokens ⭐ 410
Tiny JavaScript tokenizer.
dependent packages 1,870total releases 21most recent commit 6 months agonpm js-tokens} Downloads
Lexmachine ⭐ 370
Lex machinary for go.
dependent packages 45total releases 6most recent commit 2 years ago
Tokenizer ⭐ 138
[DISCONTINUED] Source code tokenizer
dependent packages 97total releases 9most recent commit a year agopackagist nette/tokenizer} Downloads
Node Tokenizer ⭐ 95
A tokenizer that looks like a stream for JavaScript and node.js
dependent packages 41total releases 11most recent commit 7 years agonpm tokenizer} Downloads
Tok ⭐ 54
Fast and customizable tokenization :speedboat:
dependent packages 1total releases 9most recent commit 5 years agopypi tok} Downloads
Chiffon ⭐ 52
A small ECMAScript parser, tokenizer and minifier written in JavaScript.
dependent packages 4total releases 35most recent commit 8 years agonpm chiffon} Downloads
Greeb ⭐ 16
Greeb is a simple Unicode-aware regexp-based tokenizer.
total releases 26most recent commit 5 years ago


Suggest An Alternative To tok
Alternative Project Comparisons
Tok vs Moo
Tok vs Jflex
Tok vs Js Tokens
Tok vs Lexmachine
Tok vs Tokenizer
Tok vs Node Tokenizer
Tok vs Djurl
Tok vs Chiffon
Tok vs Greeb
Popular Regular Expression Projects
Ripgrep ⭐ 44,855
ripgrep recursively searches directories for a regex pattern while respecting your gitignore
dependent packages 348total releases 37latest release November 26, 2023most recent commit 7 days agocargo ignore} Downloads
Learn Regex ⭐ 44,809
Learn regex the easy way
most recent commit 10 months ago
Androidutilcode ⭐ 32,157
:fire: Android developers should collect the following utils(updating).
dependent packages 25total releases 4latest release October 14, 2022most recent commit a year ago
Fd ⭐ 30,369
A simple, fast and user-friendly alternative to 'find'
total releases 31latest release October 21, 2023most recent commit 3 months ago
Z ⭐ 15,648
z - jump around
total releases 1latest release February 27, 2018most recent commit 5 months ago
Popular Tokenizer Projects
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 3 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 3 months agopypi tokenizers} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141latest release November 11, 2023most recent commit 4 months agonpm file-type} Downloads
Popular Text Processing Categories
Tex
Format
Translation
Address
Character
Highlighter
Regular Expression
Formatter
Selector
Localization

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Regular Expression
Tokenizer
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.