Awesome Open Source

Programming Languages

Search results for tokenizer lex

15 search results found

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

Lexmachine ⭐ 370

Lex machinary for go.

Replaced by foonathan/lexy

bison / YACC / LEX in JavaScript (LALR(1), SLR(1), etc. lexer/parser generator)

Lex is an implementation of lex tool in Ruby.

Parsertest ⭐ 31

A demo project showing the use of yacc and lex (bison, flex) in a Xcode project

A universal lexer library in Swift.

Generic tokenizer written in Nim language 👑 Powered by Nim's Macros

Rftokenizer ⭐ 17

A character-wise tokenizer for morphologically rich languages

a Chinese tokenizer

The half diminished C compiler tutorial. x86 32 bit assembly, BSD license

Lexical analyzer for Javascript developers

🌳 Python lexer that can remember tokens and state and only reparse changed parts of a text document

Snapdragon Scanner ⭐ 5

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

[DEPRECATED] Please use phplrt/lexer instead

Related Searches

Python Tokenizer (341)

Token Tokenizer (224)

Python Lex (173)

1-15 of 15 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.