Awesome Open Source

Programming Languages

Search results for lexer tokenizer

52 search results found

Chevrotain ⭐ 2,381

Parser Building Toolkit for JavaScript

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

The fast scanner generator for Java™ with full Unicode support

Php Parser ⭐ 500

🌿 NodeJS PHP Parser - extract AST or tokens

Lexmachine ⭐ 370

Lex machinary for go.

Replaced by foonathan/lexy

Html5gum ⭐ 140

A WHATWG-compliant HTML5 tokenizer and tag soup parser

Works For Me ⭐ 129

Collection of developer toolkits

bison / YACC / LEX in JavaScript (LALR(1), SLR(1), etc. lexer/parser generator)

Tokenize.jl ⭐ 97

Tokenization for Julia source code

A C# Lua, GLua and Luau parser, code analysis, transformation and generation library.

Wordtokenizers.jl ⭐ 63

High performance tokenizers for natural language processing and other related tasks

Tokenizer ⭐ 57

Tokenizer (lexer) for golang

Lex is an implementation of lex tool in Ruby.

A modern CSS value compiler in JavaScript

Parsello ⭐ 39

A portable single file parser/lexer/tokenizer.

Snl Compiler ⭐ 33

SNL(Small Nested Language) Compiler. Maven jUnit Tokenizer Lexer Syntax Parser. 编译原理词法分析语法分析

A very small and very fast spec compliant css lexer

SQL parser (as understood by SQLite)

Pascal Interpreter ⭐ 32

A simple interpreter for a large subset of Pascal language written for educational purposes

Goselect ⭐ 28

SQL like 'select' interface for files

A universal lexer library in Swift.

The Parser Construction Kit ("Puck"): A Parser Generator and Grammar Translator in C#

Generic tokenizer written in Nim language 👑 Powered by Nim's Macros

JSON Parser Object PHP is a library for parsing the data in JSON format.

The maeel programming language

Snapdragon Lexer ⭐ 19

Converts a string into an array of tokens, with useful methods for looking ahead and behind, capturing, matching, et cetera.

Html Lexer ⭐ 18

HTML5 compliant lexer

Tokenizer ⭐ 15

⚙️ A simple tokenizer for deno

Parser combinators in Kotlin for Kotlin Multiplatform

Tokenstream ⭐ 12

A versatile token stream for handwritten parsers.

TOML parser for Free Pascal Compiler.

Antlr4 Ace Ext ⭐ 11

Tokenizer for ACE editor to do syntax highlighting using an ANTLR4 lexer.

Tokenizer ⭐ 11

Natural Language Tokenizer

🧪 An imperative, dynamically-typed, interpreted, general-purpose programming language

[MODULE] Fast JSON lexer (tokenizer) with no memory footprint and no garbage collector pressure (zero-alloc). 5x faster compared to Go's default encoding/json tokenizer.

Php Rtflex ⭐ 8

A pure PHP lexer / tokenizer for RTF files

🌳 Python lexer that can remember tokens and state and only reparse changed parts of a text document

Lexical analyzer for Javascript developers

Erl_tokenize ⭐ 7

An Erlang source code tokenizer written in Rust.

Crossandra ⭐ 7

Crossandra - a fast and simple tokenization library for Python operating on enums and regular expressions, with a decent amount of configuration.

A simple tokenizer I made for JS while making a project that required tokens.

Interpreter In Prolog ⭐ 7

An interpreter capable of executing a simple scripting language from a plain-text file

Searchbox ⭐ 6

Text input augmented with Lexer magic✨ to support advanced search features.

A performant, type-safe .NET library for building tokenizers and parsers.

A simple lexer written entirely in JavaScript, with no dependencies!

Simplified token definition and lexer creation library for use with Chevrotain.

Calculator.clj ⭐ 5

Learning Clojure by writing a lexer, parser, evaluator

[DEPRECATED] Please use phplrt/lexer instead

Snapdragon Scanner ⭐ 5

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

Dockerfile Syntax Highlighter ⭐ 5

Brackets syntax highlighter for Dockerfiles

A compiler to translate regular expressions (regular grammars) and LL1 BNF languages (subset of context free grammars) to generated scanners and/or parsers.

Related Searches

Python Tokenizer (341)

1-52 of 52 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.