Awesome Open Source

Programming Languages

Search results for regular expression tokenizer

regular-expression x

2 search results found

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

The fast scanner generator for Java™ with full Unicode support

Js Tokens ⭐ 410

Tiny JavaScript tokenizer.

Lexmachine ⭐ 370

Lex machinary for go.

Tokenizer ⭐ 138

[DISCONTINUED] Source code tokenizer

Node Tokenizer ⭐ 95

A tokenizer that looks like a stream for JavaScript and node.js

Simple yet helpful library for writing Django urls by an easy, short and intuitive way.

Fast and customizable tokenization 🚤

A small ECMAScript parser, tokenizer and minifier written in JavaScript.

Greeb is a simple Unicode-aware regexp-based tokenizer.

Tokenizer2 ⭐ 13

tokenize any text stream given some basic regex rules to match tokens

Words N Numbers ⭐ 11

Tokenizing strings of text. Regex extracting arrays of words and optionally numbers, emojis, tags, usernames and email addresses from strings. For Node.js and the browser. When you need more than just [a-z] regular expressions.

A text processing tool including tag(HTML, URL, Email) extraction and removing, punctuation normalization, simple segmentation, and so on.

Tokenreplacer ⭐ 10

Token Replacer is a simple and small Java Library that helps replacing tokens in strings. You can replace the tokens with static values or create values "on-the-fly" by calling a generator. You can even pass arguments to the generator which makes it pretty powerful.

Crossandra ⭐ 7

Crossandra - a fast and simple tokenization library for Python operating on enums and regular expressions, with a decent amount of configuration.

Php Tokenizer ⭐ 7

A zero-dependency tokenizer for PHP

Snapdragon Scanner ⭐ 5

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

Php Tokenizer ⭐ 5

A string tokenizer written in PHP using regular expressions

Nlp Natural Language Processing ⭐ 5

This repository is a collection of six minor projects focused on Natural Language Processing (NLP) along with relevant datasets. The projects are designed to help individuals gain a better understanding of NLP by applying concepts to real-world problems. Additionally, the repository includes a file that provides a comprehensive overview of NLP .

Lexical Parser ⭐ 5

Lexical Parser written in Javascript, using Node.js's environment

A Java-based configurable tokenizer and sentence splitter

A regex-based lexical analyzer for JavaScript

A compiler to translate regular expressions (regular grammars) and LL1 BNF languages (subset of context free grammars) to generated scanners and/or parsers.

[DEPRECATED] Please use phplrt/lexer instead

Related Searches

Javascript Regular Expression (2,421)

Java Regular Expression (530)

Python Tokenizer (341)

1-2 of 2 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.