Awesome Open Source

Programming Languages

Search results for javascript tokenizer

89 search results found

File Type ⭐ 3,366

Detect the file type of a Buffer/Uint8Array/ArrayBuffer

Chevrotain ⭐ 2,381

Parser Building Toolkit for JavaScript

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

Elasticsearch Inquisitor ⭐ 712

Site plugin for Elasticsearch to help understand and debug queries.

React Typeahead ⭐ 660

Pure react-based typeahead and typeahead-tokenizer

Php Parser ⭐ 500

🌿 NodeJS PHP Parser - extract AST or tokens

Naive-Bayes Classifier for node.js

Js Tokens ⭐ 410

Tiny JavaScript tokenizer.

Tabloid ⭐ 365

A minimal programming language inspired by clickbait headlines

Llama Tokenizer Js ⭐ 250

JS tokenizer for LLaMA

Segmentit ⭐ 208

任何 JS 环境可用的中文分词包，fork from leizongmin/node-segment

JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification

Vietnamese NLP Toolkit for Node

EasyLms is a development management system with Laravel

Works For Me ⭐ 129

Collection of developer toolkits

Textboxlist ⭐ 104

MooTools tokenizer

Megamark ⭐ 96

😻 Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer

Css Selector Tokenizer ⭐ 78

Parses and stringifies CSS selectors.

Monaco Ace Tokenizer ⭐ 72

Syntax highlighting support for additional languages in monaco-editor

Dom Drawer ⭐ 70

A small toy to help you draw the DOM structure on Canvas

A small ECMAScript parser, tokenizer and minifier written in JavaScript.

Handbook ⭐ 51

[WIP] A handbook on the unified ecosystem, including supported syntaxes, ASTs, and guides on how to use the libraries

Nlp Toolkit ⭐ 48

some helper functions for NLP operations

Wink Tokenizer ⭐ 47

Multilingual tokenizer that automatically tags each token with its type

Chinese Tokenizer ⭐ 39

Tokenizes Chinese texts into words.

Gpt3 Tokenizer ⭐ 38

Isomorphic Tokenizer for GPT-3 and Codex Models by OpenAI.

Node Sentence Tokenizer ⭐ 36

Tokenize paragraphs into sentences, and smaller tokens.

A very small and very fast spec compliant css lexer

Pascal Interpreter ⭐ 32

A simple interpreter for a large subset of Pascal language written for educational purposes

Minilisp ⭐ 31

Making a Mini-Lisp: Introduction to Transpilers and how tokenizer, parser and interpreter work together. via @AngularClass

Graphql Query Compress ⭐ 30

Compress GraphQL Query String

Nlp Js Tools French ⭐ 29

POS Tagger, lemmatizer and stemmer for french language in javascript

Input Tokenizer ⭐ 27

jQuery plugin that allows a user to type keywords, which will be broken up into tokens/tags and displayed, similarly to tagging a post on Tumblr or Stack-Overflow.

Excel Formula Tokenizer ⭐ 26

Tokenize Excel formulas

React Tokenizer ⭐ 26

A React.js component for tokenizing user input

Tokenize Text ⭐ 25

Javascript text tokenizer that is easy to use and compose.

Js Summarize ⭐ 25

An NLP Summarizer built with Javascript.

Lisp Esque Language ⭐ 24

💠The Lel programming language

Scss Tokenizer ⭐ 23

A tokenzier for Sass' SCSS syntax

NodeJS NLP Library for Bahasa Indonesia.

html 5 tokenizer

Classificator ⭐ 22

naive bayes classifier

Toy_lang ⭐ 21

The first language I made.

Unicode Tokenizer ⭐ 20

Unicode Tokenizer following the Unicode Line Breaking algorithm

Sketch Tokenizer ⭐ 19

Easily add design tokens to layouts

Snapdragon Lexer ⭐ 19

Converts a string into an array of tokens, with useful methods for looking ahead and behind, capturing, matching, et cetera.

Html Lexer ⭐ 18

HTML5 compliant lexer

A PDF collection reader with built-in full-text search engine

Tokenizer ⭐ 17

OLD, OLD Tokenizer allows you to create Mail.app like autocompletion fields - like facebook "compose message" "to" field

Answer.js ⭐ 14

Better in-page search using natural queries, implemented client-side with a Porter-stemmer, part of speech tagger and naive tokenizer. Reads current/specified page content and identifies passages that contain answers to user queries.

Pratt Parser ⭐ 14

TDOP parser https://tdop.github.io

Paddletokenizer ⭐ 13

使用 PaddlePaddle 实现基于深度神经网络的中文分词引擎 | A DNN Chinese Tokenizer by Using PaddlePaddle

Node Atok ⭐ 13

Fast, easy and flexible tokenizer for Node Streams

🗒️ Tiny extensible markdown parser/renderer single-file JS library

Tokenizer2 ⭐ 13

tokenize any text stream given some basic regex rules to match tokens

Redisgrip ⭐ 12

Redis GUI Client

Words N Numbers ⭐ 11

Tokenizing strings of text. Regex extracting arrays of words and optionally numbers, emojis, tags, usernames and email addresses from strings. For Node.js and the browser. When you need more than just [a-z] regular expressions.

Jokenizer ⭐ 11

🐉 JavaScript expression tokenizer and evaluator

Inc Clojure Brush ⭐ 11

(inc clojure-brush) ; A smarter Syntax Highlighter brush for Clojure

Antlr4 Ace Ext ⭐ 11

Tokenizer for ACE editor to do syntax highlighting using an ANTLR4 lexer.

Tiptap Annotation Magic ⭐ 9

An extension for the Tiptap editor, enabling the annotation of text. Comes with support for overlapping annotations, useful for e.g. NLP tokenization.

😸 A LOLCODE interpreter written in JavaScript

A tokenizer for French

Shortcode Tokenizer ⭐ 9

Tokenizes a string containing shortcodes and optionally outputs an AST

Polix Rpc ⭐ 9

rpc plugin for polix

Blockchain Indexer ⭐ 9

Lexical analyzer for Javascript developers

Node Atok Parser ⭐ 8

Parser generator based on node-atok

Pl0 Compiler ⭐ 7

A simple PL/0 compiler in JavaScript

A simple tokenizer I made for JS while making a project that required tokens.

Confluence2markdown ⭐ 7

Convert old Confluence markup to Markdown

Solr Whosonfirst ⭐ 7

Who's on first?

Simplified token definition and lexer creation library for use with Chevrotain.

Kiritobuf ⭐ 6

Interface description language | Kirito ⚔

Esperanto tokenizer

Node Pstrscan ⭐ 6

A simple, fast string tokenizer for lexical scanning operations.

Mini Tokenizer ⭐ 6

tiny tokenizer for simple parsing.

A simple lexer written entirely in JavaScript, with no dependencies!

Js Combinators ⭐ 6

Parser Combinators in JavaScript

Searchbox ⭐ 6

Text input augmented with Lexer magic✨ to support advanced search features.

Copy Paste Detector ⭐ 6

copy-paste-detector is a copy/paste-detector for various languages

Extract Lemmatized Nonstop Words ⭐ 5

Extracts a pure list of stemmed words of a text filtered by stop words

Dockerfile Syntax Highlighter ⭐ 5

Brackets syntax highlighter for Dockerfiles

Snapdragon Scanner ⭐ 5

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

Lexical Parser ⭐ 5

Lexical Parser written in Javascript, using Node.js's environment

Wat Tokenizer ⭐ 5

Tokenize WebAssembly Text-format into list of lists

A regex-based lexical analyzer for JavaScript

Glsl Token String ⭐ 5

converts an array of GLSL tokens to a plain GLSL source string

Related Searches

Javascript Reactjs (121,118)

Javascript Html (48,022)

Javascript Css (46,402)

Javascript Deployment (32,768)

Javascript Express (27,853)

Javascript Command Line (22,145)

Javascript Testing (21,652)

Javascript Redux (21,545)

Javascript Vue (19,769)

Javascript Mongodb (18,598)

1-89 of 89 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.