Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for token tokenizer
token
x
tokenizer
x
61 search results found
Php Token Stream
⭐
6,457
Wrapper around PHP's tokenizer extension.
Superpower
⭐
848
A C# parser construction toolkit with high-quality error reporting
Php Parser
⭐
500
🌿 NodeJS PHP Parser - extract AST or tokens
Js Tokens
⭐
410
Tiny JavaScript tokenizer.
Lexmachine
⭐
370
Lex machinary for go.
Nmt Chatbot
⭐
309
NMT Chatbot
Coccoc Tokenizer
⭐
295
high performance tokenizer for Vietnamese language
Tsql Parser
⭐
278
Library Written in C# For Parsing SQL Server T-SQL Scripts in .Net
Segmentit
⭐
208
任何 JS 环境可用的中文分词包,fork from leizongmin/node-segment
Syntok
⭐
158
Text tokenization and sentence segmentation (segtok v2)
Lex
⭐
142
Replaced by foonathan/lexy
Tokenizer
⭐
138
[DISCONTINUED] Source code tokenizer
Vokenization
⭐
133
PyTorch code for EMNLP 2020 Paper "Vokenization: Improving Language Understanding with Visual Supervision"
Elasticsearch Analysis Bosonnlp
⭐
105
BosonNLP Analysis for ElasticSearch
Tokenize.jl
⭐
97
Tokenization for Julia source code
Node Tokenizer
⭐
95
A tokenizer that looks like a stream for JavaScript and node.js
Streamparser Json
⭐
87
Streaming JSON parser in Javascript for Node.js and the browser
Punkt Segmenter
⭐
86
Ruby port of the NLTK Punkt sentence segmentation algorithm
Pragmatic_tokenizer
⭐
83
A multilingual tokenizer to split a string into tokens
Go Gpt 3 Encoder
⭐
74
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
String Calc
⭐
69
PHP calculator library for mathematical terms (expressions) passed as strings
Tokenizer
⭐
65
Tokenize CSS according to the CSS Syntax
Howtowriteacompiler
⭐
63
How to write a compiler from scratch in 30 minutes
Nlp Id
⭐
58
Kumparan's NLP Services
Lex
⭐
55
Lex is an implementation of lex tool in Ruby.
Tokenizer
⭐
53
Convert source code into numerical tokens
Chiffon
⭐
52
A small ECMAScript parser, tokenizer and minifier written in JavaScript.
Hindi Tokenizer
⭐
50
This is a package in Python which implements a tokenizer, stemmer for Hindi language
Alm
⭐
47
Smart Language Model
Wink Tokenizer
⭐
47
Multilingual tokenizer that automatically tags each token with its type
Senna
⭐
46
NLP SENNA (http://ml.nec-labs.com/senna) interface to LuaJIT
Refmt
⭐
45
Object mapping for golang.
Elasticsearch Extended Analyze
⭐
44
Extend Analyze API Plugin for Elasticsearch
Botok
⭐
43
🏷 བོད་ཏོག [pʰøtɔk̚] Tibetan word tokenizer in Python
Nlpstack
⭐
42
NLP toolkit (tokenizer, POS-tagger, parser, etc.)
Tokenizer
⭐
41
.NET Tokenization Library
Node Sentence Tokenizer
⭐
36
Tokenize paragraphs into sentences, and smaller tokens.
Parser Tutorial
⭐
35
write your own json parser.
Pascal Interpreter
⭐
32
A simple interpreter for a large subset of Pascal language written for educational purposes
Tokenizer
⭐
27
A tokenizer for Icelandic text
Input Tokenizer
⭐
27
jQuery plugin that allows a user to type keywords, which will be broken up into tokens/tags and displayed, similarly to tagging a post on Tumblr or Stack-Overflow.
Tokenizer.js
⭐
27
A modern runtime text tokenizer, for the modern web.
Tokenize Text
⭐
25
Javascript text tokenizer that is easy to use and compose.
Swilex
⭐
24
A universal lexer library in Swift.
Toktok
⭐
23
Generic tokenizer written in Nim language 👑 Powered by Nim's Macros
Jpophp
⭐
23
JSON Parser Object PHP is a library for parsing the data in JSON format.
Trainable Tokenizer
⭐
22
Fast and trainable tokenizer for natural languages relying on maximum entropy methods.
Salty_tokenizer
⭐
21
Add security tokens to FiveM server events through obfuscated client events.
Unicode Tokenizer
⭐
20
Unicode Tokenizer following the Unicode Line Breaking algorithm
Snapdragon Lexer
⭐
19
Converts a string into an array of tokens, with useful methods for looking ahead and behind, capturing, matching, et cetera.
Strtok3
⭐
19
Promise based streaming tokenizer
Sketch Tokenizer
⭐
19
Easily add design tokens to layouts
Tokenizer
⭐
19
Package tokenizer provides encoding for tokens that can carry user data.
Html Lexer
⭐
18
HTML5 compliant lexer
Rftokenizer
⭐
17
A character-wise tokenizer for morphologically rich languages
Fuzzy_tools
⭐
17
Fuzzy document finding in Ruby
Jsont
⭐
16
A minimal and portable JSON tokenizer for building highly effective and strict parsers (in C and C++)
Vocab
⭐
16
Vocabulary using n-grams
Gerpt2
⭐
15
German small and large versions of GPT2.
Tivars_lib_cpp
⭐
14
A C++ library to interact with TI-z80 (82/83/84 series) calculators files (programs, lists, matrices, etc.)
Dig Lsh Clustering
⭐
14
Clustering documents based on LSH
Pratt Parser
⭐
14
TDOP parser https://tdop.github.io
Tokenizer2
⭐
13
tokenize any text stream given some basic regex rules to match tokens
Tokenflow
⭐
12
Data driven tokenizer for entities, attributes, quantifiers, numbers and intents in NLP scenarios.
Vespa Kuromoji Linguistics
⭐
12
Phptokenizer
⭐
12
Wrapper around token_get_all. Easy way to extract and modify yout code
Deltas
⭐
12
A library for generating deltas of the difference between two sequences of tokens.
Tokenizercfc
⭐
11
Nonce manager for Coldfusion forms for more information, wiki, issue tracker, and feature requests visit
Antlr4 Ace Ext
⭐
11
Tokenizer for ACE editor to do syntax highlighting using an ANTLR4 lexer.
Interactive Elastic Analyzer
⭐
11
Interactive Elasticsearch Analyzer
Laravel Token
⭐
10
Laravel token management
Ipa
⭐
10
NLP Preprocessing Pipeline Wrappers
Tokenreplacer
⭐
10
Token Replacer is a simple and small Java Library that helps replacing tokens in strings. You can replace the tokens with static values or create values "on-the-fly" by calling a generator. You can even pass arguments to the generator which makes it pretty powerful.
Android Multiautocomplete
⭐
9
A lightweight and powerful abstraction over MultiAutoCompleteTextView and Tokenizer
Nlp
⭐
9
NLP Library written in rust
Pypascaltokenizer
⭐
9
Tokenizer for Pascal syntax (Delphi/FreePascal) written in Python 3
Shortcode Tokenizer
⭐
9
Tokenizes a string containing shortcodes and optionally outputs an AST
Zearch
⭐
9
ragecoded code search with json endpoint
Gtokenizer
⭐
9
A Ruby gem for simple string tokenization, based on the method used by Google for their Google NGrams app (http://ngrams.googlelabs.com/)
Lexr
⭐
8
Lexical analyzer for Javascript developers
Tokenizer
⭐
8
Tokenizer package as used by Hippo.
Boreas
⭐
8
A neat css parser written in TypeScript
Tap
⭐
8
TAP: A Static Analysis Model for PHP Vulnerabilities Based on Token and Deep Learning Technology
Haskell Css Syntax
⭐
8
Tokenizer and parser for the CSS syntax
Php Tokenizer
⭐
7
A zero-dependency tokenizer for PHP
Tokesies
⭐
7
A string tokenizer library for Rust
Stripe Watchos Tokenizer
⭐
6
Allows to convert authorized PKPayment objects to Stripe tokens
Parsing
⭐
6
A performant, type-safe .NET library for building tokenizers and parsers.
Sept
⭐
6
A simple extensible path template generator
Pickmybrain
⭐
6
a php search engine utilizing compressed inverted index architecture
Arduino Stringtokenizer Library
⭐
6
A very simple arduino library to use java like string-tokenizer functions to split a string with delimiters.
Perplex
⭐
6
A simple lexer written entirely in JavaScript, with no dependencies!
Design Tokenizer Integration Example
⭐
6
An un-opinionated starter kit that demonstrates one way to integrate design tokens into your designer/developer workflow.
Mascara
⭐
6
A natural language tokenizer
Glsl Token String
⭐
5
converts an array of GLSL tokens to a plain GLSL source string
Transpyler
⭐
5
A framework for generating internationalized versions of Python.
Mathy_core
⭐
5
Computer Algebra System for converting text inputs to trees, manipulating them with rules, and evaluating their values.
Htmltokenizer
⭐
5
Tokenize your HTML
Tokdiff
⭐
5
Tokenizer-based character diff tool
Spag
⭐
5
A compiler to translate regular expressions (regular grammars) and LL1 BNF languages (subset of context free grammars) to generated scanners and/or parsers.
Related Searches
Javascript Token (8,107)
Python Token (4,812)
Token Oauth (2,402)
Token Jwt (2,294)
Php Token (2,264)
Java Token (2,161)
Ruby Token (1,845)
Token Authentication (1,822)
Golang Token (1,792)
Token Contracts (1,494)
1-61 of 61 search results
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.