Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for tokenize
tokenize
x
20 search results found
Micromark
⭐
1,538
small, safe, and great commonmark (optionally gfm) compliant markdown parser
Wink Nlp
⭐
1,057
Developer friendly Natural Language Processing ✨
Markdown Rs
⭐
631
CommonMark compliant markdown parser in Rust with ASTs and extensions
Tokenmonster
⭐
399
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
Snapdragon
⭐
219
snapdragon is an extremely pluggable, powerful and easy-to-use parser-renderer factory.
Mdast Util From Markdown
⭐
153
mdast utility to parse markdown
Pynlp
⭐
105
A pythonic wrapper for Stanford CoreNLP.
Tokenize2
⭐
82
Tokenize2 is a plugin which allows your users to select multiple items from a predefined list or ajax, using autocompletion as they type to find each item. You may have seen a similar type of text entry when filling in the recipients field sending messages on facebook or tags on tumblr.
Wink Nlp Utils
⭐
81
NLP Functions for amplifying negations, managing elisions, creating ngrams, stems, phonetic codes to tokens and more.
Deid Examples
⭐
58
Examples scripts that showcase how to use Private AI Text to de-identify, redact, hash, tokenize, mask and synthesize PII in text.
Extract Comments
⭐
36
Extract JavaScript code comments from a string or glob of files.
Htmldoc
⭐
20
A token based HTML Document parser and minifier written in PHP. Extract attribute values and text using CSS selectors.
Parsers Compilers
⭐
16
Lexers, tokenizers, parsers, compilers, renderers, stringifiers... What's the difference, and how do they work?
Babel Extract Comments
⭐
12
Uses babel to extract JavaScript code comments from a string. Returns an array of comment objects, with line, column, index, comment type and comment string.
Tokenize Comment
⭐
10
Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.
Brown Water Python
⭐
9
More detailed documentation for the Python tokenize module
Transfromer_nn_block
⭐
9
Implemented transformer NN block for Machine translation, text classfication, Natural language inference as well as Machine reading comprehension model.
Tivars_lib_py
⭐
8
A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files
Untokenize
⭐
6
Transforms tokens into original source code (while preserving whitespace)
Snapdragon Scanner
⭐
5
Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.
1-20 of 20 search results
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.