Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Tokenization Open Source Projects

Open source projects categorized as Tokenization
Categories > Tokenization
Spacy ⭐ 28,628
💫 Industrial-strength Natural Language Processing (NLP) in Python
dependent packages 1,367total releases 226most recent commit 9 days agopypi spacy} Downloads
Lunasec ⭐ 1,355
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunase
total releases 3most recent commit 6 months ago
Databunker ⭐ 1,188
Secure SDK/vault for personal records/PII built to comply with GDPR
total releases 1most recent commit 4 months ago
Ravencoin ⭐ 1,041
Ravencoin Core integration/staging tree
most recent commit a year ago
Youtokentome ⭐ 943
Unsupervised text tokenizer focused on computational efficiency
dependent packages 17total releases 14most recent commit 20 days agopypi youtokentome} Downloads
Trankit ⭐ 693
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
dependent packages 2total releases 20most recent commit 5 months agopypi trankit} Downloads
Spacy Streamlit ⭐ 688
👑 spaCy building blocks and visualizers for Streamlit apps
total releases 17most recent commit 9 months ago
Datacamp Python Data Science Track ⭐ 655
All the slides, accompanying code and exercises all stored in this repo. 🎈
most recent commit 9 months ago
Ekphrasis ⭐ 583
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
total releases 54most recent commit 2 years ago
Nlp Cube ⭐ 551
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
dependent packages 1total releases 24most recent commit 10 days agopypi nlpcube} Downloads
Php Text Analysis ⭐ 484
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
dependent packages 6total releases 42most recent commit 7 months agopackagist yooper/php-text-analysis} Downloads
Tokenmonster ⭐ 399
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
total releases 15most recent commit 5 months ago
Clangkit ⭐ 342
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
most recent commit 3 years ago
Vibrato ⭐ 275
🎤 vibrato: Viterbi-based accelerated tokenizer
dependent packages 1total releases 11most recent commit 3 months agocargo vibrato} Downloads
Sudachi.rs ⭐ 253
Sudachi in Rust 🦀 and new generation of SudachiPy
dependent packages 34total releases 39most recent commit 3 months agopypi SudachiPy} Downloads
Codechain ⭐ 245
CodeChain's official implementation in Rust.
most recent commit 3 years ago
Tokenscript ⭐ 237
TokenScript schema, specs and paper
most recent commit 4 months ago
Razdel ⭐ 226
Rule-based token, sentence segmentation for Russian language
dependent packages 7total releases 5most recent commit 9 months agopypi razdel} Downloads
Tokenizer ⭐ 224
Fast and customizable text tokenization library with BPE and SentencePiece support
dependent packages 5total releases 68most recent commit 9 months agopypi pyonmttok} Downloads
Tokenscript ⭐ 215
TokenScript schema, specs and paper
most recent commit 10 months ago
Vaporetto ⭐ 206
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
dependent packages 3total releases 16most recent commit 5 months agocargo vaporetto} Downloads
Python_natural_language_processing ⭐ 164
This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.
most recent commit 2 years ago
Tokencost ⭐ 140
Easy token price estimates for LLMs
most recent commit 3 months ago
The 10 Most Depended On Tokenization Open Source Projects
Spacy ⭐ 28,628
💫 Industrial-strength Natural Language Processing (NLP) in Python
dependent packages 1,367total releases 226latest release October 16, 2023most recent commit 9 days agopypi spacy} Downloads
Sudachi.rs ⭐ 253
Sudachi in Rust 🦀 and new generation of SudachiPy
dependent packages 34total releases 39latest release February 15, 2023most recent commit 3 months agopypi SudachiPy} Downloads
Youtokentome ⭐ 943
Unsupervised text tokenizer focused on computational efficiency
dependent packages 17total releases 14latest release February 12, 2020most recent commit 20 days agopypi youtokentome} Downloads
Wink Tokenizer ⭐ 47
Multilingual tokenizer that automatically tags each token with its type
dependent packages 15total releases 19latest release January 27, 2022most recent commit 2 years agonpm wink-tokenizer} Downloads
Simplemma ⭐ 100
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
dependent packages 9total releases 14latest release January 20, 2023most recent commit 7 months agopypi simplemma} Downloads
Razdel ⭐ 226
Rule-based token, sentence segmentation for Russian language
dependent packages 7total releases 5latest release March 26, 2020most recent commit 9 months agopypi razdel} Downloads
The 10 Latest Releases In Tokenization Open Source Projects
Vgs Collect Android ⭐ 6
VGS Collect Android SDK
dependent packages 2total releases 29latest release November 03, 2023most recent commit 4 months ago
Tivars_lib_py ⭐ 8
A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files
total releases 7latest release October 29, 2023most recent commit 3 months ago
Spacy ⭐ 28,628
💫 Industrial-strength Natural Language Processing (NLP) in Python
dependent packages 1,367total releases 226latest release October 16, 2023most recent commit 9 days agopypi spacy} Downloads
Nlpashto ⭐ 10
Pashto Natural Language Processing Toolkit
total releases 23latest release October 09, 2023most recent commit 6 months ago
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.