Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Tokenization Open Source Projects

Open source projects categorized as Python Tokenization
Categories > Tokenization
Spacy ⭐ 28,628
💫 Industrial-strength Natural Language Processing (NLP) in Python
dependent packages 1,367total releases 226most recent commit 17 days agopypi spacy} Downloads
Trankit ⭐ 693
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
dependent packages 2total releases 20most recent commit 5 months agopypi trankit} Downloads
Spacy Streamlit ⭐ 688
👑 spaCy building blocks and visualizers for Streamlit apps
total releases 17most recent commit 9 months ago
Datacamp Python Data Science Track ⭐ 655
All the slides, accompanying code and exercises all stored in this repo. 🎈
most recent commit 9 months ago
Ekphrasis ⭐ 583
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
total releases 54most recent commit 2 years ago
Sudachi.rs ⭐ 253
Sudachi in Rust 🦀 and new generation of SudachiPy
dependent packages 34total releases 39most recent commit 3 months agopypi SudachiPy} Downloads
Razdel ⭐ 226
Rule-based token, sentence segmentation for Russian language
dependent packages 7total releases 5most recent commit 9 months agopypi razdel} Downloads
Tokenizer ⭐ 224
Fast and customizable text tokenization library with BPE and SentencePiece support
dependent packages 5total releases 68most recent commit 9 months agopypi pyonmttok} Downloads
Tokencost ⭐ 140
Easy token price estimates for LLMs
most recent commit 4 months ago
Vtext ⭐ 110
Simple NLP in Rust with Python bindings
dependent packages 1total releases 4most recent commit 3 years agopypi vtext} Downloads
Simplemma ⭐ 100
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
dependent packages 9total releases 14most recent commit 7 months agopypi simplemma} Downloads
Nlp Cheat Sheet Python ⭐ 98
NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition
most recent commit a year ago
Tweebanknlp ⭐ 94
[LREC 2022] An off-the-shelf pre-trained Tweet NLP Toolkit (NER, tokenization, lemmatization, POS tagging, dependency parsing) + Tweebank-NER dataset
most recent commit 3 months ago
Lima ⭐ 92
The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.
total releases 22most recent commit 6 months ago
Icetk ⭐ 69
A unified tokenization tool for Images, Chinese and English.
dependent packages 4total releases 7most recent commit a year agopypi icetk} Downloads
Nlpcloud Python ⭐ 63
NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and more...
dependent packages 29total releases 41most recent commit 6 months agopypi nlpcloud} Downloads
Python Fpe ⭐ 62
FPE - Format Preserving Encryption with FF3 in Python
dependent packages 2total releases 6most recent commit 9 months agopypi ff3} Downloads
Spacy Server ⭐ 57
🦜 Containerized HTTP API for industrial-strength NLP via spaCy and sense2vec
most recent commit 3 years ago
Charformer Pytorch ⭐ 51
Implementation of the GBST block from the Charformer paper, in Pytorch
total releases 4most recent commit 3 years ago
Mbti Personality Classifier ⭐ 51
A model which uses your social media posting predict your MBTI personality type.
most recent commit 4 months ago
Attacut ⭐ 47
A Fast and Accurate Neural Thai Word Segmenter
dependent packages 3total releases 17most recent commit 3 years agopypi attacut} Downloads
Python ⭐ 38
Rosette API Client Library for Python
total releases 31most recent commit 3 months ago
Xontrib Output Search ⭐ 35
Get identifiers, paths, URLs and words from the previous command output and use them for the next command in xonsh shell.
dependent packages 1total releases 12most recent commit 2 months agopypi xontrib-output-search} Downloads
The 10 Most Depended On Python Tokenization Open Source Projects
Spacy ⭐ 28,628
💫 Industrial-strength Natural Language Processing (NLP) in Python
dependent packages 1,367total releases 226latest release October 16, 2023most recent commit 17 days agopypi spacy} Downloads
Sudachi.rs ⭐ 253
Sudachi in Rust 🦀 and new generation of SudachiPy
dependent packages 34total releases 39latest release February 15, 2023most recent commit 3 months agopypi SudachiPy} Downloads
Simplemma ⭐ 100
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
dependent packages 9total releases 14latest release January 20, 2023most recent commit 7 months agopypi simplemma} Downloads
Razdel ⭐ 226
Rule-based token, sentence segmentation for Russian language
dependent packages 7total releases 5latest release March 26, 2020most recent commit 9 months agopypi razdel} Downloads
Tokenizer ⭐ 224
Fast and customizable text tokenization library with BPE and SentencePiece support
dependent packages 5total releases 68latest release January 11, 2023most recent commit 9 months agopypi pyonmttok} Downloads
Icetk ⭐ 69
A unified tokenization tool for Images, Chinese and English.
dependent packages 4total releases 7latest release June 28, 2022most recent commit a year agopypi icetk} Downloads
Attacut ⭐ 47
A Fast and Accurate Neural Thai Word Segmenter
dependent packages 3total releases 17latest release March 13, 2020most recent commit 3 years agopypi attacut} Downloads
Trankit ⭐ 693
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
dependent packages 2total releases 20latest release March 26, 2022most recent commit 5 months agopypi trankit} Downloads
Python Fpe ⭐ 62
FPE - Format Preserving Encryption with FF3 in Python
dependent packages 2total releases 6latest release December 31, 2021most recent commit 9 months agopypi ff3} Downloads
The 10 Latest Releases In Python Tokenization Open Source Projects
Tivars_lib_py ⭐ 8
A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files
total releases 7latest release October 29, 2023most recent commit 3 months ago
Spacy ⭐ 28,628
💫 Industrial-strength Natural Language Processing (NLP) in Python
dependent packages 1,367total releases 226latest release October 16, 2023most recent commit 17 days agopypi spacy} Downloads
Nlpashto ⭐ 10
Pashto Natural Language Processing Toolkit
total releases 23latest release October 09, 2023most recent commit 6 months ago
Tokenization Scorer ⭐ 10
Simple-to-use scoring function for arbitrarily tokenized texts.
total releases 2latest release April 29, 2023most recent commit 9 months ago
Sudachi.rs ⭐ 253
Sudachi in Rust 🦀 and new generation of SudachiPy
dependent packages 34total releases 39latest release February 15, 2023most recent commit 3 months agopypi SudachiPy} Downloads
Bigrams ⭐ 6
Non-intrusive ngrams generations
total releases 3latest release January 25, 2023most recent commit 4 months ago
Simplemma ⭐ 100
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
dependent packages 9total releases 14latest release January 20, 2023most recent commit 7 months agopypi simplemma} Downloads
Tokenizer ⭐ 224
Fast and customizable text tokenization library with BPE and SentencePiece support
dependent packages 5total releases 68latest release January 11, 2023most recent commit 9 months agopypi pyonmttok} Downloads
Lima ⭐ 92
The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.
total releases 22latest release December 22, 2022most recent commit 6 months ago
Popular Programming Languages Categories
Python
Javascript
Java
Typescript
C Plus Plus
Php
Golang
C
C Sharp
Shell
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.