The Top 23 Python Tokenizer Open Source Projects

Open source projects categorized as Python Tokenizer

Categories > Programming Languages > Python

Categories > Compilers > Tokenizer

Gpt2 Chinese ⭐ 7,249

Chinese version of GPT2 training code, using BERT tokenizer.

most recent commit 4 months ago

Hazm ⭐ 1,104

Persian NLP Toolkit

dependent packages 13total releases 20most recent commit 11 days ago

Natasha ⭐ 1,085

Solves basic Russian NLP tasks, API for lower level Natasha projects

dependent packages 9total releases 19most recent commit 7 months ago

Soynlp ⭐ 801

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

dependent packages 9total releases 33most recent commit 2 years ago

Wordless ⭐ 649

An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation

most recent commit 3 months ago

Ru_transformers ⭐ 627

most recent commit 3 years ago

Ekphrasis ⭐ 583

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

total releases 54most recent commit 2 years ago

Gretel Synthetics ⭐ 490

Synthetic data generators for structured and unstructured text, featuring differentially private learning.

dependent packages 1total releases 53most recent commit 3 months ago

Ckip Transformers ⭐ 439

CKIP Transformers

most recent commit a year ago

Nagisa ⭐ 365

A Japanese tokenizer based on recurrent neural networks

dependent packages 7total releases 22most recent commit 3 months ago

Structural Probes ⭐ 340

Codebase for testing whether hidden states of neural networks encode discrete structures.

most recent commit 2 years ago

Melusine ⭐ 335

Melusine is a high-level library for emails classification and feature extraction "dédiée aux courriels français".

total releases 53most recent commit 3 months ago

Kobart ⭐ 320

Korean BART

most recent commit 2 years ago

Smoothnlp ⭐ 320

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

total releases 21most recent commit 4 years ago

Deepcut ⭐ 319

A Thai word tokenization library using Deep Neural Network

dependent packages 3total releases 30most recent commit 4 years ago

Sudachipy ⭐ 318

Python version of Sudachi, a Japanese tokenizer.

most recent commit 2 years ago

Mail Parser ⭐ 311

Tokenizer for raw mails

dependent packages 20total releases 84most recent commit 2 years ago

Nmt Chatbot ⭐ 309

NMT Chatbot

most recent commit 4 years ago

Text2text ⭐ 268

Text2Text: Crosslingual NLP/G toolkit

total releases 134most recent commit 3 months ago

Bitextor ⭐ 260

Bitextor generates translation memories from multilingual websites

most recent commit 7 months ago

Tokenizer ⭐ 224

Fast and customizable text tokenization library with BPE and SentencePiece support

dependent packages 5total releases 68most recent commit 9 months ago

Konoha ⭐ 200

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.

dependent packages 1total releases 10most recent commit 3 months ago

Kobert Transformers ⭐ 185

KoBERT on 🤗 Huggingface Transformers 🤗 (with Bug Fixed)

dependent packages 3total releases 11most recent commit 5 months ago

The 10 Most Depended On Python Tokenizer Open Source Projects

Mail Parser ⭐ 311

Tokenizer for raw mails

dependent packages 20total releases 84latest release February 26, 2021most recent commit 2 years ago

Segtok ⭐ 151

Segtok v2 is here: https://github.com/fnl/syntok -- A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic features.

dependent packages 16total releases 23latest release December 15, 2021most recent commit 2 years ago

Hazm ⭐ 1,104

Persian NLP Toolkit

dependent packages 13total releases 20latest release October 01, 2023most recent commit 11 days ago

Syntok ⭐ 158

Text tokenization and sentence segmentation (segtok v2)

dependent packages 12total releases 16latest release March 12, 2022most recent commit 2 years ago

Stanfordnlp ⭐ 101

[Deprecated] This library has been renamed to "Stanza". Latest development at: https://github.com/stanfordnlp/stanza

dependent packages 11total releases 5latest release May 16, 2019most recent commit 7 months ago

Soynlp ⭐ 801

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

dependent packages 9total releases 33latest release August 25, 2019most recent commit 2 years ago

Natasha ⭐ 1,085

Solves basic Russian NLP tasks, API for lower level Natasha projects

dependent packages 9total releases 19latest release July 24, 2023most recent commit 7 months ago

Simplemma ⭐ 100

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

dependent packages 9total releases 14latest release January 20, 2023most recent commit 7 months ago

Farasapy ⭐ 43

A Python implementation of Farasa toolkit

dependent packages 8total releases 14latest release August 15, 2021most recent commit 3 years ago

Nagisa ⭐ 365

A Japanese tokenizer based on recurrent neural networks

dependent packages 7total releases 22latest release July 30, 2023most recent commit 3 months ago

The 10 Latest Releases In Python Tokenizer Open Source Projects

Cereja ⭐ 21

Cereja is a bundle of useful functions we don't want to rewrite and .. just pure fun!

dependent packages 4total releases 125latest release December 01, 2023most recent commit 4 months ago

Gretel Synthetics ⭐ 490

Synthetic data generators for structured and unstructured text, featuring differentially private learning.

dependent packages 1total releases 53latest release November 28, 2023most recent commit 3 months ago

Dadmatools ⭐ 142

DadmaTools is a Persian NLP tools developed by Dadmatech Co.

total releases 66latest release November 25, 2023most recent commit 3 months ago

Spacy Experimental ⭐ 87

🧪 Cutting-edge experimental spaCy components and features

dependent packages 3total releases 9latest release November 02, 2023most recent commit 6 months ago

Python Ucto ⭐ 29

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).

dependent packages 1total releases 22latest release October 31, 2023most recent commit 6 months ago

Text2text ⭐ 268

Text2Text: Crosslingual NLP/G toolkit

total releases 134latest release October 21, 2023most recent commit 3 months ago

Hazm ⭐ 1,104

Persian NLP Toolkit

dependent packages 13total releases 20latest release October 01, 2023most recent commit 11 days ago

Ud Kanbun ⭐ 59

Tokenizer POS-tagger and Dependency-parser for Classical Chinese

dependent packages 2total releases 249latest release September 25, 2023most recent commit 4 months ago

Unidic2ud ⭐ 27

Tokenizer POS-tagger Lemmatizer and Dependency-parser for modern and contemporary Japanese

dependent packages 3total releases 208latest release September 25, 2023most recent commit 4 months ago

Somajo ⭐ 128

A tokenizer and sentence splitter for German and English web and social media texts.

dependent packages 6total releases 55latest release September 23, 2023most recent commit 4 months ago

Popular Programming Languages Categories