Tokenizer Alternatives

Natural Language Tokenizer
Suggest Alternative
Alternatives To liuzl/tokenizer
Project Name Stars Downloads Repos Using This Packages Using This Most Recent Commit Total Releases Latest Release Open Issues License Language
catalyst-team/catalyst 3,151 19 19 almost 3 years ago 108 April 29, 2022 6 apache-2.0 Python
Accelerated deep learning R&D
PaddlePaddle/awesome-DeepLearning 2,670 0 0 over 2 years ago 0 462 apache-2.0 Jupyter Notebook
深度学习入门课、资深课、特色课、学术案例、产业实践案例、深度学习知识百科及面试题库The course, case and knowledge of Deep Learning and AI
go-ego/gse 2,352 14 21 over 2 years ago 82 January 16, 2023 12 apache-2.0 Go
Go efficient multilingual NLP and text segmentation; support English, Chinese, Japanese and others.
rockingdingo/deepnlp 1,311 0 0 about 8 years ago 8 January 01, 2018 44 mit Python
Deep Learning NLP Pipeline implemented on Tensorflow
fukuball/jieba-php 1,193 29 8 almost 4 years ago 17 August 15, 2022 21 mit PHP
"結巴"中文分詞:做最好的 PHP 中文分詞、中文斷詞組件。 / "Jieba" (Chinese for "to stutter") Chinese text segmentation: built to be the best PHP Chinese word segmentation module.
natasha/natasha 1,085 3 9 over 2 years ago 19 July 24, 2023 24 mit Python
Solves basic Russian NLP tasks, API for lower level Natasha projects
SeanLee97/xmnlp 940 1 0 over 3 years ago 14 November 12, 2022 3 apache-2.0 Python
xmnlp:提供中文分词, 词性标注, 命名体识别,情感分析,文本纠错,文本转拼音,文本摘要,偏旁部首,句子表征及文本相似度计算等功能
PyThaiNLP/pythainlp 902 24 51 over 2 years ago 101 November 26, 2023 35 apache-2.0 Python
Thai Natural Language Processing in Python.
messense/jieba-rs 585 5 15 almost 3 years ago 40 July 16, 2023 9 mit Rust
The Jieba Chinese Word Segmentation Implemented in Rust
cbaziotis/ekphrasis 583 7 0 over 3 years ago 54 May 17, 2022 18 mit Python
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
Alternatives To liuzl/tokenizer
Select To Compare


Alternative Project Comparisons
Popular Natural Language Processing Projects
Popular Segmentation Projects
Popular Machine Learning Categories
Related Searches
Get A Weekly Email With Trending Projects
No Spam. Unsubscribe easily at any time.
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2026 Awesome Open Source.  All rights reserved.