Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Word Segmentation Open Source Projects

Open source projects categorized as Word Segmentation
Categories > Machine Learning > Word Segmentation
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34most recent commit 3 months agopypi sentencepiece} Downloads
Pkuseg Python ⭐ 6,001
pkuseg多领域中文分词工具; The pkuseg toolkit for multi-domain Chinese word segmentation
dependent packages 8total releases 22most recent commit a year agopypi pkuseg} Downloads
Lac ⭐ 3,644
百度NLP：分词，词性标注，命名实体识别，词重要性
dependent packages 12total releases 15most recent commit 3 years agopypi LAC} Downloads
Symspell ⭐ 3,022
SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm
dependent packages 3total releases 8most recent commit a month agonuget SymSpell} Downloads
Subword Nmt ⭐ 1,937
Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
dependent packages 18total releases 8most recent commit 2 years agopypi subword-nmt} Downloads
Youtokentome ⭐ 943
Unsupervised text tokenizer focused on computational efficiency
dependent packages 17total releases 14most recent commit a month agopypi youtokentome} Downloads
Pythainlp ⭐ 902
Thai Natural Language Processing in Python.
dependent packages 51total releases 101most recent commit 3 months agopypi pythainlp} Downloads
Fasthan ⭐ 730
fastHan是基于fastNLP与pytorch实现的中文自然语言处理工具，像spacy一样调用方
total releases 9most recent commit 5 months ago
Symspellpy ⭐ 693
Python port of SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm
dependent packages 25total releases 22most recent commit 10 months agopypi symspellpy} Downloads
Jieba Rs ⭐ 585
The Jieba Chinese Word Segmentation Implemented in Rust
dependent packages 15total releases 40most recent commit 9 months agocargo jieba-rs} Downloads
Ekphrasis ⭐ 583
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
total releases 54most recent commit 2 years ago
M3tl ⭐ 544
BERT for Multitask Learning
most recent commit a year ago
Vncorenlp ⭐ 472
A Vietnamese natural language processing toolkit (NAACL 2018)
most recent commit a year ago
Ckip Transformers ⭐ 439
CKIP Transformers
most recent commit a year ago
Nagisa ⭐ 365
A Japanese tokenizer based on recurrent neural networks
dependent packages 7total releases 22most recent commit 3 months agopypi nagisa} Downloads
Jumanpp ⭐ 334
Juman++ (a Morphological Analyzer Toolkit)
most recent commit a year ago
Kiwi ⭐ 330
Kiwi(지능형 한국어 형태소 분석기)
most recent commit 4 months ago
Adaseq ⭐ 295
AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models
total releases 11most recent commit 5 months ago
Pycantonese ⭐ 290
Cantonese Linguistics and NLP
total releases 24most recent commit a year ago
Python Wordsegment ⭐ 268
English word segmentation, written in pure-Python, and based on a trillion-word corpus.
most recent commit 4 years ago
Multi Criteria Cws ⭐ 260
Simple Solution for Multi-Criteria Chinese Word Segmentation
most recent commit 5 years ago
Yaha ⭐ 258
yaha
total releases 3most recent commit 6 years ago
Pytorch Nlu ⭐ 226
Pytorch-NLU，一个中文文本分类、序列标注工具包，支持中文长文本、短文本的多类、多标签分类任 Ptorch NLU, a Chinese text classification and sequence annotation toolkit, supports multi class and multi label classification tasks of Chinese long text and short text, and supports sequence annotation tasks such as Chinese named entity recognition, part of speech tagging and word segmentation.
total releases 2most recent commit 9 months ago
The 10 Most Depended On Word Segmentation Open Source Projects
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 3 months agopypi sentencepiece} Downloads
Pythainlp ⭐ 902
Thai Natural Language Processing in Python.
dependent packages 51total releases 101latest release November 26, 2023most recent commit 3 months agopypi pythainlp} Downloads
Subword Nmt ⭐ 1,937
Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
dependent packages 18total releases 8latest release December 08, 2021most recent commit 2 years agopypi subword-nmt} Downloads
Youtokentome ⭐ 943
Unsupervised text tokenizer focused on computational efficiency
dependent packages 17total releases 14latest release February 12, 2020most recent commit a month agopypi youtokentome} Downloads
Jieba Rs ⭐ 585
The Jieba Chinese Word Segmentation Implemented in Rust
dependent packages 15total releases 40latest release July 16, 2023most recent commit 9 months agocargo jieba-rs} Downloads
Lac ⭐ 3,644
百度NLP：分词，词性标注，命名实体识别，词重要性
dependent packages 12total releases 15latest release May 25, 2021most recent commit 3 years agopypi LAC} Downloads
Hanzi Tools ⭐ 58
Converts from Chinese characters to pinyin, between simplified and traditional, and does word segmentation.
dependent packages 9total releases 31latest release December 16, 2021most recent commit 2 years agonpm hanzi-tools} Downloads
Pkuseg Python ⭐ 6,001
pkuseg多领域中文分词工具; The pkuseg toolkit for multi-domain Chinese word segmentation
dependent packages 8total releases 22latest release June 19, 2020most recent commit a year agopypi pkuseg} Downloads
Nagisa ⭐ 365
A Japanese tokenizer based on recurrent neural networks
dependent packages 7total releases 22latest release July 30, 2023most recent commit 3 months agopypi nagisa} Downloads
The 10 Latest Releases In Word Segmentation Open Source Projects
Pythainlp ⭐ 902
Thai Natural Language Processing in Python.
dependent packages 51total releases 101latest release November 26, 2023most recent commit 3 months agopypi pythainlp} Downloads
Kiwipiepy ⭐ 182
Python API for Kiwi
dependent packages 4total releases 42latest release November 20, 2023most recent commit 5 months agopypi kiwipiepy} Downloads
Adaseq ⭐ 295
AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models
total releases 11latest release November 15, 2023most recent commit 5 months ago
Fasthan ⭐ 730
fastHan是基于fastNLP与pytorch实现的中文自然语言处理工具，像spacy一样调用方
total releases 9latest release November 11, 2023most recent commit 5 months ago
Toiro ⭐ 110
A comparison tool of Japanese tokenizers
total releases 8latest release July 31, 2023most recent commit 9 months ago
Nagisa ⭐ 365
A Japanese tokenizer based on recurrent neural networks
dependent packages 7total releases 22latest release July 30, 2023most recent commit 3 months agopypi nagisa} Downloads
Jieba Rs ⭐ 585
The Jieba Chinese Word Segmentation Implemented in Rust
dependent packages 15total releases 40latest release July 16, 2023most recent commit 9 months agocargo jieba-rs} Downloads
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 3 months agopypi sentencepiece} Downloads
Symspellcpppy ⭐ 35
Fast SymSpell written in c++ and exposes to python via pybind11
total releases 18latest release November 19, 2022most recent commit a year ago
Sentencepiece ⭐ 21
R package for Byte Pair Encoding / Unigram modelling based on Sentencepiece
dependent packages 1total releases 6latest release November 13, 2022most recent commit a year ago
Popular Machine Learning Categories
Deep Learning
Machine Learning
Pytorch
Tensorflow
Natural Language Processing
Neural Network
Neural
Computer Vision
Convolutional Neural Networks
Opencv
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.