Project Name	Stars	Repos Using This	Packages Using This	Most Recent Commit	Total Releases	Latest Release	Open Issues	License	Language
Microtokenizer	119	3	1	3 years ago	53	September 28, 2021		mit	Python
一个微型&算法全面的中文分词引擎 \| A micro tokenizer for Chinese
Chinese_tokenizer_benchmark	10			6 years ago			1		Python
中文分词软件基准测试 \| Chinese tokenizer benchmark

Alternatives To Chinese_tokenizer_benchmark

Select To Compare

Microtokenizer ⭐ 119

一个微型&算法全面的中文分词引擎 | A micro tokenizer for Chinese

dependent packages 1total releases 53most recent commit 3 years ago

Chinese_tokenizer_benchmark ⭐ 10

中文分词软件基准测试 | Chinese tokenizer benchmark

most recent commit 6 years ago

Suggest An Alternative To Chinese_tokenizer_benchmark

Alternative Project Comparisons

Chinese_tokenizer_benchmark vs Microtokenizer

Popular Tokenizer Projects

Sentencepiece ⭐ 8,851

Unsupervised text tokenizer for Neural Network-based text generation.

dependent packages 787total releases 34latest release May 02, 2023most recent commit 3 months ago

Tokenizers ⭐ 8,056

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

dependent packages 362total releases 85latest release November 14, 2023most recent commit 3 months ago

Gpt2 Chinese ⭐ 7,249

Chinese version of GPT2 training code, using BERT tokenizer.

most recent commit 4 months ago

Php Token Stream ⭐ 6,457

Wrapper around PHP's tokenizer extension.

dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years ago

packagist phpunit/php-token-stream} Downloads

Tokenizer ⭐ 5,084

A small library for converting tokenized PHP source code into XML (and potentially other formats)

dependent packages 11total releases 8latest release November 20, 2023most recent commit 5 months ago

Popular Dag Projects

Airflow ⭐ 34,299

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

dependent packages 320total releases 169latest release November 27, 2023most recent commit 10 days ago

Argo Workflows ⭐ 14,264

Workflow Engine for Kubernetes

dependent packages 51total releases 449latest release November 27, 2023most recent commit 6 hours ago

Workflow ⭐ 12,038

C++ Parallel Computing and Asynchronous Networking Framework

most recent commit 3 months ago

Micrograd ⭐ 6,997

A tiny scalar-valued autograd engine and a neural net library on top of it with PyTorch-like API

dependent packages 2total releases 1latest release April 18, 2020most recent commit 4 months ago

Ethminer ⭐ 5,886

Ethereum miner with OpenCL, CUDA and stratum support

most recent commit 6 months ago

Popular Compilers Categories