Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

Icu Tokenizer

ICU based universal language tokenizer
Stars
13
License
mit
Open Issues
1
Most Recent Commit
2 years ago
Programming Language
Python
Total Releases
1
Latest Release
June 18, 2020
Categories
Site
Repo
Alternatives To Icu Tokenizer
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Tokenizer224pypi pyonmttok} Downloads1559 months ago68January 11, 20232mitC++
Fast and customizable text tokenization library with BPE and SentencePiece support
Icu Tokenizer13
2 years ago1June 18, 20201mitPython
ICU based universal language tokenizer
Alternatives To Icu Tokenizer
Select To Compare
Tokenizer ⭐ 224
Fast and customizable text tokenization library with BPE and SentencePiece support
dependent packages 5total releases 68most recent commit 9 months agopypi pyonmttok} Downloads
Icu Tokenizer ⭐ 13
ICU based universal language tokenizer
total releases 1most recent commit 2 years ago


Suggest An Alternative To ICU-tokenizer
Alternative Project Comparisons
Icu Tokenizer vs Tokenizer
Popular Tokenizer Projects
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 3 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 3 months agopypi tokenizers} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8latest release November 20, 2023most recent commit 5 months agopackagist theseer/tokenizer} Downloads
Popular Icu Projects
996.leave ⭐ 6,090
逃离996
most recent commit 3 months ago
Js Lingui ⭐ 4,087
🌍 📖 A readable, automated, and optimized (3 kb) internationalization for JavaScript
dependent packages 145total releases 100latest release December 01, 2023most recent commit 3 months agonpm @lingui/macro} Downloads
Icu ⭐ 2,384
The home of the ICU project source code.
dependent packages 528total releases 61latest release October 30, 2023most recent commit 3 months ago
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Jit
Ghc
Ida

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Language
Tokenizer
Icu
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.