Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

Sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.
Stars
8,851
License
apache-2.0
Open Issues
32
Most Recent Commit
5 months ago
Programming Language
C++
Monthly Downloads
pypi sentencepiece Downloads
Dependent Repos
120
Dependent Packages
787
Total Releases
34
Latest Release
May 02, 2023
Categories
Repo
Alternatives To Sentencepiece
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Sentencepiece8,851pypi sentencepiece} Downloads1207875 months ago34May 02, 202332apache-2.0C++
Unsupervised text tokenizer for Neural Network-based text generation.
Bitextor260
9 months ago4gpl-3.0Python
Bitextor generates translation memories from multilingual websites
Sentencepiece Jni27
a year ago3mitC++
Java JNI wrapper for SentencePiece: unsupervised text tokenizer for Neural Network-based text generation.
Alternatives To Sentencepiece
Select To Compare
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34most recent commit 5 months agopypi sentencepiece} Downloads


Suggest An Alternative To sentencepiece
Alternative Project Comparisons
Sentencepiece vs Bitextor
Sentencepiece vs Sentencepiece Jni
Popular Tokenizer Projects
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 5 months agopypi tokenizers} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8latest release November 20, 2023most recent commit 7 months agopackagist theseer/tokenizer} Downloads
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141latest release November 11, 2023most recent commit 5 months agonpm file-type} Downloads
Popular Neural Machine Translation Projects
Opennmt Py ⭐ 6,449
Open Source Neural Machine Translation and (Large) Language Models in PyTorch
dependent packages 8total releases 35latest release November 02, 2023most recent commit 5 months agopypi OpenNMT-py} Downloads
Ctranslate2 ⭐ 2,437
Fast inference engine for Transformer models
dependent packages 23total releases 103latest release December 05, 2023most recent commit 5 months agopypi ctranslate2} Downloads
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Jit
Ghc
Ida
Related Searches
C Plus Plus Segmentation
C Plus Plus Natural Language Processing
C Plus Plus Tokenizer
Natural Language Processing Tokenizer
C Plus Plus Whitespace
Natural Language Processing Segmentation
Natural Language Processing Neural Machine Translation
C Plus Plus Word Segmentation
Natural Language Processing Word Segmentation
Segmentation Tokenizer

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
C Plus Plus
Natural Language Processing
Segmentation
Tokenizer
Neural Machine Translation
Word Segmentation
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.