Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

Twokenize

Python standalone tokenizer
Stars
12
License
No license specified
Open Issues
1
Most Recent Commit
9 years ago
Programming Language
Python
Categories
Repo
Alternatives To Twokenize
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Somajo128pypi SoMaJo} Downloads265 months ago55September 23, 20233gpl-3.0Python
A tokenizer and sentence splitter for German and English web and social media texts.
Twokenize12
9 years ago1Python
Python standalone tokenizer
Alternatives To Twokenize
Select To Compare
Somajo ⭐ 128
A tokenizer and sentence splitter for German and English web and social media texts.
dependent packages 6total releases 55most recent commit 5 months agopypi SoMaJo} Downloads
Twokenize ⭐ 12
Python standalone tokenizer
most recent commit 9 years ago


Suggest An Alternative To twokenize
Alternative Project Comparisons
Twokenize vs Somajo
Popular Tokenizer Projects
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 4 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 4 months agopypi tokenizers} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8latest release November 20, 2023most recent commit 5 months agopackagist theseer/tokenizer} Downloads
Popular Social Projects
Social Analyzer ⭐ 10,841
API, CLI, and Web App for analyzing and finding a person's profile in 1000 social media \ websites
dependent packages 1total releases 1latest release June 25, 2021most recent commit 4 months agonpm social-analyzer} Downloads
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Jit
Ghc
Ida

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Social
Tokenizer
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.