Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

Nim Tokenizer

Implementation of a simple BPE tokenizer, but in Nim
Alternatives To Nim Tokenizer
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Transformers.jl479
5 months ago23mitJulia
Julia Implementation of Transformer models
Deepcut319pypi deepcut} Downloads734 years ago30November 06, 2019mitPython
A Thai word tokenization library using Deep Neural Network
Rust Tokenizers232cargo rust_tokenizers} Downloads78 months ago34October 01, 20234apache-2.0Rust
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
Doc2vec Api92
2 years ago1lgpl-2.1Python
document embedding and machine learning script for beginners
Tokenizer75129 months ago24June 25, 20234apache-2.0Go
NLP tokenizers written in Go language
Vietnamese Electra59
3 years ago2Jupyter Notebook
Electra pre-trained model using Vietnamese corpus
Edunlp42
4 months ago8apache-2.0Python
A library for advanced Natural Language Processing towards multi-modal educational items.
Transformers Embedder34
7 months ago52October 01, 2021Python
A Word Level Transformer layer based on PyTorch and 🤗 Transformers.
Tokenizer27pypi tokenizer} Downloads356 months ago57August 11, 20235otherPython
A tokenizer for Icelandic text
Flask Deep Learning Nlp Api23
4 years agoPython
Flask API to productize a document classification model. Classification model was built using Keras with tensorflow backend
Alternatives To Nim Tokenizer
Select To Compare
Deepcut ⭐ 319
A Thai word tokenization library using Deep Neural Network
dependent packages 3total releases 30most recent commit 4 years agopypi deepcut} Downloads
Tokenizer ⭐ 75
NLP tokenizers written in Go language
dependent packages 12total releases 24most recent commit 9 months ago
Tokenizer ⭐ 27
A tokenizer for Icelandic text
dependent packages 5total releases 57most recent commit 6 months agopypi tokenizer} Downloads


Suggest An Alternative To nim-tokenizer
Alternative Project Comparisons
Nim Tokenizer vs Transformers.jl
Nim Tokenizer vs Deepcut
Nim Tokenizer vs Rust Tokenizers
Nim Tokenizer vs Doc2vec Api
Nim Tokenizer vs Tokenizer
Nim Tokenizer vs Vietnamese Electra
Nim Tokenizer vs Edunlp
Nim Tokenizer vs Transformers Embedder
Nim Tokenizer vs Tokenizer
Nim Tokenizer vs Flask Deep Learning Nlp Api
Popular Tokenizer Projects
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 4 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 4 months agopypi tokenizers} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141latest release November 11, 2023most recent commit 4 months agonpm file-type} Downloads
Popular Deep Learning Projects
Tensorflow ⭐ 180,196
An Open Source Machine Learning Framework for Everyone
dependent packages 78total releases 46latest release October 23, 2019most recent commit 4 months ago
Transformers ⭐ 124,049
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
dependent packages 2,484total releases 125latest release November 15, 2023most recent commit a month agopypi transformers} Downloads
Stable Diffusion Webui ⭐ 118,856
Stable Diffusion web UI
total releases 2latest release January 17, 2022most recent commit 4 months ago
Opencv ⭐ 75,512
Open Source Computer Vision Library
dependent packages 37total releases 29latest release February 15, 2023most recent commit a month ago
Pytorch ⭐ 74,794
Tensors and Dynamic neural networks in Python with strong GPU acceleration
dependent packages 8,272total releases 39latest release November 15, 2023most recent commit 4 months agopypi torch} Downloads
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Jit
Ghc
Ida
Related Searches
Deep Learning Artificial Intelligence
Deep Learning Language Model
Artificial Intelligence Language Model
Tokenizer Language Model
Deep Learning Tokenizer
Artificial Intelligence Nim
Deep Learning Nim
Artificial Intelligence Tokenizer

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Deep Learning
Artificial Intelligence
Nim
Language Model
Tokenizer
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.