Awesome Open Source

Programming Languages

Search results for tokenizer bpe

8 search results found

Gpt Tokenizer ⭐ 309

JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.

Tokenizer ⭐ 224

Fast and customizable text tokenization library with BPE and SentencePiece support

Tiktoken Rs ⭐ 163

Ready-made tokenizer library for working with GPT and tiktoken

Go Gpt 3 Encoder ⭐ 74

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

Gpt3 Tokenizer ⭐ 15

GPT3 encoder & decoder tool written in Swift

Phishytics Machine Learning For Phishing ⭐ 12

Machine Learning for Phishing Website Detection

Llmsharp Tokenizers ⭐ 9

BPE Tokenizer implementations in C# for Anthropic, OpenAI LLM offerings

Kotlin multiplatform BPE tokenizer library for OpenAI models

Related Searches

Python Tokenizer (341)

Token Tokenizer (219)

Natural Language Processing Tokenizer (191)

C Tokenizer (69)

C Plus Plus Tokenizer (62)

Golang Tokenizer (57)

C Sharp Tokenizer (56)

Typescript Tokenizer (55)

Rust Tokenizer (45)

Tokenizer Tokenization (43)

1-8 of 8 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.