Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Golang Tokenizer Open Source Projects

Open source projects categorized as Golang Tokenizer
Categories > Compilers > Tokenizer
Kagome ⭐ 769
Self-contained Japanese Morphological Analyzer written in pure Go
dependent packages 27total releases 74most recent commit 3 months ago
Goro ⭐ 673
PHP in Go
total releases 1most recent commit 3 months ago
Tokenmonster ⭐ 399
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
total releases 15most recent commit 5 months ago
Sentences ⭐ 391
A multilingual command line sentence tokenizer in Golang
dependent packages 127total releases 7most recent commit 6 months ago
Lexmachine ⭐ 370
Lex machinary for go.
dependent packages 45total releases 6most recent commit 2 years ago
Shield ⭐ 131
Bayesian text classifier with flexible tokenizers and storage backends for Go
total releases 1most recent commit 4 years ago
Lingo ⭐ 102
package lingo provides the data structures and algorithms required for natural language processing
dependent packages 1total releases 1most recent commit 4 years ago
Jargon ⭐ 98
Tokenizers and lemmatizers for Go
total releases 30most recent commit 2 years ago
Tokenizer ⭐ 75
NLP tokenizers written in Go language
dependent packages 12total releases 24most recent commit 8 months ago
Go Gpt 3 Encoder ⭐ 74
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
total releases 3most recent commit 6 months ago
Howtowriteacompiler ⭐ 63
How to write a compiler from scratch in 30 minutes
most recent commit 5 years ago
Agency ⭐ 59
A fast user agent string parser for Go.
total releases 1most recent commit 7 years ago
Tokenizer ⭐ 57
Tokenizer (lexer) for golang
dependent packages 1total releases 4most recent commit 6 months ago
Refmt ⭐ 45
Object mapping for golang.
dependent packages 315total releases 11most recent commit 5 months ago
Uax29 ⭐ 35
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and graphemes.
dependent packages 6total releases 40most recent commit 6 months ago
Nlpt ⭐ 35
Natural Language Processing Toolkit written in Go (DEPRECATED see individual packages prefixed nlpt-)
total releases 1most recent commit 8 years ago
Cwsharp Go ⭐ 30
cwsharp-go, Golang中文分词库，支持多种分词模式，支持自定义字典和扩展。
total releases 1most recent commit 6 years ago
Goselect ⭐ 28
SQL like 'select' interface for files
most recent commit 2 years ago
Bleve Sego Tokenizer ⭐ 24
a Chinese tokenizer for bleve, using sego as the segmenter
total releases 1most recent commit 9 years ago
Tokenizer ⭐ 19
Package tokenizer provides encoding for tokens that can carry user data.
total releases 1most recent commit 7 years ago
Go Khaiii ⭐ 19
most recent commit 2 years ago
Shamoji ⭐ 13
The shamoji (杓文字) is a word filtering package
total releases 5most recent commit a year ago
Nihongo ⭐ 12
Japanese utilities for Go
total releases 1most recent commit 6 years ago
The 10 Most Depended On Golang Tokenizer Open Source Projects
Refmt ⭐ 45
Object mapping for golang.
dependent packages 315total releases 11latest release December 10, 2022most recent commit 5 months ago
Sentences ⭐ 391
A multilingual command line sentence tokenizer in Golang
dependent packages 127total releases 7latest release May 26, 2021most recent commit 6 months ago
Lexmachine ⭐ 370
Lex machinary for go.
dependent packages 45total releases 6latest release July 15, 2022most recent commit 2 years ago
Kagome ⭐ 769
Self-contained Japanese Morphological Analyzer written in pure Go
dependent packages 27total releases 74latest release September 27, 2023most recent commit 3 months ago
Tokenizer ⭐ 75
NLP tokenizers written in Go language
dependent packages 12total releases 24latest release June 25, 2023most recent commit 8 months ago
Uax29 ⭐ 35
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and graphemes.
dependent packages 6total releases 40latest release May 26, 2023most recent commit 6 months ago
Lingo ⭐ 102
package lingo provides the data structures and algorithms required for natural language processing
dependent packages 1total releases 1latest release September 18, 2020most recent commit 4 years ago
Tokenizer ⭐ 57
Tokenizer (lexer) for golang
dependent packages 1total releases 4latest release October 23, 2023most recent commit 6 months ago
Go Tokenizer ⭐ 11
A Text Tokenizer library for Golang
dependent packages 1total releases 3latest release November 02, 2021most recent commit 4 months ago
The 10 Latest Releases In Golang Tokenizer Open Source Projects
Zearch ⭐ 9
ragecoded code search with json endpoint
latest release November 23, 2023most recent commit 6 years ago
Tokenizer ⭐ 57
Tokenizer (lexer) for golang
dependent packages 1total releases 4latest release October 23, 2023most recent commit 6 months ago
Kagome ⭐ 769
Self-contained Japanese Morphological Analyzer written in pure Go
dependent packages 27total releases 74latest release September 27, 2023most recent commit 3 months ago
Tokenmonster ⭐ 399
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
total releases 15latest release September 05, 2023most recent commit 5 months ago
Tokenizer ⭐ 75
NLP tokenizers written in Go language
dependent packages 12total releases 24latest release June 25, 2023most recent commit 8 months ago
Assocentity ⭐ 11
Package assocentity returns the mean distance from tokens to an entity and its synonyms
total releases 25latest release May 27, 2023most recent commit a year ago
Uax29 ⭐ 35
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and graphemes.
dependent packages 6total releases 40latest release May 26, 2023most recent commit 6 months ago
Go Gpt 3 Encoder ⭐ 74
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
total releases 3latest release February 27, 2023most recent commit 6 months ago
Refmt ⭐ 45
Object mapping for golang.
dependent packages 315total releases 11latest release December 10, 2022most recent commit 5 months ago
Shamoji ⭐ 13
The shamoji (杓文字) is a word filtering package
total releases 5latest release December 01, 2022most recent commit a year ago
Popular Programming Languages Categories
Python
Javascript
Java
Typescript
C Plus Plus
Php
Golang
C
C Sharp
Shell
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.