Awesome Open Source

Programming Languages

Search results for unicode tokenizer

12 search results found

Making text a first-class citizen in TensorFlow.

The fast scanner generator for Java™ with full Unicode support

Tokenizer ⭐ 224

Fast and customizable text tokenization library with BPE and SentencePiece support

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and graphemes.

Natural Language Processing Toolkit written in Go (DEPRECATED see individual packages prefixed nlpt-)

Sqlite3 Unicodesn ⭐ 29

SQLite unicode full-text-search tokenizer with Snowball stemming

Unicode Tokenizer ⭐ 20

Unicode Tokenizer following the Unicode Line Breaking algorithm

Pyidaungsu ⭐ 19

Python library for Myanmar language

Greeb is a simple Unicode-aware regexp-based tokenizer.

Tokenizer ⭐ 11

Natural Language Tokenizer

A natural language tokenizer

Cjk Tokenizer ⭐ 5

Related Searches

Character Unicode (849)

Python Unicode (840)

Javascript Unicode (743)

Python Tokenizer (341)

C Plus Plus Unicode (337)

1-12 of 12 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.