Awesome Open Source

Programming Languages

Search results for unicode tokenizer

19 search results found

Making text a first-class citizen in TensorFlow.

The fast scanner generator for Java™ with full Unicode support

Tokenizer ⭐ 207

Fast and customizable text tokenization library with BPE and SentencePiece support

Natural Language Processing Toolkit written in Go (DEPRECATED see individual packages prefixed nlpt-)

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and graphemes.

Sqlite3 Unicodesn ⭐ 29

SQLite unicode full-text-search tokenizer with Snowball stemming

Unicode Tokenizer ⭐ 20

Unicode Tokenizer following the Unicode Line Breaking algorithm

Pyidaungsu ⭐ 19

Python library for Myanmar language

Greeb is a simple Unicode-aware regexp-based tokenizer.

Tokenizer ⭐ 11

Natural Language Tokenizer

A natural language tokenizer

Cjk Tokenizer ⭐ 5

Cjk Mmseg ⭐ 3

An implementation of Chinese / CJK MMSEG tokenizer algorithm with Sogou Dictionary. MMSEG 中文分词工具实现。

Overview Js Tokenizer ⭐ 3

Splits text into tokens in as many languages as possible

Nlp Project Hindi Language Tokenizer ⭐ 2

This package tends to implement a Tokenizer and a stemmer for Hindi language.

Cjk Tokenzier ⭐ 2

A unigram CJK tokenizer

natural language tokenization using various types of tokenizer, including a state function lexer, unicode matcher, and basic white space splitter

Elasticsearch Analysis Customwordboundarystandard ⭐ 1

An extension to Lucene's Standard Tokenizer with custom word boundaries

Unicode operators preprocessing for Python.

Related Searches

Character Unicode (849)

Python Unicode (836)

Javascript Unicode (740)

Python Tokenizer (341)

C Plus Plus Unicode (337)

1-19 of 19 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2023 Awesome Open Source. All rights reserved.