Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

Berserker

Berserker - BERt chineSE woRd toKenizER
Alternatives To Berserker
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Gpt2 Chinese7,249
4 months ago105mitPython
Chinese version of GPT2 training code, using BERT tokenizer.
Ckip Transformers439
a year ago1gpl-3.0Python
CKIP Transformers
Simple411
5 months ago10mitC++
支持中文和拼音的 SQLite fts5 全文搜索扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
Segmentit208npm segmentit} Downloads16a year ago17December 22, 20196mitJavaScript
任何 JS 环境可用的中文分词包，fork from leizongmin/node-segment
Sqlitesubstringsearch76
8 years agoC
An open source tokenizer which supports fast substring search with sqlite FTS (full text search)
Cang Jie65cargo cang-jie} Downloads66 months ago20November 04, 2023mitRust
Chinese tokenizer for tantivy, based on jieba-rs
Ud Kanbun59pypi udkanbun} Downloads124 months ago249September 25, 2023mitPython
Tokenizer POS-tagger and Dependency-parser for Classical Chinese
Rasa_chinese46
3 years ago1apache-2.0Python
rasa_chinese 专门针对中文语言的 rasa 组件扩展包，提供了许多针对中文语言的组件
Chinese Tokenizer39npm chinese-tokenizer} Downloads374 years ago11June 05, 20192mitJavaScript
Tokenizes Chinese texts into words.
Pnlp25pypi pnlp} Downloads113 months ago38December 25, 2022apache-2.0Python
NLP预/后处理工具。
Alternatives To Berserker
Select To Compare
Ckip Transformers ⭐ 439
CKIP Transformers
most recent commit a year ago
Segmentit ⭐ 208
任何 JS 环境可用的中文分词包，fork from leizongmin/node-segment
dependent packages 6total releases 17most recent commit a year agonpm segmentit} Downloads
Cang Jie ⭐ 65
Chinese tokenizer for tantivy, based on jieba-rs
dependent packages 6total releases 20most recent commit 6 months agocargo cang-jie} Downloads
Ud Kanbun ⭐ 59
Tokenizer POS-tagger and Dependency-parser for Classical Chinese
dependent packages 2total releases 249most recent commit 4 months agopypi udkanbun} Downloads
Chinese Tokenizer ⭐ 39
Tokenizes Chinese texts into words.
dependent packages 7total releases 11most recent commit 4 years agonpm chinese-tokenizer} Downloads
Pnlp ⭐ 25
NLP预/后处理工具。
dependent packages 1total releases 38most recent commit 3 months agopypi pnlp} Downloads


Suggest An Alternative To berserker
Alternative Project Comparisons
Berserker vs Gpt2 Chinese
Berserker vs Ckip Transformers
Berserker vs Simple
Berserker vs Segmentit
Berserker vs Sqlitesubstringsearch
Berserker vs Cang Jie
Berserker vs Ud Kanbun
Berserker vs Rasa_chinese
Berserker vs Chinese Tokenizer
Berserker vs Pnlp
Popular Tokenizer Projects
Sentencepiece ⭐ 8,851
Unsupervised text tokenizer for Neural Network-based text generation.
dependent packages 787total releases 34latest release May 02, 2023most recent commit 3 months agopypi sentencepiece} Downloads
Tokenizers ⭐ 8,056
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
dependent packages 362total releases 85latest release November 14, 2023most recent commit 3 months agopypi tokenizers} Downloads
Php Token Stream ⭐ 6,457
Wrapper around PHP's tokenizer extension.
dependent packages 188total releases 36latest release November 30, 2020most recent commit 3 years agopackagist phpunit/php-token-stream} Downloads
Tokenizer ⭐ 5,084
A small library for converting tokenized PHP source code into XML (and potentially other formats)
dependent packages 11total releases 8latest release November 20, 2023most recent commit 5 months agopackagist theseer/tokenizer} Downloads
File Type ⭐ 3,366
Detect the file type of a Buffer/Uint8Array/ArrayBuffer
dependent packages 1,894total releases 141latest release November 11, 2023most recent commit 4 months agonpm file-type} Downloads
Popular Chinese Projects
Howtocook ⭐ 57,819
程序员在家做饭方法指南。Programmer's guide about how to cook at home (Chinese only).
dependent packages 1total releases 4latest release July 16, 2022most recent commit 3 months agonpm how-to-cook} Downloads
Element ⭐ 53,857
A Vue.js 2.0 UI Toolkit for Web
dependent packages 4total releases 7latest release September 22, 2020most recent commit 3 months agonpm qj-element-ui} Downloads
Popular Compilers Categories
Llvm
Abstract Syntax Tree
Gcc
Clang
Emscripten
Wsl
Lexer
Jit
Ghc
Ida
Related Searches
Python Dataset
Python Tensorflow
Python Natural Language Processing
Python Chinese
Dataset Tensorflow
Dataset Natural Language Processing
Tensorflow Natural Language Processing
Python Bert
Natural Language Processing Bert
Python Tokenizer

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Dataset
Tensorflow
Natural Language Processing
Chinese
Sequence To Sequence
Tokenizer
Chinese Nlp
Chinese Word Segmentation
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.