Awesome Open Source
Awesome Open Source
Combined Topics
corpus
x
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
The Top 418 Corpus Open Source Projects on Github
Categories
>
Data Processing
>
Corpus
Nlp_chinese_corpus
⭐
5,454
大规模中文自然语言处理语料 Large Scale Chinese Corpus for NLP
Corpora
⭐
4,215
A collection of small corpuses of interesting data for the creation of bots and similar stuff.
Chinese Names Corpus
⭐
2,684
中文人名语料库。人名生成器。中文姓名,姓氏,名字,称呼,日本人名,翻译人名,英文人名。可用于中文分词、人名实体识别。
Awesome Deeplearning Resources
⭐
2,442
Deep Learning and deep reinforcement learning research papers and some codes
Clue
⭐
2,276
中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard
Weibo_terminater
⭐
2,265
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
Awesome Chatbot
⭐
1,556
Awesome Chatbot Projects,Corpus,Papers,Tutorials.Chinese Chatbot =>:
Dialog_corpus
⭐
1,487
用于训练中英文对话系统的语料库 Datasets for Training Chatbot System
Chatterbot Corpus
⭐
1,009
A multilingual dialog corpus
Company Names Corpus
⭐
891
公司名语料库。机构名语料库。公司简称,缩写,品牌词,企业名。可用于中文分词、机构名实体识别。
Insuranceqa Corpus Zh
⭐
843
🚁 保险行业语料库，聊天机器人
Seq2seq Chatbot
⭐
788
Chatbot in 200 lines of code using TensorLayer
Quanteda
⭐
682
An R package for the Quantitative Analysis of Textual Data
Cluedatasetsearch
⭐
674
搜索所有中文NLP数据集，附常用英文NLP数据集
Cluepretrainedmodels
⭐
536
高质量中文预训练模型集合：最先进大模型、最快小模型、相似度专门模型
Awesome Persian Nlp Ir
⭐
494
Curated List of Persian Natural Language Processing and Information Retrieval Tools and Resources
Bookcorpus
⭐
473
Crawl BookCorpus
Small Chinese Corpus
⭐
429
Some useful Chinese corpus datasets 中文语料小数据
Wordless
⭐
424
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
Weixin_public_corpus
⭐
408
微信公众号语料库
Fuzzdata
⭐
389
Fuzzing resources for feeding various fuzzers with input. 🔧
Chinese Nlp Corpus
⭐
378
Collections of Chinese NLP corpus
Cluecorpus2020
⭐
317
Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料
Trafilatura
⭐
290
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
Korpora
⭐
242
Korean corpus repository
Nlp_bahasa_resources
⭐
198
A Curated List of Dataset and Usable Library Resources for NLP in Bahasa Indonesia
Nlvr
⭐
194
Cornell NLVR and NLVR2 are natural language grounding datasets. Each example shows a visual input and a sentence describing it, and is annotated with the truth-value of the sentence.
Fakenewscorpus
⭐
184
A dataset of millions of news articles scraped from a curated list of data sources.
Cblue
⭐
180
中文医疗信息处理基准CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
Khcoder
⭐
154
KH Coder: for Quantitative Content Analysis or Text Mining
Ua Gec
⭐
150
UA-GEC: Grammatical Error Correction and Fluency Corpus for the Ukrainian Language
Malay Dataset
⭐
147
Text corpus for Bahasa Malaysia, https://malaya.readthedocs.io/en/latest/Dataset.html
Wp2txt
⭐
141
WP2TXT extracts plain text data from Wikipedia dump file (encoded in XML/compressed with Bzip2) stripping all the MediaWiki markups and other metadata.
Gossiping Chinese Corpus
⭐
136
PTT 八卦版問答中文語料
Awesome Hungarian Nlp
⭐
133
A curated list of NLP resources for Hungarian
Efaqa Corpus Zh
⭐
128
❤️Emotional First Aid Dataset, 心理咨询问答、聊天机器人语料库
Code Docstring Corpus
⭐
118
Preprocessed Python functions and docstrings for automated code documentation (code2doc) and automated code generation (doc2code) tasks.
Pubmed Rct
⭐
115
PubMed 200k RCT dataset: a large dataset for sequential sentence classification.
Lexicon Thai
⭐
105
คลังศัพท์ภาษาไทย
Prosody
⭐
104
Helsinki Prosody Corpus and A System for Predicting Prosodic Prominence from Text
Chatbot Training Corpus
⭐
103
总结了一些可以用作聊天机器人训练实作的文字语聊，包含中英文不同语言
Colibri Core
⭐
103
Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate and query pattern models.
Sejong Corpus
⭐
103
Korean sejong corpus download and simple analysis
Indonesian Nlp Resources
⭐
98
data resource untuk NLP bahasa indonesia
Pyclue
⭐
98
Python toolkit for Chinese Language Understanding(CLUE) Evaluation benchmark
Lenta.ru News Dataset
⭐
88
Corpus of Russian news articles collected from Lenta.Ru
Speech Corpus Collection
⭐
87
A Collection of Speech Corpus for ASR and TTS
Dataset List
⭐
86
lists of text corpus and more (mainly Japanese)
Blacklab
⭐
79
A corpus retrieval engine based on Apache Lucene
Datasets
⭐
78
Poetry-related datasets developed by THUAIPoet (Jiuge) group.
Russian_news_corpus
⭐
76
Russian mass media stemmed texts corpus / Корпус лемматизированных (морфологически нормализованных) текстов российских СМИ
Bible Corpus
⭐
76
A multilingual parallel corpus created from translations of the Bible.
Ja.text8
⭐
74
Japanese text8 corpus for word embedding.
Pansori
⭐
74
Tools for ASR Corpus Generation from Online Video
Pre Modern_chinese_corpus_dataset
⭐
68
一个近代汉语语料库数据集 This is a pre-modern Chinese ( From Song dynasty in 10th century AD to Republic of China in the early 20th Century ) language corpus.These language resources are all txt format,arranged by Dynasty（Song,Yuan,Ming,Early-Qing,Late-Qing and Republic of China）.The relevant authors' information and types of literature also have been labelled.
Sova Dataset
⭐
67
Kanji Frequency
⭐
66
Kanji usage frequency data collected from various sources
Sadedegel
⭐
63
A General Purpose NLP library for Turkish
Aidatatang_200zh
⭐
63
Aidatatang_200zh is an open source Chinese Mandarin speech corpus released by DataTang Technology Co., Ltd (www.datatang.com).
How2 Dataset
⭐
62
This repository contains code and metadata of How2 dataset
Kwdlc
⭐
58
Kyoto University Web Document Leads Corpus
Japanese Words To Vectors
⭐
57
Word2vec (word to vectors) approach for Japanese language using Gensim and Mecab.
Coarij
⭐
54
Corpus of Annual Reports in Japan
Folia
⭐
53
FoLiA: Format for Linguistic Annotation - FoLiA is a rich XML-based annotation format for the representation of language resources (including corpora) with linguistic annotations. A wide variety of linguistic annotations are supported, making FoLiA a useful format for NLP tasks and data interchange. Note that the actual Python library for processing FoLiA is implemented as part of PyNLPl, this contains higher-level tools that use the library as well as the full documentation, validation schemas, and set definitions
Jrte Corpus
⭐
51
Japanese Realistic Textual Entailment Corpus (NLP 2020, LREC 2020)
Opendialog
⭐
51
An Open-Source Package for Chinese Open-domain Conversational Chatbot (中文闲聊对话系统，一键部署微信闲聊机器人)
Corporaexplorer
⭐
48
An R package for dynamic exploration of text collections
German_nouns
⭐
48
A list of ~ 90,000 German nouns and their grammatical properties compiled from WiktionaryDE as CSV file. Plus some methods to query the data.
Streusle
⭐
47
STREUSLE: a corpus with comprehensive lexical semantic annotation (multiword expressions, supersenses)
Gum
⭐
45
Repository for the Georgetown University Multilayer Corpus (GUM)
Book Names Corpus
⭐
45
图书名语料库。含部分电影、游戏名称。
Ronec
⭐
43
Romanian Named Entity Corpus (RONEC) version 2.0
Cluemotionanalysis2020
⭐
42
CLUE Emotion Analysis Dataset 细粒度情感分析数据集
Ocr2text
⭐
41
Convert a PDF via OCR to a TXT file in UTF-8 encoding
Open Discourse
⭐
41
Open Discourse is the first fully comprehensive corpus of the plenary proceedings of the federal German Parliament (Bundestag).
Fuzzing Corpus
⭐
41
My fuzzing corpus
Egret Wenda Corpus
⭐
39
A Public Corpus for Machine Learning
Named Entity Recognition Template
⭐
38
Build a deep learning model for predicting the named entities from text.
Ud_portuguese Bosque
⭐
36
This Universal Dependencies (UD) Portuguese treebank.
Cljs Corpus
⭐
35
A greppable archive of ClojureScript code
Mitie_chinese_wikipedia_corpus
⭐
35
Pre-trained Wikipedia corpus by MITIE
Languagecodes
⭐
35
We present a list of languages with their codes, families, regions and etc. We also present a list of multi-lingual corpora (with urls).
Open2ch Dialogue Corpus
⭐
34
おーぷん2ちゃんねるをクロールして作成した対話コーパス
Voxceleb
⭐
34
mirror of VoxCeleb dataset - a large-scale speaker identification dataset
Mirastext
⭐
34
MirasText
Kyotocorpus
⭐
33
Kyoto University Text Corpus
Text Classification Cn
⭐
33
中文文本分类实践，基于搜狗新闻语料库，采用传统机器学习方法以及预训练模型等方法
Spice Corpus
⭐
32
An open-access corpus of conversational bilingual speech in Cantonese and English
Awesome Cantonese Nlp
⭐
29
A curated list of resources dedicated to Natural Language Processing (NLP) of Cantonese | 粵語 NLP
Poetrycorpus
⭐
29
Поэтический корпус русского языка
Tif
⭐
28
Text Interchange Formats
Typing Assistant
⭐
27
Typing Assistant provides the ability to autocomplete words and suggests predictions for the next word. This makes typing faster, more intelligent and reduces effort.
Dialogue Corpus
⭐
25
Switchboard Corpus
⭐
25
Utilities for Processing the Switchboard Dialogue Act Corpus
Aspen
⭐
24
🔎 📖 ✨ Custom, private search engine for text documents built with NextJS/React/ES6/ES7
Mev Corpus
⭐
24
MEV Data Corpus
Eventstoryline
⭐
24
Event StoryLine Corpus - annotated data, baselines and evaluation scripts, evaluation data.
Pubmed Pico Detection
⭐
22
PubMed PICO Element Detection Dataset
Tvsub
⭐
22
TVsub: DCU-Tencent Chinese-English Dialogue Corpus
Dialogue Datasets
⭐
21
collect the open dialog corpus and some useful data processing utils.
1-100 of 418 projects
Next >
Related Projects
Python Corpus Projects (162)
Nlp Corpus Projects (106)
Dataset Corpus Projects (69)
Python Nlp Corpus Projects (53)
Natural Language Processing Corpus Projects (51)
Machine Learning Corpus Projects (41)
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.
