GPT2 for Multiple Languages

[x] Simplifed GPT2 train scripts（based on Grover, supporting TPUs）
[x] Ported bert tokenizer, multilingual corpus compatible
[x] 1.5B GPT2 pretrained Chinese model ( ~15G corpus, 10w steps )
[x] Batteries-included Colab demo #
[x] 1.5B GPT2 pretrained Chinese model ( ~30G corpus, 22w steps )

Pretrained Model

Size	Language	Corpus	Vocab	Link1	Link2	SHA256
1.5B Params	Chinese	~30G	CLUE ( 8021 tokens )	Google Drive	Baidu Pan (ffz6)	e698cc97a7f5f706f84f58bb469d614e 51d3c0ce5f9ab9bf77e01e3fcb41d482
1.5B Params	Chinese	~15G	Bert ( 21128 tokens )	Google Drive	Baidu Pan (q9vr)	4a6e5124df8db7ac2bdd902e6191b807 a6983a7f5d09fb10ce011f9a073b183e

Corpus from THUCNews and nlp_chinese_corpus

Using Cloud TPU Pod v3-256 to train 22w steps

loss

Google Colab

With just 2 clicks (not including Colab auth process), the 1.5B pretrained Chinese model demo is ready to go:

[Colab Notebook]

Train

Disclaimer

The contents in this repository are for academic research purpose, and we do not provide any conclusive remarks.

Citation

@misc{GPT2-ML,
  author = {Zhibo Zhang},
  title = {GPT2-ML: GPT-2 for Multiple Languages},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/imcaspar/gpt2-ml}},
}

Reference

google-research/bert

rowanz/grover

Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC)

Press

[机器之心] 只需单击三次，让中文GPT-2为你生成定制故事

[科学空间] 现在可以用Keras玩中文GPT2了

Project Name	Stars	Repos Using This	Packages Using This	Most Recent Commit	Total Releases	Latest Release	Open Issues	License	Language
Transformers	116,723	64	2,484	12 hours ago	125	November 15, 2023	911	apache-2.0	Python
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Spleeter	23,845		6	22 days ago	37	June 10, 2022	223	mit	Python
Deezer source separation library including pretrained models.
Silero Models	4,088		4	2 months ago	4	June 12, 2022	8	other	Jupyter Notebook
Silero Models: pre-trained speech-to-text, text-to-speech and text-enhancement models made embarrassingly simple
Albert_zh	3,723			a year ago			97		Python
A LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS, 海量中文预训练ALBERT模型
Efficient Ai Backbones	3,568			6 months ago			58		Python
Efficient AI Backbones including GhostNet, TNT and MLP, developed by Huawei Noah's Ark Lab.
Clue	3,345			6 months ago			73		Python
中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard
Pinto_model_zoo	3,050			4 days ago			10	mit	Python
A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
Imgclsmob	2,399		9	2 years ago	67	September 21, 2021	6	mit	Python
Sandbox for training deep learning networks
Efficientnet	1,853	2	12	2 years ago	9	September 15, 2020	58	apache-2.0	Python
Implementation of EfficientNet model. Keras and TensorFlow Keras.
Gpt2 Ml	1,674			7 months ago			22	apache-2.0	Python
GPT2 for Multiple Languages, including pretrained models. GPT2 多语言支持, 15亿参数中文预训练模型

Gpt2 Ml