nlp journey

All implemented in tensorflow 2.0，codes

1. Basics

• tutorials
• frequent questions

2. Books

1. Handbook of Graphical Models. online
2. Deep Learning. online
3. Neural Networks and Deep Learning. online
4. Speech and Language Processing. online

3. Papers

01) Transformer papers

1. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. paper
2. GPT-2: Language Models are Unsupervised Multitask Learners. paper
3. Transformer-XL: Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. paper
4. XLNet: Generalized Autoregressive Pretraining for Language Understanding. paper
5. RoBERTa: Robustly Optimized BERT Pretraining Approach. paper
6. DistilBERT: a distilled version of BERT: smaller, faster, cheaper and lighter. paper
7. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. paper
8. T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. paper
9. ELECTRA: pre-training text encoders as discriminators rather than generators. paper
10. GPT3: Language Models are Few-Shot Learners. paper

02) Models

1. LSTM(Long Short-term Memory). paper
2. Sequence to Sequence Learning with Neural Networks. paper
3. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. paper
4. Residual Network(Deep Residual Learning for Image Recognition). paper
5. Dropout(Improving neural networks by preventing co-adaptation of feature detectors). paper
6. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. paper

03) Summaries

1. An overview of gradient descent optimization algorithms. paper
2. Analysis Methods in Neural Language Processing: A Survey. paper
3. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. paper
4. A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications. paper
5. A Gentle Introduction to Deep Learning for Graphs. paper
6. A Survey on Deep Learning for Named Entity Recognition. paper
7. More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction. paper
8. Deep Learning Based Text Classification: A Comprehensive Review. paper
9. Pre-trained Models for Natural Language Processing: A Survey. paper
10. A Survey on Contextual Embeddings. paper
11. A Survey on Knowledge Graphs: Representation, Acquisition and Applications. paper
12. Knowledge Graphs. paper
13. Pre-trained Models for Natural Language Processing: A Survey. paper

04) Pre-training

1. A Neural Probabilistic Language Model. paper
2. word2vec Parameter Learning Explained. paper
3. Language Models are Unsupervised Multitask Learners. paper
4. An Empirical Study of Smoothing Techniques for Language Modeling. paper
5. Efficient Estimation of Word Representations in Vector Space. paper
6. Distributed Representations of Sentences and Documents. paper
7. Enriching Word Vectors with Subword Information(FastText). paper
8. GloVe: Global Vectors for Word Representation. online
9. ELMo (Deep contextualized word representations). paper
10. Pre-Training with Whole Word Masking for Chinese BERT. paper

05) Classification

1. Bag of Tricks for Efficient Text Classification (FastText). paper
2. Convolutional Neural Networks for Sentence Classification. paper
3. Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. paper

06) Text generation

1. A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation. paper
2. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. paper

07) Text Similarity

1. Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks. paper
2. Learning Text Similarity with Siamese Recurrent Networks. paper
3. A Deep Architecture for Matching Short Texts. paper

08) QA

1. A Question-Focused Multi-Factor Attention Network for Question Answering. paper
2. The Design and Implementation of XiaoIce, an Empathetic Social Chatbot. paper
3. A Knowledge-Grounded Neural Conversation Model. paper
4. Neural Generative Question Answering. paper
5. Sequential Matching Network A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots．paper
6. Modeling Multi-turn Conversation with Deep Utterance Aggregation．paper
7. Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network．paper
8. Deep Reinforcement Learning For Modeling Chit-Chat Dialog With Discrete Attributes. paper

09) NMT

1. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. paper
2. Neural Machine Translation by Jointly Learning to Align and Translate. paper
3. Transformer (Attention Is All You Need). paper

10) Summary

1. Get To The Point: Summarization with Pointer-Generator Networks. paper
2. Deep Recurrent Generative Decoder for Abstractive Text Summarization. paper

11) Relation extraction

1. Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks. paper
2. Neural Relation Extraction with Multi-lingual Attention. paper
3. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation. paper
4. End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures. paper

4. Articles

• 如何学习自然语言处理（综合版）. url
• TRANSFORMERS FROM SCRATCH. url
• The Illustrated Transformer.url
• Attention-based-model. url
• Modern Deep Learning Techniques Applied to Natural Language Processing. url
• 难以置信！LSTM和GRU的解析从未如此清晰（动图+视频）。url
• 从语言模型到Seq2Seq：Transformer如戏，全靠Mask. url
• Applying word2vec to Recommenders and Advertising. url
• 2019 NLP大全：论文、博客、教程、工程进展全梳理. url

5. Github

• CLUE. github
• transformers. github
• HanLP. github
• ML-For-Beginners. github

6. Blog

• 52nlp
• 科学空间/信息时代
• 刘建平Pinard
• 零基础入门深度学习
• Jay Alammar
• Andrej Karpathy blog
• Edwin Chen
• Distill