Pytorch Seq2seq

Tutorials on implementing a few sequence-to-sequence (seq2seq) models with PyTorch and TorchText.
Alternatives To Pytorch Seq2seq
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Pytorch Tutorial27,137
4 months ago85mitPython
PyTorch Tutorial for Deep Learning Researchers
4 days ago193
坚持分享 GitHub 上高质量、有趣实用的开源技术教程、开发者工具、编程网站、技术资讯。A list cool, interesting projects of GitHub.
Awesome Pytorch List14,103
6 months ago4
A comprehensive list of pytorch related content on github,such as different models,implementations,helper libraries,tutorials etc.
Nlp Tutorial13,226
10 days ago35mitJupyter Notebook
Natural Language Processing Tutorial for Deep Learning Researchers
Haystack11,8413012 hours ago100November 09, 2023407apache-2.0Python
:mag: LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
The Incredible Pytorch10,408
8 days ago1mit
The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.
12 hours ago181bsd-3-clausePython
PyTorch tutorials.
Pytorch Tutorial7,372
8 months ago28mitJupyter Notebook
Build your neural network easy and fast, 莫烦Python中文教学
Machine Learning Collection6,129
7 days ago96mitPython
A resource for learning about Machine learning & Deep Learning
Pytorch Seq2seq4,548
6 months ago56mitJupyter Notebook
Tutorials on implementing a few sequence-to-sequence (seq2seq) models with PyTorch and TorchText.
Alternatives To Pytorch Seq2seq
Select To Compare

Alternative Project Comparisons

PyTorch Seq2Seq

Note: This repo only works with torchtext 0.9 or above which requires PyTorch 1.8 or above. If you are using torchtext 0.8 then please use this branch

This repo contains tutorials covering understanding and implementing sequence-to-sequence (seq2seq) models using PyTorch 1.8, torchtext 0.9 and spaCy 3.0, using Python 3.8.

If you find any mistakes or disagree with any of the explanations, please do not hesitate to submit an issue. I welcome any feedback, positive or negative!

Getting Started

To install PyTorch, see installation instructions on the PyTorch website.

To install torchtext:

pip install torchtext

We'll also make use of spaCy to tokenize our data. To install spaCy, follow the instructions here making sure to install both the English and German models with:

python -m spacy download en_core_web_sm
python -m spacy download de_core_news_sm


  • 1 - Sequence to Sequence Learning with Neural Networks Open In Colab

    This first tutorial covers the workflow of a PyTorch with torchtext seq2seq project. We'll cover the basics of seq2seq networks using encoder-decoder models, how to implement these models in PyTorch, and how to use torchtext to do all of the heavy lifting with regards to text processing. The model itself will be based off an implementation of Sequence to Sequence Learning with Neural Networks, which uses multi-layer LSTMs.

  • 2 - Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation Open In Colab

    Now we have the basic workflow covered, this tutorial will focus on improving our results. Building on our knowledge of PyTorch and torchtext gained from the previous tutorial, we'll cover a second second model, which helps with the information compression problem faced by encoder-decoder models. This model will be based off an implementation of Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, which uses GRUs.

  • 3 - Neural Machine Translation by Jointly Learning to Align and Translate Open In Colab

    Next, we learn about attention by implementing Neural Machine Translation by Jointly Learning to Align and Translate. This further allievates the information compression problem by allowing the decoder to "look back" at the input sentence by creating context vectors that are weighted sums of the encoder hidden states. The weights for this weighted sum are calculated via an attention mechanism, where the decoder learns to pay attention to the most relevant words in the input sentence.

  • 4 - Packed Padded Sequences, Masking, Inference and BLEU Open In Colab

    In this notebook, we will improve the previous model architecture by adding packed padded sequences and masking. These are two methods commonly used in NLP. Packed padded sequences allow us to only process the non-padded elements of our input sentence with our RNN. Masking is used to force the model to ignore certain elements we do not want it to look at, such as attention over padded elements. Together, these give us a small performance boost. We also cover a very basic way of using the model for inference, allowing us to get translations for any sentence we want to give to the model and how we can view the attention values over the source sequence for those translations. Finally, we show how to calculate the BLEU metric from our translations.

  • 5 - Convolutional Sequence to Sequence Learning Open In Colab

    We finally move away from RNN based models and implement a fully convolutional model. One of the downsides of RNNs is that they are sequential. That is, before a word is processed by the RNN, all previous words must also be processed. Convolutional models can be fully parallelized, which allow them to be trained much quicker. We will be implementing the Convolutional Sequence to Sequence model, which uses multiple convolutional layers in both the encoder and decoder, with an attention mechanism between them.

  • 6 - Attention Is All You Need Open In Colab

    Continuing with the non-RNN based models, we implement the Transformer model from Attention Is All You Need. This model is based soley on attention mechanisms and introduces Multi-Head Attention. The encoder and decoder are made of multiple layers, with each layer consisting of Multi-Head Attention and Positionwise Feedforward sublayers. This model is currently used in many state-of-the-art sequence-to-sequence and transfer learning tasks.


Here are some things I looked at while making these tutorials. Some of it may be out of date.

Popular Pytorch Projects
Popular Tutorials Projects
Popular Machine Learning Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Jupyter Notebook
Recurrent Neural Networks
Sequence To Sequence
Encoder Decoder
Pytorch Tutorial
Neural Machine Translation