Awesome Open Source
Awesome Open Source

Visual Chatbot

Demo for the paper (Now upgraded to Pytorch, for the Lua-Torch version please see commit).

Visual Dialog (CVPR 2017 Spotlight) Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, Dhruv Batra
Arxiv Link:
Live demo:

Visual Chatbot


Visual Dialog requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Given an image, dialog history, and a follow-up question about the image, the AI agent has to answer the question. Putting it all together, we demonstrate the first ‘visual chatbot’!

What has changed since the last version?

The model-building code is completely shifted to Pytorch, we have put in a much improved Bottom Up Top Down captioning model from Pythia and Mask-RCNN feature extractor from maskrcnn-benchmark. The Visdial model is borrowed from visdial-challenge-starter code.

Please follow the instructions below to get the demo running on your local machine. For the previous version of this repository which supports Torch-Lua based models see commit.

Setup and Dependencies

Start with installing the Build Essentials , Redis Server and RabbiMQ Server.

sudo apt-get update

# download and install build essentials
sudo apt-get install -y git python-pip python-dev
sudo apt-get install -y autoconf automake libtool 
sudo apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler

# download and install redis-server and rabbitmq-server
sudo apt-get install -y redis-server rabbitmq-server
sudo rabbitmq-plugins enable rabbitmq_management
sudo service rabbitmq-server restart 
sudo service redis-server restart

Environment Setup

You can use Anaconda or Miniconda to setup this code base. Download and install Anaconda or Miniconda distribution based on Python3+ from their downloads page and proceed below.

# clone and download submodules
git clone
git submodule update init --recursive

# create and activate new environment
conda create -n vischat python=3.6.8
conda activate vischat

# install the requirements of chatbot and visdial-starter code
cd visual-chatbot/
pip install -r requirements.txt


Download the BUTD, Mask-RCNN and VisDial model checkpoints and their configuration files.

sh viscap/

Install Submodules

Install Pythia to use BUTD captioning model and maskrcnn-benchmark for feature extraction.

# install fastText (dependency of pythia)
cd viscap/captioning/fastText
pip install -e .

# install pythia for using butd model
cd ../pythia/
sed -i '/torch/d' requirements.txt
pip install -e .

# install maskrcnn-benchmark for feature extraction
cd ../vqa-maskrcnn-benchmark/
python build
python develop
cd ../../../

Cuda Installation

Note: CUDA and cuDNN is only required if you are going to use GPU. Download and install CUDA and cuDNN from nvidia website.


We use PunktSentenceTokenizer from nltk, download it if you haven't already.

python -c "import nltk;'punkt')"

Let's run this now!

Setup the database

# create the database
python makemigrations chat
python migrate

Run server and worker

Launch two separate terminals and run worker and server code.

# run rabbitmq worker on first terminal
# warning: on the first-run glove file ~ 860 Mb is downloaded, this is a one-time thing

# run development server on second terminal
python runserver

You are all set now. Visit and you will have your demo running successfully.


If you run into incompatibility issues, please take a look here and here.

Model Checkpoint and Features Used

Performance on v1.0 test-std (trained on v1.0 train + val):

Model [email protected] [email protected] [email protected] MeanR MRR NDCG
lf-gen-mask-rcnn-x101-demo 0.3930 0.5757 0.6404 18.4950 0.4863 0.5967

Extracted features from VisDial v1.0 used to train the above model are here:

Note: Above features have key image_id (from earlier versions) renamed as image_ids.

Cite this work

If you find this code useful, consider citing our work:

  title={{V}isual {D}ialog},
  author={Abhishek Das and Satwik Kottur and Khushi Gupta and Avi Singh
    and Deshraj Yadav and Jos\'e M.F. Moura and Devi Parikh and Dhruv Batra},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},




Credits and Acknowledgements

Related Awesome Lists
Top Programming Languages
Top Projects

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Javascript (1,091,101
Python (837,714
Machine Learning (38,608
Deep Learning (37,667
Django (35,427
Redis (24,021
Websocket (14,040
Dialog (6,846
Chatbot (6,505
Rabbitmq (6,288
Rcnn (1,345