Awesome Open Source
Awesome Open Source

Attentional Pooling for Action Recognition

[project page] [paper]

If this code helps with your work/research, please consider citing

Rohit Girdhar and Deva Ramanan. Attentional Pooling for Action Recognition. Advances in Neural Information Processing Systems (NIPS), 2017.

    title = {Attentional Pooling for Action Recognition},
    author = {Girdhar, Rohit and Ramanan, Deva},
    booktitle = {NIPS},
    year = 2017


This code was trained and tested with

  1. CentOS 6.5
  2. Python 2.7
  3. TensorFlow 1.1.0-rc2 (6a1825e2)

Getting started

Clone the code and create some directories for outputs

$ git clone --recursive
$ export ROOT=`pwd`/AttentionalPoolingAction
$ cd $ROOT/src/
$ mkdir -p expt_outputs data
$ # compile some custom ops
$ cd custom_ops; make; cd ..

Data setup

You can download the tfrecord files for MPII I used from here and uncompress on to a fast local disk. If you want to create your own tfrecords, you can use the following steps, which is what I used to create the linked tfrecord files

Convert the MPII data into tfrecords. The system also can read from individual JPEG files, but that needs a slightly different intial setup.

First download the MPII images and annotations, and un-compress the files.

$ cd $ROOT/utils/dataset_utils
$ # Set the paths for MPII images and annotations file in
$ python  # Will generate the tfrecord files

Keypoint labels for other datasets

While MPII dataset comes with pose labels, I also experiment with HMDB-51 and HICO, pose for which was computed using an initial version of OpenPose. I provide the extracted keypoints here: HMDB51 and HICO.

Testing pre-trained models

First download and unzip the pretrained models to a $ROOT/src/pretrained_models/. The models can be run by

# Baseline model (no attention)
$ python --cfg ../experiments/001_MPII_ResNet_pretrained.yaml
# With attention
$ python --cfg ../experiments/002_MPII_ResNet_withAttention_pretrained.yaml
# With pose regularized attention
$ python --cfg ../experiments/003_MPII_ResNet_withPoseAttention_pretrained.yaml

Expected performance on MPII Validation set

Method mAP Accuracy
Baseline (no attention) 26.2 33.5
With attention 30.3 37.2
With pose regularized attention 30.6 37.8


Train an attentional pooled model on MPII dataset, using python --cfg <path to YAML file>.

$ cd $ROOT/src
$ python --cfg ../experiments/002_MPII_ResNet_withAttention.yaml
# To train the model with pose regularized attention, use the following config
$ python --cfg ../experiments/003_MPII_ResNet_withPoseAttention.yaml
# To train the baseline without attention, use the following config
$ python --cfg ../experiments/001_MPII_ResNet.yaml

Testing and evaluation

Test the model trained above on the validation set, using python --cfg <path to YAML file>.

$ python --cfg ../experiments/002_MPII_ResNet_withAttention.yaml
# To evaluate the model with pose regularized attention
$ python --cfg ../experiments/003_MPII_ResNet_withPoseAttention.yaml
# To evaluate the model without attention
$ python --cfg ../experiments/001_MPII_ResNet.yaml

The performance of these models should be similar to the above released pre-trained models.

Train + test on the final test set

This is for getting the final number on MPII test set.

# Train on the train + val set
$ python --cfg ../experiments/004_MPII_ResNet_withAttention_train+val.yaml
# Test on the test set
$ python --cfg ../experiments/004_MPII_ResNet_withAttention_train+val.yaml --save
# Convert the output into the MAT files as expected by MPII authors (requires matlab/octave)
$ cd ../utils;
$ bash ../src/expt_outputs/004_MPII_ResNet_withAttention_train+val.yaml/<filename.h5>
# Now the generated mat file can be emailed to MPII authors for test evaluation

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
python (50,968
tensorflow (2,100
computer-vision (1,190
attention-mechanism (120
action-recognition (69
attention-model (28
nips-2017 (18

Find Open Source By Browsing 7,000 Topics Across 59 Categories