neural-motifs

Like this work, or scene understanding in general? You might be interested in checking out my brand new dataset VCR: Visual Commonsense Reasoning, at visualcommonsense.com!

This repository contains data and code for the paper Neural Motifs: Scene Graph Parsing with Global Context (CVPR 2018) For the project page (as well as links to the baseline checkpoints), check out rowanzellers.com/neuralmotifs. If the paper significantly inspires you, we request that you cite our work:

Bibtex

@inproceedings{zellers2018scenegraphs,
  title={Neural Motifs: Scene Graph Parsing with Global Context},
  author={Zellers, Rowan and Yatskar, Mark and Thomson, Sam and Choi, Yejin},
  booktitle = "Conference on Computer Vision and Pattern Recognition",  
  year={2018}
}

Setup

Install python3.6 and pytorch 3. I recommend the Anaconda distribution. To install PyTorch if you haven't already, use conda install pytorch=0.3.0 torchvision=0.2.0 cuda90 -c pytorch.
Update the config file with the dataset paths. Specifically:
- Visual Genome (the VG_100K folder, image_data.json, VG-SGG.h5, and VG-SGG-dicts.json). See data/stanford_filtered/README.md for the steps I used to download these.
- You'll also need to fix your PYTHONPATH: export PYTHONPATH=/home/rowan/code/scene-graph
Compile everything. run make in the main directory: this compiles the Bilinear Interpolation operation for the RoIs as well as the Highway LSTM.
Pretrain VG detection. The old version involved pretraining COCO as well, but we got rid of that for simplicity. Run ./scripts/pretrain_detector.sh Note: You might have to modify the learning rate and batch size, particularly if you don't have 3 Titan X GPUs (which is what I used). You can also download the pretrained detector checkpoint here.
Train VG scene graph classification: run ./scripts/train_models_sgcls.sh 2 (will run on GPU 2). OR, download the MotifNet-cls checkpoint here: Motifnet-SGCls/PredCls.
Refine for detection: run ./scripts/refine_for_detection.sh 2 or download the Motifnet-SGDet checkpoint.
Evaluate: Refer to the scripts ./scripts/eval_models_sg[cls/det].sh.

help

Feel free to open an issue if you encounter trouble getting it to work!

Project Name	Stars	Most Recent Commit	Open Issues	License	Language
Awesome Gcn	377	4 years ago	1
resources for graph convolutional networks （图卷积神经网络相关资源）
Social Stgcnn	360	4 months ago	1	mit	Python
Code for "Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction" CVPR 2020
Neural Motifs	348	4 years ago	26	mit	Python
Code for Neural Motifs: Scene Graph Parsing with Global Context (CVPR 2018)
Gcn_clustering	314	2 years ago	28	mit	Jupyter Notebook
Code for CVPR'19 paper Linkage-based Face Clustering via GCN
Drrg	248	2 months ago	12	mit	C++
Deep relational reasoning graph network for arbitrary shape text detection; Accepted by CVPR 2020 (Oral). http://arxiv.org/abs/2003.07493
Asg2cap	175	6 months ago	17	mit	Python
Code accompanying the paper "Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs" (Chen et al., CVPR 2020, Oral).
Ppgnet	126	4 years ago	6	mit	Python
Source code for our CVPR 2019 paper - PPGNet: Learning Point-Pair Graph for Line Segment Detection
Gpa Detection	93	2 years ago	10		Python
Implementation of Cross-domain Detection via Graph-induced Prototype Alignment (CVPR 2020 Oral).
Ai Paper Drawer	88	2 years ago
人工智能论文关键点集结。This project aims to collect key points of AI papers.
Egt	30	3 years ago	1		Java
Code for the CVPR'19 paper "Explore-Exploit Graph Traversal for Image Retrieval"