This is a general purpose repository for Federated Self-Superivised Learning for video understanding build on top of MMCV and Flower.

• Yasar Abbas Ur Rehman
  
• Yan Gao
• Pedro Porto Buarque de Gusmão
  
• Jiajun Shen
  
• Nicholas D. Lane
  

For both centralized and federated video SSL pretraining, we use Kinetics-400. We evaluate the quality of learned representations by applying them on two downstream datasets: UCF-101 and HMDB-51.

As a part of reproducibility, we have provideded the dataset partitions of the Kinetics-400 dataset for federated learning in the Data folder with iid and non-iid data distribution.

One can generate the non-iid version of kinetics-400 with 100 clients (8 classes per client) by running:
python scripts/k400_non_iid.py

The iid version of kinetics-400 with 100 clients (8 classes per client) can be generated by running:
python scripts/kinetics_json_splitter.py

Note that the above two python files assume that you have already downloaded the official trainlist of kineticss-400.

We provide a series of federated-SSL pretrined models of VCOP, Speed, and CtP. All these models are federated pretrained on the non-iid version of Kinetics-400 (8 classes/client) see Table 1 in the manuscript. The annotations can be found in the Data/Kinetics-400_annotations/ in this repository.

• FedVSSL has been added to the Flower baselines.
• Check out the teaser of our work on the YouTube.
• The preprint of our paper is now available on arXiv

For a complete list of the required packages please see the requirement.txt file. One can easily install, all the requirement by running pip install -r requirement.txt.

• Anaconda environment with python 3.8
• Flower
  
• Microsoft CtP Framework
• MMCV

We recommend installing Microsoft CtP Framework as it contain all the Self-supervised learning frameworks build on top of MMCV framework. Here we provided a modifed version of that framework for FedVSSL, in particular.

The abstract definition of classes are provided by reproduce_papers/fedssl/videossl.py.

After FL pretraining one can use the following code to fine-tune the model on UCF or HMDB.

import subprocess
import os
import CtP 
process_obj = subprocess.run(["bash", "CtP/tools/dist_train.sh",\
"CtP/configs/ctp/r3d_18_kinetics/finetune_ucf101.py", "4",\
f"--work_dir /finetune/ucf101/",
f"--data_dir /DATA",\
f"--pretrained /path to the pretrained checkpoint",\
f"--validate"])

For the detailed results regarding the below checkpoints, please see Table 4 in the manuscript.

If you encounter any issues, feel free to open an issue in the github

@article{rehman2022federated,
  title={Federated Self-supervised Learning for Video Understanding},
  author={Rehman, Yasar Abbas Ur and Gao, Yan and Shen, Jiajun and de Gusmao, Pedro Porto Buarque and Lane, Nicholas},
  journal={arXiv preprint arXiv:2207.01975},
  year={2022}
}

We would like to thank Daniel J. Beutel for providing the initial blueprint of Federated self-supervised learning with flower. Also thanks to Akhil Mathur for providing the useful suggestions.