In remote sensing, multi-image super-resolution (MISR) is a challenging problem. The release of the PROBA-V Kelvin dataset has aroused our great interest.

We believe that multiple images contain more information than a single image, so it is necessary to improve image utilization significantly. Besides, the time span of multiple images taken by the PROBA-V satellite is long, so the impact of image position needs to be reduced.

In this repository, we demonstrate a novel Transformer-based MISR framework, namely TR-MISR, which gets the state-of-the-art on the PROBA-V Kelvin dataset. TR-MISR does not pursue the complexity of the encoder and decoder but the fusion capability of the fusion module. Specifically, we rearrange the feature maps encoded by low-resolution images to a set of feature vectors. By adding a learnable embedding vector, these feature vectors can be fused through multi-layers of the Transformer with self-attention. Then, we decode the output of the embedding vector to get a patch of a high-resolution image.

TR-MISR can receive more unclear input images without performance degradation, and it does not require pre-training. Overall, TR-MISR is an attempt to apply Transformers to a specific low-level vision task.

• Recommended GPU platform: An NVIDIA ® Tesla ® V100 GPU.
• If you are using another GPU and facing a memory shortage, please reduce the batch size or choose a smaller model hyperparameter as appropriate.

• Setup a python environment and install dependencies. Our python version == 3.6.12

pip install -r requirements.txt

• Download the training/validation set assigned by RAMS, RAMS/probav_data at master · EscVM/RAMS (github.com)
• Run the split_data_fit script to crop images in each scene for the training set.

python ./split_data_fit.py

• Run the save_clearance script to precompute clearance scores for low-resolution images.

python ./save_clearance.py

• You can easily get the complete preprocessed dataset on Google Drive or Baidu Cloud (code:gflb).

In the config file, the main settings are shown in the following Table.

If the above processes are prepared, then it's time for training.

python ./src/train.py

If you need to record the training log, run

python ./src/train.py 2>&1 | tee re.log

The re.log file can be used to print out the training details in each epoch.

python ./train_all_read_log.py   # for training all data
python ./train_read_log.py       # for training and evaluation

You can also view the training logs with tensorboardX.

tensorboard --logdir='tb_logs/'

• Fix the model paths trained in the NIR band and RED band, respectively.
• The val script outputs a val_plot.png to visualize the results of each scene obtained by TR-MISR compared to the baseline.

python /src/val.py

The test script is mainly used to submit the results to the leaderboard since the ground truth is not involved in the testing set. The test script will output a submission zip with 16-bit images (located in './submission/') and a visual result with 8-bit images (located in './submission_vis/'). The test platform is still open, allowing more methods to challenge their performance limits.

python /src/test.py

The leaderboard is shown as follows:

PROBA-V Benchmark (Multi-Frame Super-Resolution) | Papers With Code

If it helps for you, please cite

@article{an2022tr,
  title={TR-MISR: Multiimage Super-Resolution Based on Feature Fusion With Transformers},
  author={An, Tai and Zhang, Xin and Huo, Chunlei and Xue, Bin and Wang, Lingfeng and Pan, Chunhong},
  journal={IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},
  volume={15},
  pages={1373--1388},
  year={2022},
  publisher={IEEE}
}