This is a building instance segmentation network combining multi-task deep learning and multi-modal remote sensing data including LiDAR features and optical image features. The backbone network is fed with multispectral images based on resnet50, and the branch network is input with LiDAR products based on resnet18. In the module files, the Depthawaregate layer and Spectralawaregate layer are embedded into the residual network structure to fuse the multi-modal features. The non_localgate layer construct a cross level global context model. ACPD is a detector based on CenterNet, which introduces deformation convolution operation in supervised learning.

For method details, please refer to the paper "https://www.mdpi.com/2072-4292/14/19/4920". The code is completed by Tensorflow-Keras 2.0. Rasterio / GDAL require installation to load data.

We established a building instance segmentation dataset (BISM) using multimodal remote sensing data. The BISM dataset covers 60 square kilometers in Boston, Massachusetts, USA, and contains about 39527 building objects. BISM can be downloaded via http://bismdataset.mikecrm.com/Yc5qJZD. The metadata information follows as table:

LiDAR/image NDVI DEM Ground-truth.

If you use our dataset, we recommend you might cite our work:

Yuan Q, Mohd Shafri HZ. Multi-Modal Feature Fusion Network with Adaptive Center Point Detector for Building Instance Extraction. Remote Sensing. 2022; 14(19):4920. https://doi.org/10.3390/rs14194920