Awesome Open Source
Awesome Open Source

Real-Time Semantic Segmentation in Mobile device

This project is an example project of semantic segmentation for mobile real-time app.

The architecture is inspired by MobileNetV2 and U-Net.

LFW, Labeled Faces in the Wild, is used as a Dataset.

The goal of this project is to detect hair segments with reasonable accuracy and speed in mobile device. Currently, it achieves 0.89 IoU.

About speed vs accuracy, more details are available at my post.

Example of predicted image.

Example application

  • iOS
  • Android (TODO)

Requirements

  • Python 3.8
  • pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
  • CoreML for iOS app.

About Model

At this time, there is only one model in this repository, MobileNetV2_unet. As a typical U-Net architecture, it has encoder and decoder parts, which consist of depthwise conv blocks proposed by MobileNets.

Input image is encoded to 1/32 size, and then decoded to 1/2. Finally, it scores the results and make it to original size.

Steps to training

Data Preparation

Data is available at LFW. To get mask images, refer issue #11 for more. After you got images and masks, put the images of faces and masks as shown below.

data/
  lfw/
    raw/
      images/
        0001.jpg
        0002.jpg
      masks/
        0001.ppm
        0002.ppm

Training

If you use 224 x 224 as input size, pre-trained weight of MobileNetV2 is available. It will be automatically downloaded when you train model with the following command.

cd src
python run_train.py params/002.yaml

Dice coefficient is used as a loss function.

Pretrained model

Input size IoU Download
224 0.89 Google Drive

Converting

As the purpose of this project is to make model run in mobile device, this repository contains some scripts to convert models for iOS and Android.

TBD

  • [x] Report speed vs accuracy in mobile device.
  • [ ] Convert pytorch to Android using TesorFlow Light
Alternatives To Mobile Semantic Segmentation
Select To Compare


Alternative Project Comparisons
Related Awesome Lists
Top Programming Languages

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Python (890,046
Deep Learning (39,311
Tensorflow (22,895
Pytorch (22,633
Segmentation (8,385
Semantic Segmentation (1,660
Unet (1,288
Mobilenet (1,161
Coreml (641
Lfw (18