awesome_lightweight_networks

(Researcher) (Engineer)

2017

---

Cifar10/100,ImageNetSOTA

---

readme~

pip install light_cnns

Table of Contents

MobileNets

ShuffleNet

Transformer

CPU

Inception

CondenseNet

Bag of Tricks for Image Classification with Convolutional Neural Networks

Code

import torch
from light_cnns import resnet50_v1b
model = resnet50_v1b()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

MobileNets

Google MobileNets.

• MobileNetV1
• MobileNetV2
• MobileNetV3
• MobileNeXt

MobileNetv1

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications MobileNetv1Googledepthwise separable convolution

Code

import torch
from light_cnns import mbv1
model = mbv1()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

MobileNetv2

• MobileNetV2: Inverted Residuals and Linear Bottlenecks

mobilenetv2 ExpansionProjectionInverted residual block

• RELU6 6
• projection layerpointwise convolution ReLU6LinearLinear Bottleneck

Code

import torch
from light_cnns import mbv2
model = mbv2()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

MobileNetV3

Searching for MobileNetV3

• https://arxiv.org/abs/1905.02244

• NASMnasNetMobileNetV2

• Large Small;

• MobileNetV1

• MobileNetV2

• squeeze and excitation(SE)

• h-swish(x)

• NASplatform-aware NASNetAdapt

• MobileNetV2head;

Code

import torch
from light_cnns import mbv3_small
#from light_cnns import mbv3_large
model_small = mbv3_small()
#model_large = mbv3_large()
model_small.eval()
print(model_small)
input = torch.randn(1, 3, 224, 224)
y = model_small(input)
print(y.size())

MobileNeXt

• Rethinking Bottleneck Structure for Efficient Mobile Network Design

MobileNetV2SandGlassSandglass Blockbottomtop

Code

import torch
from light_cnns import mobilenext
model = mobilenext()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

ShuffleNet

• ShuffleNetv1
• ShuffleNetV2

ShuffleNetv1

• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Code

import torch
from light_cnns import shufflenetv1
model = shufflenetv1()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

ShuffleNetv2

• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

Code

import torch
from light_cnns import shufflenetv2
model = shufflenetv2()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

• AdderNet
• GhostNet

AdderNet)

• AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence

L1 L1

CIFARImageNetAdderNetCNN

Code

import torch
from light_cnns import resnet20
model = resnet20()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

GhostNet

• GhostNet: More Features from Cheap Operations

GhostGhost feature mapsGhostGhostGhost bottleneckGhostNetImageNetGhostNetTop-175.7%MobileNetV375.2%

Code

import torch
from light_cnns import ghostnet
model = ghostnet()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

CANet

• Coordinate Attention for Efficient Mobile Network Design

(SE)Coordinate Attention

Code

import torch
from light_cnns import mbv2_ca
model = mbv2_ca()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

ECANet

• ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

• https://arxiv.org/abs/1910.03151

ECANetCNNECANetSENetECASENet

---

import torch
from light_cnns import mbv2_eca
model = mbv2_eca()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

ResNeSt

• ResNeSt: Split-Attention Networks

• https://hangzhang.org/files/resnest.pdf

ResNeSt ""Multi-path Feature-map [email protected] SENetSKNet ResNeXt attention group levelSplit-Attentionfeature-mapfeature-map

1. GoogleNet Multi-pathkernels
2. ResNeXtResNet bottlemulti-path
3. SE-Net channel-attention
4. SK-Net feature-map attentionmultiple scale featurefeature map informationSoftmaxchannel-wise

---

import torch
from light_cnns import resnest50_v1b
model = resnest50_v1b()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

SANet

• SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

• https://arxiv.org/abs/2102.00240

shuffle attentionSpatial AttentionChannel AttentionSAShuffle unit

import torch
from light_cnns import mbv2_sa
model = mbv2_sa()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

Triplet attention

• Rotate to Attend: Convolutional Triplet Attention Module

• https://arxiv.org/abs/2010.03045

Triplet AttentionTriplet Branch(cross dimension interaction)Triplet AttentionBackbone.

import torch
from light_cnns import mbv2_triplet
model = mbv2_triplet()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

CPU

LCNet

PP-LCNet: A Lightweight CPU Convolutional Neural Network

• https://arxiv.org/pdf/2109.15099.pdf

• CPU?
• CPU

LCNetShuffleNetV2MobileNetV2MobileNetV3GhostNet-

import torch
from light_cnns import lcnet_baseline
model = lcnet_baseline()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

Transformer

MobileViT: ,Transformer

• https://arxiv.org/abs/2110.02178

MobileViTCNNViT CNN ViT

import torch
from light_cnns import mobilevit_s
model = mobilevit_s()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

Inception

Going Deeper with Convolutions (GoogleNet)

• https://arxiv.org/abs/1409.4842

1. 1x1

:

import torch
from light_cnns import googlenet
model = googlenet()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

Rethinking the Inception Architecture for Computer Vision

• https://arxiv.org/abs/1512.00567

Inceptionv2InceptionV1

1. BNBN.
2. 5x5 3x3 nxn 1xn nx1 .

import torch
from light_cnns import inception_v2
model = inception_v2()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

Inception Net v3 Inception v2 5

1. max poolingpoolingconcatShuffleNet

2. RMSProp

3. Factorized 7x7

4. BatchNorm

5. label smoothing;

import torch
from light_cnns import inception_v3
model = inception_v3()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

• https://arxiv.org/abs/1602.07261

1. Inception v4 stemInception
2. stemInception Inception v4Inception AB C
3. reduction block

import torch
from light_cnns import inception_v4
model = inception_v4()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

Xception: Deep Learning with Depthwise Separable Convolutions

https://arxiv.org/abs/1610.02357

InceptionXceptionXceptionExtreme Inception XceptionInception.Xception

1. Inception3x3
2. 1x1.
3. 3x31x1
4. 3x3InceptionExtream InceptionXception3x3 3x31x1InceptionInception

import torch
from light_cnns import xception
model = xception()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

Inception Convolution with Efficient Dilation SearchCVPR2021 oral

• https://arxiv.org/abs/2012.13587

()EDOeffective dilation searchinception (dilated)

import torch
from light_cnns import ic_resnet50
patter = './pattern_zoo/detection/ic_resnet50_k9.json'
model = ic_resnet50(pattern_path=patter)
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

ESPNet

ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation

• https://arxiv.org/abs/1803.06815v2

ESPNetESP Modulepoint-wise,ESPNetGPU//112FPS/21FPS/9FPS

import torch
from light_cnns import espnetv1
model = espnetv1()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())

ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network

ESPNetv2ESPNetv1

1. point-wiseESPEESP(Extremely Efficient Spatial Pyramid)ESPNet

2. cyclic learning rate schedulerscheduler

EESP(Strided EESP with shortcut connection to an input image),

import torch
from light_cnns import espnetv2
model = espnetv2()
model.eval()
print(model)
input = torch.randn(1, 3, 224, 224)
y = model(input)
print(y.size())