Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Deepspeed | 25,642 | 12 | 10 hours ago | 53 | May 25, 2022 | 898 | apache-2.0 | Python | ||
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. | ||||||||||
Nni | 12,973 | 8 | 22 | a day ago | 51 | June 22, 2022 | 285 | mit | Python | |
An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning. | ||||||||||
Pocketflow | 2,553 | 3 years ago | 73 | other | Python | |||||
An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications. | ||||||||||
Model Optimization | 1,402 | 3 | 18 | 3 days ago | 26 | March 18, 2022 | 197 | apache-2.0 | Python | |
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning. | ||||||||||
Compressai | 760 | 1 | a month ago | 17 | March 17, 2022 | 14 | bsd-3-clause-clear | Python | ||
A PyTorch library and evaluation platform for end-to-end compression research | ||||||||||
Compression | 753 | 1 | 4 days ago | 15 | May 30, 2022 | 6 | apache-2.0 | Python | ||
Data compression in TensorFlow | ||||||||||
Nncf | 616 | 3 | a day ago | 12 | July 05, 2022 | 53 | apache-2.0 | Python | ||
Neural Network Compression Framework for enhanced OpenVINOโข inference | ||||||||||
Deep Compression Alexnet | 599 | a year ago | 1 | bsd-2-clause | Python | |||||
Deep Compression on AlexNet | ||||||||||
Embedded Neural Network | 535 | 4 years ago | n,ull | |||||||
collection of works aiming at reducing model sizes or the ASIC/FPGA accelerator for machine learning | ||||||||||
Awesome Deep Neural Network Compression | 452 | 14 hours ago | 4 | Python | ||||||
Summary, Code for Deep Neural Network Quantization |
Key Features โข Installation โข Documentation โข Usage โข Tutorials and Samples โข Third-party integration โข Model Zoo
Neural Network Compression Framework (NNCF) provides a suite of post-training and training-time algorithms for neural networks inference optimization in OpenVINOโข with minimal accuracy drop.
NNCF is designed to work with models from PyTorch, TensorFlow, ONNX and OpenVINOโข.
NNCF provides samples that demonstrate the usage of compression algorithms for different use cases and models. Compression results achievable with the NNCF-powered samples can be found in a table at the end of this document.
The framework is organized as a Python* package that can be built and used in a standalone mode. The framework architecture is unified to make it easy to add different compression algorithms for both PyTorch and TensorFlow deep learning frameworks.
Compression algorithm | OpenVINO | PyTorch | TensorFlow | ONNX |
---|---|---|---|---|
Post-Training Quantization | Supported | Supported | Supported | Supported |
Compression algorithm | PyTorch | TensorFlow |
---|---|---|
Quantization Aware Training | Supported | Supported |
Mixed-Precision Quantization | Supported | Not supported |
Binarization | Supported | Not supported |
Sparsity | Supported | Supported |
Filter pruning | Supported | Supported |
Movement pruning | Experimental | Not supported |
NOTE: Limited support for TensorFlow models. The models created using Sequential or Keras Functional API are only supported.
This documentation covers detailed information about NNCF algorithms and functions needed for the contribution to NNCF.
The latest user documentation for NNCF is available here.
NNCF API documentation can be found here.
The NNCF PTQ is the simplest way to apply 8-bit quantization. To run the algorithm you only need your model and a small (~300 samples) calibration dataset.
OpenVINO is the preferred backend to run PTQ with, and PyTorch, TensorFlow and ONNX are also supported.
import nncf
import openvino.runtime as ov
import torch
from torchvision import datasets
# Instantiate your uncompressed model
model = ov.Core().read_model("/model_path")
# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path")
dataset_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1)
# Step 1: Initialize transformation function
def transform_fn(data_item):
images, _ = data_item
return images
# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)
import nncf
import torch
from torchvision import datasets, models
# Instantiate your uncompressed model
model = models.mobilenet_v2()
# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path")
dataset_loader = torch.utils.data.DataLoader(val_dataset)
# Step 1: Initialize the transformation function
def transform_fn(data_item):
images, _ = data_item
return images
# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)
import nncf
import tensorflow as tf
import tensorflow_datasets as tfds
# Instantiate your uncompressed model
model = tf.keras.applications.MobileNetV2()
# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = tfds.load("/path", split="validation",
shuffle_files=False, as_supervised=True)
# Step 1: Initialize transformation function
def transform_fn(data_item):
images, _ = data_item
return images
# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(val_dataset, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)
import onnx
import nncf
import torch
from torchvision import datasets
# Instantiate your uncompressed model
onnx_model = onnx.load_model("/model_path")
# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path")
dataset_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1)
# Step 1: Initialize transformation function
input_name = onnx_model.graph.input[0].name
def transform_fn(data_item):
images, _ = data_item
return {input_name: images.numpy()}
# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(onnx_model, calibration_dataset)
Below is an example of Accuracy Aware Quantization pipeline where model weights and compression parameters may be fine-tuned to achieve a higher accuracy.
import torch
import nncf.torch # Important - must be imported before any other external package that depends on torch
from nncf import NNCFConfig
from nncf.torch import create_compressed_model, register_default_init_args
# Instantiate your uncompressed model
from torchvision.models.resnet import resnet50
model = resnet50()
# Load a configuration file to specify compression
nncf_config = NNCFConfig.from_json("resnet50_int8.json")
# Provide data loaders for compression algorithm initialization, if necessary
import torchvision.datasets as datasets
representative_dataset = datasets.ImageFolder("/path")
init_loader = torch.utils.data.DataLoader(representative_dataset)
nncf_config = register_default_init_args(nncf_config, init_loader)
# Apply the specified compression algorithms to the model
compression_ctrl, compressed_model = create_compressed_model(model, nncf_config)
# Now use compressed_model as a usual torch.nn.Module
# to fine-tune compression parameters along with the model weights
# ... the rest of the usual PyTorch-powered training pipeline
# Export to ONNX or .pth when done fine-tuning
compression_ctrl.export_model("compressed_model.onnx")
torch.save(compressed_model.state_dict(), "compressed_model.pth")
NOTE (PyTorch): Due to the way NNCF works within the PyTorch backend, import nncf
must be done before any other import of torch
in your package or in third-party packages that your code utilizes, otherwise the compression may be applied incompletely.
import tensorflow as tf
from nncf import NNCFConfig
from nncf.tensorflow import create_compressed_model, register_default_init_args
# Instantiate your uncompressed model
from tensorflow.keras.applications import ResNet50
model = ResNet50()
# Load a configuration file to specify compression
nncf_config = NNCFConfig.from_json("resnet50_int8.json")
# Provide dataset for compression algorithm initialization
representative_dataset = tf.data.Dataset.list_files("/path/*.jpeg")
nncf_config = register_default_init_args(nncf_config, representative_dataset, batch_size=1)
# Apply the specified compression algorithms to the model
compression_ctrl, compressed_model = create_compressed_model(model, nncf_config)
# Now use compressed_model as a usual Keras model
# to fine-tune compression parameters along with the model weights
# ... the rest of the usual TensorFlow-powered training pipeline
# Export to Frozen Graph, TensorFlow SavedModel or .h5 when done fine-tuning
compression_ctrl.export_model("compressed_model.pb", save_format="frozen_graph")
For a more detailed description of NNCF usage in your training code, see this tutorial.
For a quicker start with NNCF-powered compression, try sample notebooks and scripts presented below.
A collection of ready-to-run Jupyter* notebooks are available to demonstrate how to use NNCF compression algorithms to optimize models for inference with the OpenVINO Toolkit:
Compact scripts demonstrating quantization and corresponding inference speed boost:
These examples provide full pipelines including compression, training and inference for classification, object detection and segmentation tasks.
NNCF may be straightforwardly integrated into training/evaluation pipelines of third-party repositories.
NNCF is integrated into OpenVINO Training Extensions as model optimization backend. So you can train, optimize and export new models based on the available model templates as well as run exported models with OpenVINO.
NNCF is used as a compression backend within the renowned transformers
repository in HuggingFace Optimum Intel.
See third_party_integration for examples of code modifications (Git patches and base commit IDs are provided) that are necessary to integrate NNCF into the following repositories:
For detailed installation instructions please refer to the Installation page.
NNCF can be installed as a regular PyPI package via pip:
pip install nncf
If you want to install both NNCF and the supported PyTorch version in one line, you can do this by simply running:
pip install nncf[torch]
Other viable options besides [torch]
are [tf]
, [onnx]
and [openvino]
.
NNCF is also available via conda:
conda install -c conda-forge nncf
You may also use one of the Dockerfiles in the docker directory to build an image with an environment already set up and ready for running NNCF sample scripts.
This repository is tested on Python* 3.8.10, PyTorch* 1.13.1 (NVidia CUDA* Toolkit 11.6) and TensorFlow* 2.11.1 (NVidia CUDA* Toolkit 11.2).
Results achieved using sample scripts, example patches to third-party repositories and NNCF configuration files provided with this repository. See README.md files for sample scripts and example patches to find instruction and links to exact configuration files and final checkpoints.
Model | Compression algorithm | Dataset | Accuracy (drop) % |
---|---|---|---|
ResNet-50 | INT8 | ImageNet | 76.46 (-0.31) |
ResNet-50 | INT8 (per-tensor only) | ImageNet | 76.39 (-0.24) |
ResNet-50 | Mixed, 43.12% INT8 / 56.88% INT4 | ImageNet | 76.05 (0.10) |
ResNet-50 | INT8 + Sparsity 61% (RB) | ImageNet | 75.42 (0.73) |
ResNet-50 | INT8 + Sparsity 50% (RB) | ImageNet | 75.50 (0.65) |
ResNet-50 | Filter pruning, 40%, geometric median criterion | ImageNet | 75.57 (0.58) |
Inception V3 | INT8 | ImageNet | 77.45 (-0.12) |
Inception V3 | INT8 + Sparsity 61% (RB) | ImageNet | 76.36 (0.97) |
MobileNet V2 | INT8 | ImageNet | 71.07 (0.80) |
MobileNet V2 | INT8 (per-tensor only) | ImageNet | 71.24 (0.63) |
MobileNet V2 | Mixed, 58.88% INT8 / 41.12% INT4 | ImageNet | 70.95 (0.92) |
MobileNet V2 | INT8 + Sparsity 52% (RB) | ImageNet | 71.09 (0.78) |
MobileNet V3 small | INT8 | ImageNet | 66.98 (0.68) |
SqueezeNet V1.1 | INT8 | ImageNet | 58.22 (-0.03) |
SqueezeNet V1.1 | INT8 (per-tensor only) | ImageNet | 58.11 (0.08) |
SqueezeNet V1.1 | Mixed, 52.83% INT8 / 47.17% INT4 | ImageNet | 57.57 (0.62) |
ResNet-18 | XNOR (weights), scale/threshold (activations) | ImageNet | 61.67 (8.09) |
ResNet-18 | DoReFa (weights), scale/threshold (activations) | ImageNet | 61.63 (8.13) |
ResNet-18 | Filter pruning, 40%, magnitude criterion | ImageNet | 69.27 (0.49) |
ResNet-18 | Filter pruning, 40%, geometric median criterion | ImageNet | 69.31 (0.45) |
ResNet-34 | Filter pruning, 50%, geometric median criterion + KD | ImageNet | 73.11 (0.19) |
GoogLeNet | Filter pruning, 40%, geometric median criterion | ImageNet | 69.47 (0.30) |
Model | Compression algorithm | Dataset | mAP (drop) % |
---|---|---|---|
SSD300-MobileNet | INT8 + Sparsity 70% (Magnitude) | VOC12+07 train, VOC07 eval | 62.95 (-0.72) |
SSD300-VGG-BN | INT8 | VOC12+07 train, VOC07 eval | 77.81 (0.47) |
SSD300-VGG-BN | INT8 + Sparsity 70% (Magnitude) | VOC12+07 train, VOC07 eval | 77.66 (0.62) |
SSD300-VGG-BN | Filter pruning, 40%, geometric median criterion | VOC12+07 train, VOC07 eval | 78.35 (-0.07) |
SSD512-VGG-BN | INT8 | VOC12+07 train, VOC07 eval | 80.04 (0.22) |
SSD512-VGG-BN | INT8 + Sparsity 70% (Magnitude) | VOC12+07 train, VOC07 eval | 79.68 (0.58) |
Model | Compression algorithm | Dataset | mIoU (drop) % |
---|---|---|---|
UNet | INT8 | CamVid | 71.89 (0.06) |
UNet | INT8 + Sparsity 60% (Magnitude) | CamVid | 72.46 (-0.51) |
ICNet | INT8 | CamVid | 67.89 (0.00) |
ICNet | INT8 + Sparsity 60% (Magnitude) | CamVid | 67.16 (0.73) |
UNet | INT8 | Mapillary | 56.09 (0.15) |
UNet | INT8 + Sparsity 60% (Magnitude) | Mapillary | 55.69 (0.55) |
UNet | Filter pruning, 25%, geometric median criterion | Mapillary | 55.64 (0.60) |
PyTorch Model | Compression algorithm | Dataset | Accuracy (Drop) % |
---|---|---|---|
BERT-base-chinese | INT8 | XNLI | 77.22 (0.46) |
BERT-base-cased | INT8 | CoNLL2003 | 99.18 (-0.01) |
BERT-base-cased | INT8 | MRPC | 84.8 (-0.24) |
BERT-large (Whole Word Masking) | INT8 | SQuAD v1.1 | F1: 92.68 (0.53) |
RoBERTa-large | INT8 | MNLI | matched: 89.25 (1.35) |
DistilBERT-base | INT8 | SST-2 | 90.3 (0.8) |
MobileBERT | INT8 | SQuAD v1.1 | F1: 89.4 (0.58) |
GPT-2 | INT8 | WikiText-2 (raw) | perplexity: 20.9 (-1.17) |
Model | Compression algorithm | Dataset | Accuracy (drop) % |
---|---|---|---|
Inception V3 | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations) | ImageNet | 78.39 (-0.48) |
Inception V3 | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations), Sparsity 61% (RB) | ImageNet | 77.52 (0.39) |
Inception V3 | Sparsity 54% (Magnitude) | ImageNet | 77.86 (0.05) |
MobileNet V2 | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations) | ImageNet | 71.63 (0.22) |
MobileNet V2 | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations), Sparsity 52% (RB) | ImageNet | 70.94 (0.91) |
MobileNet V2 | Sparsity 50% (RB) | ImageNet | 71.34 (0.51) |
MobileNet V2 (TensorFlow Hub MobileNet V2) | Sparsity 35% (Magnitude) | ImageNet | 71.87 (-0.02) |
MobileNet V3 (Small) | INT8 (per-channel symmetric for weights, per-tensor asymmetric half-range for activations) | ImageNet | 67.79 (0.59) |
MobileNet V3 (Small) | INT8 (per-channel symmetric for weights, per-tensor asymmetric half-range for activations) + Sparsity 42% (Magnitude) | ImageNet | 67.44 (0.94) |
MobileNet V3 (Large) | INT8 (per-channel symmetric for weights, per-tensor asymmetric half-range for activations) | ImageNet | 75.04 (0.76) |
MobileNet V3 (Large) | INT8 (per-channel symmetric for weights, per-tensor asymmetric half-range for activations) + Sparsity 42% (RB) | ImageNet | 75.24 (0.56) |
ResNet-50 | INT8 | ImageNet | 74.99 (0.06) |
ResNet-50 | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations) + Sparsity 65% (RB) | ImageNet | 74.36 (0.69) |
ResNet-50 | Sparsity 80% (RB) | ImageNet | 74.38 (0.67) |
ResNet-50 | Filter pruning, 40%, geometric median criterion | ImageNet | 74.96 (0.09) |
ResNet-50 | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations) + Filter pruning, 40%, geometric median criterion | ImageNet | 75.09 (-0.04) |
Model | Compression algorithm | Dataset | mAP (drop) % |
---|---|---|---|
RetinaNet | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations) | COCO 2017 | 33.12 (0.31) |
RetinaNet | Magnitude sparsity (50%) | COCO 2017 | 33.10 (0.33) |
RetinaNet | Filter pruning, 40% | COCO 2017 | 32.72 (0.71) |
RetinaNet | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations) + filter pruning 40% | COCO 2017 | 32.67 (0.76) |
YOLO v4 | INT8 (per-channel symmetric for weights, per-tensor asymmetric half-range for activations) | COCO 2017 | 46.20 (0.87) |
YOLO v4 | Magnitude sparsity, 50% | COCO 2017 | 46.49 (0.58) |
Model | Compression algorithm | Dataset | mAP (drop) % |
---|---|---|---|
Mask-R-CNN | INT8 (per-tensor symmetric for weights, per-tensor asymmetric half-range for activations) | COCO 2017 | 37.19 (0.14) |
Mask-R-CNN | Magnitude sparsity, 50% | COCO 2017 | 36.94 (0.39) |
ONNX Model | Compression algorithm | Dataset | Accuracy (Drop) % |
---|---|---|---|
ResNet-50 | INT8 (Post-Training) | ImageNet | 74.63 (0.21) |
ShuffleNet | INT8 (Post-Training) | ImageNet | 47.25 (0.18) |
GoogleNet | INT8 (Post-Training) | ImageNet | 66.36 (0.3) |
SqueezeNet V1.0 | INT8 (Post-Training) | ImageNet | 54.3 (0.54) |
MobileNet V2 | INT8 (Post-Training) | ImageNet | 71.38 (0.49) |
DenseNet-121 | INT8 (Post-Training) | ImageNet | 60.16 (0.8) |
VGG-16 | INT8 (Post-Training) | ImageNet | 72.02 (0.0) |
ONNX Model | Compression algorithm | Dataset | mAP (drop) % |
---|---|---|---|
SSD1200 | INT8 (Post-Training) | COCO2017 | 20.17 (0.17) |
Tiny-YOLOv2 | INT8 (Post-Training) | VOC12 | 29.03 (0.23) |
@article{kozlov2020neural,
title = {Neural network compression framework for fast model inference},
author = {Kozlov, Alexander and Lazarevich, Ivan and Shamporov, Vasily and Lyalyushkin, Nikolay and Gorbachev, Yury},
journal = {arXiv preprint arXiv:2002.08679},
year = {2020}
}
Refer to the CONTRIBUTING.md file for guidelines on contributions to the NNCF repository.