Bdl Benchmarks

Bayesian Deep Learning Benchmarks
Alternatives To Bdl Benchmarks
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Awesome Semantic Segmentation9,908
2 years ago13
:metal: awesome-semantic-segmentation
Fashion Mnist9,856
a year ago24mitPython
A MNIST-like fashion product database. Benchmark :point_down:
a day ago22May 05, 2022175apache-2.0Python
OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark
3 years ago17
Visual Tracking Paper List
Benchmarking Gnns2,137
2 months ago5mitJupyter Notebook
Repository for benchmarking graph neural networks
Benchm Ml1,839
9 months ago11mitR
A minimal benchmark for scalability, speed and accuracy of commonly used open source implementations (R packages, Python scikit-learn, H2O, xgboost, Spark MLlib etc.) of the top machine learning algorithms for binary classification (random forests, gradient boosted trees, deep neural networks etc.).
Fast Deep Equal1,596587,9781,4328 months ago14June 08, 202023mitJavaScript
The fastest deep equality check with Date, RegExp and ES6 Map, Set and typed arrays support
Pycm1,382587 days ago39April 27, 202212mitPython
Multi-class confusion matrix library in Python
a day ago2June 14, 2022106mitPython
Avalanche: an End-to-End Library for Continual Learning based on PyTorch.
a year ago9mitPython
State-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc.
Alternatives To Bdl Benchmarks
Select To Compare

Alternative Project Comparisons

Bayesian Deep Learning Benchmarks

This repository is no longer being updated.

Please refer to the Diabetic Retinopathy Detection implementation in Google's 'uncertainty-baselines' repo for up-to-date baseline implementations.


In order to make real-world difference with Bayesian Deep Learning (BDL) tools, the tools must scale to real-world settings. And for that we, the research community, must be able to evaluate our inference tools (and iterate quickly) with real-world benchmark tasks. We should be able to do this without necessarily worrying about application-specific domain knowledge, like the expertise often required in medical applications for example. We require benchmarks to test for inference robustness, performance, and accuracy, in addition to cost and effort of development. These benchmarks should be at a variety of scales, ranging from toy MNIST-scale benchmarks for fast development cycles, to large data benchmarks which are truthful to real-world applications, capturing their constraints.

Our BDL benchmarks should

  • provide a transparent, modular and consistent interface for the evaluation of deep probabilistic models on a variety of downstream tasks;
  • rely on expert-driven metrics of uncertainty quality (actual applications making use of BDL uncertainty in the real-world), but abstract away the expert-knowledge and eliminate the boilerplate steps necessary for running experiments on real-world datasets;
  • make it easy to compare the performance of new models against well tuned baselines, models that have been well-adopted by the machine learning community, under a fair and realistic setting (e.g., computational resources, model sizes, datasets);
  • provide reference implementations of baseline models (e.g., Monte Carlo Dropout Inference, Mean Field Variational Inference, Deep Ensembles), enabling rapid prototyping and easy development of new tools;
  • be independent of specific deep learning frameworks (e.g., not depend on TensorFlow, PyTorch, etc.), and integrate with the SciPy ecosystem (i.e., NumPy, Pandas, Matplotlib). Benchmarks are framework-agnostic, while baselines are framework-dependent.

In this repo we strive to provide such well-needed benchmarks for the BDL community, and collect and maintain new baselines and benchmarks contributed by the community. A colab notebook demonstrating the MNIST-like workflow of our benchmarks is available here.

We highly encourage you to contribute your models as new baselines for others to compete against, as well as contribute new benchmarks for others to evaluate their models on!

List of Benchmarks

Bayesian Deep Learning Benchmarks (BDL Benchmarks or bdlb for short), is an open-source framework that aims to bridge the gap between the design of deep probabilistic machine learning models and their application to real-world problems. Our currently supported benchmarks are:


BDL Benchmarks is shipped as a PyPI package (Python3 compatible) installable as:

pip3 install git+

The data downloading and preparation is benchmark-specific, and you can follow the relevant guides at baselines/<benchmark>/ (e.g. baselines/diabetic_retinopathy_diagnosis/


For example, the Diabetic Retinopathy Diagnosis benchmark comes with several baselines, including MC Dropout, MFVI, Deep Ensembles, and more. These models are trained with images of blood vessels in the eye:

The models try to predict diabetic retinopathy, and use their uncertainty for prescreening (sending patients the model is uncertain about to an expert for further examination). When you implement a new model, you can easily benchmark your model against existing baseline results provided in the repo, and generate plots using expert metrics (such as the AUC of retained data when referring 50% most uncertain patients to an expert):

You can even play with a colab notebook to see the workflow of the benchmark, and contribute your model for others to benchmark against.

Cite as

Please cite individual benchmarks when you use these, as well as the baselines you compare against. For the Diabetic Retinopathy Diagnosis benchmark please see here.


The repository is developed and maintained by the Oxford Applied and Theoretical Machine Learning group, with sponsorship from:

Contact Us

Email us for questions at [email protected], or submit any issues to improve the framework.

Popular Benchmark Projects
Popular Deep Learning Projects
Popular Software Performance Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Jupyter Notebook