Hyperparameter Optimization of Machine Learning Algorithms

This code provides a hyper-parameter optimization implementation for machine learning algorithms, as described in the paper:
L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.

To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization.

This paper and code will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.

PS: A comprehensive Automated Machine Learning (AutoML) tutorial code can be found in: AutoML-Implementation-for-Static-and-Dynamic-Data-Analytics
- Including automated data pre-processing, automated feature engineering, automated model selection, hyperparameter optimization, and automated model updating (concept drift adaptation).

Paper

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
One-column version: arXiv
Two-column version: Elsevier

Quick Navigation

Section 3: Important hyper-parameters of common machine learning algorithms
Section 4: Hyper-parameter optimization techniques introduction
Section 5: How to choose optimization techniques for different machine learning models
Section 6: Common Python libraries/tools for hyper-parameter optimization
Section 7: Experimental results (sample code in "HPO_Regression.ipynb" and "HPO_Classification.ipynb")
Section 8: Open challenges and future research directions
Summary table for Sections 3-6: Table 2: A comprehensive overview of common ML models, their hyper-parameters, suitable optimization techniques, and available Python libraries
Summary table for Sections 8: Table 10: The open challenges and future directions of HPO research

Implementation

Sample code for hyper-parameter optimization implementation for machine learning algorithms is provided in this repository.

Sample code for Regression problems

HPO_Regression.ipynb
Dataset used: Boston-Housing

Sample code for Classification problems

HPO_Classification.ipynb
Dataset used: MNIST

Machine Learning & Deep Learning Algorithms

Random forest (RF)
Support vector machine (SVM)
K-nearest neighbor (KNN)
Artificial Neural Networks (ANN)

Hyperparameter Configuration Space

ML Model	Hyper-parameter	Type	Search Space
RF Classifier	n_estimators	Discrete	[10,100]
	max_depth	Discrete	[5,50]
	min_samples_split	Discrete	[2,11]
	min_samples_leaf	Discrete	[1,11]
	criterion	Categorical	'gini', 'entropy'
	max_features	Discrete	[1,64]
SVM Classifier	C	Continuous	[0.1,50]
	kernel	Categorical	'linear', 'poly', 'rbf', 'sigmoid'
KNN Classifier	n_neighbors	Discrete	[1,20]
ANN Classifier	optimizer	Categorical	'adam', 'rmsprop', 'sgd'
	activation	Categorical	'relu', 'tanh'
	batch_size	Discrete	[16,64]
	neurons	Discrete	[10,100]
	epochs	Discrete	[20,50]
	patience	Discrete	[3,20]
RF Regressor	n_estimators	Discrete	[10,100]
	max_depth	Discrete	[5,50]
	min_samples_split	Discrete	[2,11]
	min_samples_leaf	Discrete	[1,11]
	criterion	Categorical	'mse', 'mae'
	max_features	Discrete	[1,13]
SVM Regressor	C	Continuous	[0.1,50]
	kernel	Categorical	'linear', 'poly', 'rbf', 'sigmoid'
	epsilon	Continuous	[0.001,1]
KNN Regressor	n_neighbors	Discrete	[1,20]
ANN Regressor	optimizer	Categorical	'adam', 'rmsprop'
	activation	Categorical	'relu', 'tanh'
	loss	Categorical	'mse', 'mae'
	batch_size	Discrete	[16,64]
	neurons	Discrete	[10,100]
	epochs	Discrete	[20,50]
	patience	Discrete	[3,20]

HPO Algorithms

Grid search
Random search
Hyperband
Bayesian Optimization with Gaussian Processes (BO-GP)
Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE)
Particle swarm optimization (PSO)
Genetic algorithm (GA)

Requirements

Contact-Info

Please feel free to contact me for any questions or cooperation opportunities. I'd be happy to help.

Email: [email protected]
GitHub: LiYangHart and Western OC2 Lab
LinkedIn: Li Yang
Google Scholar: Li Yang and OC2 Lab

Citation

If you find this repository useful in your research, please cite this article as:

L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.

@article{YANG2020295,
title = "On hyperparameter optimization of machine learning algorithms: Theory and practice",
author = "Li Yang and Abdallah Shami",
volume = "415",
pages = "295 - 316",
journal = "Neurocomputing",
year = "2020",
issn = "0925-2312",
doi = "https://doi.org/10.1016/j.neucom.2020.07.061",
url = "http://www.sciencedirect.com/science/article/pii/S0925231220311693"
}

Project Name	Stars	Repos Using This	Packages Using This	Most Recent Commit	Total Releases	Latest Release	Open Issues	License	Language
Ray	25,902	80	199	4 hours ago	76	June 09, 2022	2,901	apache-2.0	Python
Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a toolkit of libraries (Ray AIR) for accelerating ML workloads.
D2l En	18,001			5 days ago			99	other	Python
Interactive deep learning book with multi-framework code, math, and discussions. Adopted at 400 universities from 60 countries including Stanford, MIT, Harvard, and Cambridge.
Nni	12,960	8	22	3 days ago	51	June 22, 2022	283	mit	Python
An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
Tpot	9,098	40	18	4 days ago	60	January 06, 2021	269	lgpl-3.0	Python
A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.
Optuna	8,173	26	214	10 hours ago	49	June 13, 2022	132	other	Python
A hyperparameter optimization framework
Wandb	6,144	39	232	7 hours ago	237	July 05, 2022	844	mit	Python
🔥 A tool for visualizing and tracking your machine learning experiments. This repo contains the CLI and Python API.
Autogluon	5,806			4 hours ago			239	apache-2.0	Python
AutoGluon: AutoML for Image, Text, Time Series, and Tabular Data
Polyaxon	3,323	4	11	4 hours ago	334	June 05, 2022	122	apache-2.0
MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle
Keras Tuner	2,718	2	44	24 days ago	16	March 25, 2022	193	apache-2.0	Python
A Hyperparameter Tuning Library for Keras
Mljar Supervised	2,625		2	3 days ago	77	March 02, 2022	118	mit	Python
Python package for AutoML on Tabular Data with Feature Engineering, Hyper-Parameters Tuning, Explanations and Automatic Documentation

Hyperparameter Optimization Of Machine Learning Algorithms