The subsemble package is an R implementation of the Subsemble algorithm. Subsemble is a general subset ensemble prediction method, which can be used for small, moderate, or large datasets. Subsemble partitions the full dataset into subsets of observations, fits a specified underlying algorithm on each subset, and uses a unique form of k-fold cross-validation to output a prediction function that combines the subset-specific fits. An oracle result provides a theoretical performance guarantee for Subsemble.
Stephanie Sapp, Mark J. van der Laan & John Canny (2014) Subsemble: An ensemble method for combining subset-specific algorithm fits. Journal of Applied Statistics, 41(6):1247-1259.
Implementation details of the subsemble R package available here:
LeDell, E. (2015) Scalable Ensemble Learning and Computationally Efficient Variance Estimation (Doctoral Dissertation). University of California, Berkeley, USA. https://github.com/ledell/phd-thesis/blob/main/ledell-phd-thesis.pdf
You can install:
the latest released version from CRAN with
install.packages("subsemble")
the latest development version from GitHub with
devtools::install_github("ledell/subsemble")
Here are some examples of how to use the subsemble package to do various types of learning tasks. These examples are also part of the subsemble
function documentation in the R package.
Load some example binary outcome data to use in all the examples below.
library(subsemble)
library(cvAUC) # >= version 1.0.1
data(admissions) # From cvAUC package
# Training data.
x <- subset(admissions, select = -c(Y))[1:400,]
y <- admissions$Y[1:400]
# Test data.
newx <- subset(admissions, select = -c(Y))[401:500,]
newy <- admissions$Y[401:500]
To set up a Subsemble, the user must decide on a base learner library (specified in the learner
argument) and a metalearning algorithm (specified in the metalearner
argument).
The Subsemble below uses only two base learners -- a Random Forest and a GLM. Both "SL.randomForest"
and "SL.glm"
are algorithm wrapper functions from the SuperLearner R package.
learner <- c("SL.randomForest", "SL.glm")
metalearner <- "SL.glm"
subsets <- 2
To customize the base learners, you can wrap these functions with another function, passing in any non-default arguments. For example, we can create a Random Forest with ntree = 1500
and mtry = 2
as follows:
SL.randomForest.1 <- function(..., ntree = 1500, mtry = 2) {
SL.randomForest(..., ntree = ntree, mtry = mtry)
}
This custom base learner can be included in the learner library as follows:
learner <- c("SL.randomForest.1", "SL.glm")
In the subsets
argument, the user can specify the number of random partitions of the training data. For instance, if subsets = 3
, then the training observations will be split into three roughly equal-sized training subsets. Alternatively, the subsets can be explicitly defined by specifying subsets
as a list of length J
(for J subsets). This will be a list of vector of of row indices.
The subsemble
function has one argument that is specific to the learning of the ensemble, and that is a list called learnControl
. Currently, there is just one element in this list, multiType
, which defines the "type" of subsemble multi-algorithm learning. The default, learnControl[["multiType"]] = "crossprod"
, stands for the "cross-product" type. This means that every subset is trained with every learner in the base library. In this example, the Subsemble will be a combination of 4 models (2 subsets x 2 learners).
learner <- c("SL.randomForest", "SL.glm")
metalearner <- "SL.glm"
subsets <- 2
fit <- subsemble(x = x, y = y, newx = newx, family = binomial(),
learner = learner, metalearner = metalearner,
subsets = subsets)
Since the newx
argument was not NULL
in the command above, the subsemble
function will automatically generate the predicted values for the test set. The ensemble predicted values for the test set can be found in the fit$pred
object. We can estimate model performance by calculating AUC on the test set.
auc <- AUC(predictions = fit$pred, labels = newy)
print(auc) # Test set AUC is: 0.9431063
By default, newx
will be NULL
and will return the predicted values for the training data. If newx = NULL
in the statement above, we could use the predict
method to generate predictions on new data after training the subsemble fit.
pred <- predict(fit, newx)
auc <- AUC(predictions = pred$pred, labels = newy)
print(auc) # Test set AUC is: 0.9431063
Next, we can modify the learnControl
argument and then train and test a new Subsemble and compare the results to the default model settings. With learnControl[["multiType"] = "divisor"
, we ensemble only 2 models (one for each subset). In the "divisor" type, the number of models that make up the subsemble is always the same as the number of subsets. If the number of unique learners is a divisor of the number of subsets (e.g. length(learner) == 2
and subsets = 8
), then the base learning algorithms will be recycled as neccessary to get a total of 8 models in the Subsemble.
The "divisor" type of multi-algorithm subsemble will train faster (since there are fewer constituent models), but will likely have worse model performance than the default "cross-product" method. Therefore, you may choose to use the "divisor" method for rapid testing / exploration, but not neccessarily for your final ensemble.
fit <- subsemble(x = x, y = y, newx = newx, family = binomial(),
learner = learner, metalearner = metalearner,
subsets = subsets,
learnControl = list(multiType = "divisor"))
auc <- AUC(predictions = fit$pred, labels = newy)
print(auc) # Test set AUC is: 0.9381229
The subsemble
function can also be used with a single base learner. In this example, there are 3 subsets and 1 learner, for a total of 3 models in the ensemble.
learner <- c("SL.randomForest")
metalearner <- "SL.glmnet"
subsets <- 3
fit <- subsemble(x = x, y = y, newx = newx, family = binomial(),
learner = learner, metalearner = metalearner,
subsets = subsets)
auc <- AUC(predictions = fit$pred, labels = newy)
print(auc) # Test set AUC is: 0.9356312
An ensemble can also be created using the full training data by setting subsets = 1
. This is equivalent to the Super Learner algorithm. In the example below, we have an ensemble of 2 models (one for each of the 2 learners).
learner <- c("SL.randomForest", "SL.glm")
metalearner <- "SL.glm"
subsets <- 1
fit <- subsemble(x = x, y = y, newx = newx, family = binomial(),
learner = learner, metalearner = metalearner,
subsets = subsets)
auc <- AUC(predictions = fit$pred, labels = newy)
print(auc) # Test set AUC is: 0.9522425
One of the benefits to using Subsemble (as opposed to other ensemble algorithms) is that the learning process can be broken up into smaller learning problems that can be easily parallelized across multiple cores or nodes. This is especially useful in cases where there is not enough memory on your machine(s) to train an ensemble on the full training set.
To perform the cross-validation and training steps in parallel using all available cores, use the parallel = "multicore"
option, which will use the native parallel package in R for parallel processing. It is recommended to use this option if you want to speed up the training (it will use all cores).
learner <- c("SL.randomForest", "SL.glm")
metalearner <- "SL.glm"
subsets <- 2
fit <- subsemble(x = x, y = y, newx = newx, family = binomial(),
learner = learner, metalearner = metalearner,
subsets = subsets, parallel = "multicore")
auc <- AUC(predictions = fit$pred, labels = newy)
print(auc) # Test set AUC is: 0.9431063
To perform the cross-validation and training steps using a SNOW cluster, the user will need to set up a SNOW cluster and pass the cluster object to the subsemble
function via the parallel
argument. If using the "SOCK"
cluster type, make sure to export any functions that are needed on the cluster nodes, like the algorithm wrapper functions.
This example uses the doSNOW library to create a 4-node cluster.
library(doSNOW)
nodes <- 4
cl <- makeCluster(nodes, type = "SOCK")
registerDoSNOW(cl)
clusterExport(cl, c(learner, metalearner))
fit <- subsemble(x = x, y = y, newx = newx, family = binomial(),
learner = learner, metalearner = metalearner,
subsets = subsets, parallel = cl)
stopCluster(cl)
auc <- AUC(predictions = fit$pred, labels = newy)
print(auc) # Test set AUC is: 0.9431063