在 mlR 中计算 AUC Leave-One-Out 交叉验证？

Question

这是一个简单的问题，只是为了确保我不会以愚蠢的方式这样做。我想使用auc 作为我在 mlr 中的度量，并且由于样本量小，我也在使用 LOO。当然，在LOO交叉验证方案中，测试样本总是只有一个实例，所以auc是无法计算出来的。当然，我们可以在之后计算它，查看预测，当我们想将它用作嵌套交叉验证的内部循环中的度量时，就会出现问题。像这样的东西（你必须定义你自己的binaryTask）：

require(mlr)    
#for example purposes we will decide which one is better, vanilla LDA or
#vanilla SVM, in the task specified below
bls = list(makeLearner("classif.lda"),makeLearner("classif.svm"))
#modelMultiplexer allows us to search whole parameter spaces between models
#as if the models themselves were parameters
lrn = makeModelMultiplexer(bls)
#to calculate AUC we need some continuous output, so we set 
#predictType to probabilities
lrn = setPredictType(lrn, "prob")
lrn = setId(lrn, "Model Multiplexer")
#here we could pass the parameters to be tested to both SVM and LDA,
#let's not pass anything so we test the vanilla classifiers instead
ps = makeModelMultiplexerParamSet(lrn)
#finally, the resample strategy, Leave-One-Out ("LOO") in our case
rdesc = makeResampleDesc("LOO")
#parameter space search strategy, in our case we only have one parameter:
#the model. So, a simple grid search will do the trick
ctrl = makeTuneControlGrid()
#The inner CV loop where we choose the best model in the validation data
tune = makeTuneWrapper(lrn, rdesc, par.set = ps, control = ctrl, measure = auc, show.info = FALSE) 
#The outer CV loop where we obtain the performace of the selected model 
#in the test data. mlR is a great interface, we could have passed a list 
#of classifiers and tasks here instead and do it all in one go 
#(beware your memory limitation!)
res = benchmark(tune, binaryTask, rdesc, measure = auc)

你根本不能像这样在两个循环中使用auc。我们如何让mlr 评估所有测试样本的度量，而不是每次都进行唯一的重新采样？

Answer 1

您可以对内部循环使用不同的重采样策略，然后使用auc：

library(mlr)    

ps = makeParamSet(
  makeNumericLearnerParam(id = "cp", default = 0.01, lower = 0, upper = 1)
)
ctrl = makeTuneControlRandom(maxit = 10)
inner = makeResampleDesc("Subsample")
lrn = makeLearner("classif.rpart", predict.type = "prob")
tune = makeTuneWrapper(lrn, resampling = inner, par.set = ps, control = ctrl, measure = auc)

outer = makeResampleDesc("LOO")
r = resample(tune, bc.task, resampling = outer, extract = getTuneResult, measure = auc)

您也可以只获取重采样结果并在其上计算任意性能度量，例如performance(r$pred, auc).