Improvements on Cross-Validation: The .632+ Bootstrap Method Export

@article{citeulike:961919, abstract = {A training set of data has been used to construct a rule for predicting future responses. What is the error rate of this rule? This is an important question both for comparing models and for assessing a final selected model. The traditional answer to this question is given by cross-validation. The cross-validation estimate of prediction error is nearly unbiased but can be highly variable. Here we discuss bootstrap estimates of prediction error, which can be thought of as smoothed versions of cross-validation. We show that a particular bootstrap method, the .632+ rule, substantially outperforms cross-validation in a catalog of 24 simulation experiments. Besides providing point estimates, we also consider estimating the variability of an error rate estimate. All of the results here are nonparametric and apply to any possible prediction rule; however, we study only classification problems with 0-1 loss in detail. Our simulations include "smooth" prediction rules like Fisher's linear discriminant function and unsmooth ones like nearest neighbors.}, author = {Efron, Bradley and Tibshirani, Robert}, citeulike-article-id = {961919}, citeulike-linkout-0 = {http://dx.doi.org/10.2307/2965703}, citeulike-linkout-1 = {http://www.jstor.org/stable/2965703}, doi = {10.2307/2965703}, issn = {01621459}, journal = {Journal of the American Statistical Association}, keywords = {bootstrap, cross\_validation}, number = {438}, pages = {548--560}, posted-at = {2008-01-30 00:14:08}, priority = {3}, publisher = {American Statistical Association}, title = {Improvements on Cross-Validation: The .632+ Bootstrap Method}, url = {http://dx.doi.org/10.2307/2965703}, volume = {92}, year = {1997} }

TY - JOUR ID - citeulike:961919 L3 - citeulike-article-id:961919 N2 - A training set of data has been used to construct a rule for predicting future responses. What is the error rate of this rule? This is an important question both for comparing models and for assessing a final selected model. The traditional answer to this question is given by cross-validation. The cross-validation estimate of prediction error is nearly unbiased but can be highly variable. Here we discuss bootstrap estimates of prediction error, which can be thought of as smoothed versions of cross-validation. We show that a particular bootstrap method, the .632+ rule, substantially outperforms cross-validation in a catalog of 24 simulation experiments. Besides providing point estimates, we also consider estimating the variability of an error rate estimate. All of the results here are nonparametric and apply to any possible prediction rule; however, we study only classification problems with 0-1 loss in detail. Our simulations include "smooth" prediction rules like Fisher's linear discriminant function and unsmooth ones like nearest neighbors. IS - 438 JF - Journal of the American Statistical Association SN - 01621459 EP - 560 TI - Improvements on Cross-Validation: The .632+ Bootstrap Method PB - American Statistical Association VL - 92 SP - 548 KW - bootstrap KW - cross_validation AU - Efron, Bradley AU - Tibshirani, Robert PY - 1997/// UR - http://dx.doi.org/10.2307/2965703 ER -