Classification Model Selection via Bilevel Programming

Cross validation is a well-known and widely used method used for model selection which involves searching a discretized grid for the combination of model hyper-parameters that minimizes the out-of-sample validation error — an estimate of the generalization error. This grid-search procedure effectively limits the size of the parameter space since many convex optimization problems must be solved at each grid point to evaluate its effectiveness. This paper proposes a novel formulation of cross-validation as a bilevel program which can systematically search the continuous hyper-parameter space. Also discussed are computational methods for solving a bilevel cross-validation program and numerical results that demonstrate the practicability of this approach for model selection in machine learning.

[BibTeX]