Classification Model Selection via Bilevel Programming

Published in Optimization Methods and Software, 2008

Recommended citation: G. Kunapuli, K. P. Bennett, J. Hu and J.-S. Pang. Classification Model Selection via Bilevel Programming Optimization Methods and Software , Volume 23, Issue 4 (2008), pp. 475-489, Special Issue on Mathematical Programming in Data Mining and Machine Learning, Guest Editors: Katya Scheinberg and Jiming Peng.

Cross validation is a well-known and widely used method used for model selection which involves searching a discretized grid for the combination of model hyper-parameters that minimizes the out-of-sample validation error — an estimate of the generalization error. This grid-search procedure effectively limits the size of the parameter space since many convex optimization problems must be solved at each grid point to evaluate its effectiveness. This paper proposes a novel formulation of cross-validation as a bilevel program which can systematically search the continuous hyper-parameter space. Also discussed are computational methods for solving a bilevel cross-validation program and numerical results that demonstrate the practicability of this approach for model selection in machine learning.