Bilevel Optimization and Machine Learning
Published in Computational Intelligence: Research Frontiers. Lecture Notes in Computer Science, 2008
Recommended citation: K. P. Bennett, G. Kunapuli, J. Hu and J.-S. Pang. Bilevel Optimization and Machine Learning Computational Intelligence: Research Frontiers. Lecture Notes in Computer Science. Volume 5050 (2008), pp. 25-47. IEEE World Congress on Computational Intelligence, WCCI 2008, Hong Kong, China, June 1-6, 2008, Plenary/Invited Lectures. http://gkunapuli.github.io/files/08regression.pdf
We examine the interplay of optimization and machine learning. Great progress has been made in machine learning by cleverly reducing machine learning problems to convex optimization problems with one or more hyper-parameters. The availability of powerful convex programming theory and algorithms has enabled a flood of new research in machine learning models and methods. But many of the steps necessary for successful machine learning models fall outside of the convex machine learning paradigm. Thus we now propose framing machine learning problems as Stackelberg games. The resulting bilevel optimization problem allows for efficient systematic search of large numbers of hyper-parameters. We discuss recent progress in solving these bilevel problems and the many interesting optimization challenges that remain. Finally, we investigate the intriguing possibility of novel machine learning models enabled by bilevel programming.