A Caltech Library Service

Support Vector Machinery for Infinite Ensemble Learning

Lin, Hsuan-Tien and Li, Ling (2008) Support Vector Machinery for Infinite Ensemble Learning. Journal of Machine Learning Research, 9 . pp. 285-312. ISSN 1533-7928.

PDF - Published Version
See Usage Policy.


Use this Persistent URL to link to this item:


Ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of some base hypotheses. Nevertheless, most existing algorithms are limited to combining only a finite number of hypotheses, and the generated ensemble is usually sparse. Thus, it is not clear whether we should construct an ensemble classifier with a larger or even an infinite number of hypotheses. In addition, constructing an infinite ensemble itself is a challenging task. In this paper, we formulate an infinite ensemble learning framework based on the support vector machine (SVM). The framework can output an infinite and nonsparse ensemble through embedding infinitely many hypotheses into an SVM kernel. We use the framework to derive two novel kernels, the stump kernel and the perceptron kernel. The stump kernel embodies infinitely many decision stumps, and the perceptron kernel embodies infinitely many perceptrons. We also show that the Laplacian radial basis function kernel embodies infinitely many decision trees, and can thus be explained through infinite ensemble learning. Experimental results show that SVM with these kernels is superior to boosting with the same base hypothesis set. In addition, SVM with the stump kernel or the perceptron kernel performs similarly to SVM with the Gaussian radial basis function kernel, but enjoys the benefit of faster parameter selection. These properties make the novel kernels favorable choices in practice.

Item Type:Article
Related URLs:
URLURL TypeDescription
Additional Information:© 2008 Hsuan-Tien Lin and Ling Li. We thank Yaser Abu-Mostafa, Amrit Pratap, Kai-Min Chung, and the anonymous reviewers for valuable suggestions. Most of the work was done in 2005, in which Hsuan-Tien Lin was supported by the Caltech Center for Neuromorphic Systems Engineering under the US NSF Cooperative Agreement EEC-9402726, and Ling Li was sponsored by the Caltech SISL Graduate Fellowship.
Funding AgencyGrant Number
Caltech Center for Neuromorphic Systems EngineeringUNSPECIFIED
National Science Foundation Cooperative AgreementEEC-9402726
Caltech SISL Graduate FellowshipUNSPECIFIED
Subject Keywords:ensemble learning, boosting, support vector machine, kernel
Record Number:CaltechAUTHORS:LINjmlr08
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:11153
Deposited By: Archive Administrator
Deposited On:18 Jul 2008 22:51
Last Modified:03 Oct 2019 00:16

Repository Staff Only: item control page