A Caltech Library Service

The p-norm generalization of the LMS algorithm for adaptive filtering

Kivinen, Jyrki and Warmuth, Manfred K. and Hassibi, Babak (2006) The p-norm generalization of the LMS algorithm for adaptive filtering. IEEE Transactions on Signal Processing, 54 (5). pp. 1782-1793. ISSN 1053-587X.

See Usage Policy.


Use this Persistent URL to link to this item:


Recently much work has been done analyzing online machine learning algorithms in a worst case setting, where no probabilistic assumptions are made about the data. This is analogous to the H/sup /spl infin// setting used in adaptive linear filtering. Bregman divergences have become a standard tool for analyzing online machine learning algorithms. Using these divergences, we motivate a generalization of the least mean squared (LMS) algorithm. The loss bounds for these so-called p-norm algorithms involve other norms than the standard 2-norm. The bounds can be significantly better if a large proportion of the input variables are irrelevant, i.e., if the weight vector we are trying to learn is sparse. We also prove results for nonstationary targets. We only know how to apply kernel methods to the standard LMS algorithm (i.e., p=2). However, even in the general p-norm case, we can handle generalized linear models where the output of the system is a linear function combined with a nonlinear transfer function (e.g., the logistic sigmoid).

Item Type:Article
Related URLs:
URLURL TypeDescription
Additional Information:© Copyright 2006 IEEE. Reprinted with permission. Manuscript received December 1, 2004; revised June 26, 2005. [Posted online: 2006-04-18] This work was supported by the National Science Foundation under Grant CCR 9821087, the Australian Research Council, the Academy of Finland under Decision 210796, and the IST Programme of the European Community under PASCAL Network of Excellence IST-2002-506778. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Dominic K. C. Ho.
Subject Keywords:Adaptive filtering, Bregman divergences, H∞ optimality, least mean squares, online learning
Issue or Number:5
Record Number:CaltechAUTHORS:KIVieeetsp06
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:6281
Deposited By: Archive Administrator
Deposited On:30 Nov 2006
Last Modified:02 Oct 2019 23:31

Repository Staff Only: item control page