Efficient sparse generalized multiple kernel learning

Haiqin Yang, Zenglin Xu, Jieping Ye, Irwin King, Michael R. Lyu

Research output: Contribution to journalArticle

69 Citations (Scopus)

Abstract

Kernel methods have been successfully applied in various applications. To succeed in these applications, it is crucial to learn a good kernel representation, whose objective is to reveal the data similarity precisely. In this paper, we address the problem of multiple kernel learning (MKL), searching for the optimal kernel combination weights through maximizing a generalized performance measure. Most MKL methods employ the L1-norm simplex constraints on the kernel combination weights, which therefore involve a sparse but non-smooth solution for the kernel weights. Despite the success of their efficiency, they tend to discard informative complementary or orthogonal base kernels and yield degenerated generalization performance. Alternatively, imposing the Lp-norm (p < 1) constraint on the kernel weights will keep all the information in the base kernels. This leads to non-sparse solutions and brings the risk of being sensitive to noise and incorporating redundant information. To tackle these problems, we propose a generalized MKL (GMKL) model by introducing an elastic-net-type constraint on the kernel weights. More specifically, it is an MKL model with a constraint on a linear combination of the L1-norm and the squared L2-norm on the kernel weights to seek the optimal kernel combination weights. Therefore, previous MKL problems based on the L1-norm or the L2-norm constraints can be regarded as special cases. Furthermore, our GMKL enjoys the favorable sparsity property on the solution and also facilitates the grouping effect. Moreover, the optimization of our GMKL is a convex optimization problem, where a local solution is the global optimal solution. We further derive a level method to efficiently solve the optimization problem. A series of experiments on both synthetic and real-world datasets have been conducted to show the effectiveness and efficiency of our GMKL.

Original languageEnglish (US)
Article number5696759
Pages (from-to)433-446
Number of pages14
JournalIEEE Transactions on Neural Networks
Volume22
Issue number3
DOIs
StatePublished - Mar 2011

Fingerprint

Learning
Weights and Measures
Convex optimization
Efficiency
Problem-Based Learning
Experiments
Noise

Keywords

  • Grouping effect
  • Kernel methods
  • level method
  • multiple Kernel learning

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Software
  • Medicine(all)

Cite this

Yang, H., Xu, Z., Ye, J., King, I., & Lyu, M. R. (2011). Efficient sparse generalized multiple kernel learning. IEEE Transactions on Neural Networks, 22(3), 433-446. [5696759]. https://doi.org/10.1109/TNN.2010.2103571

Efficient sparse generalized multiple kernel learning. / Yang, Haiqin; Xu, Zenglin; Ye, Jieping; King, Irwin; Lyu, Michael R.

In: IEEE Transactions on Neural Networks, Vol. 22, No. 3, 5696759, 03.2011, p. 433-446.

Research output: Contribution to journalArticle

Yang, H, Xu, Z, Ye, J, King, I & Lyu, MR 2011, 'Efficient sparse generalized multiple kernel learning', IEEE Transactions on Neural Networks, vol. 22, no. 3, 5696759, pp. 433-446. https://doi.org/10.1109/TNN.2010.2103571
Yang, Haiqin ; Xu, Zenglin ; Ye, Jieping ; King, Irwin ; Lyu, Michael R. / Efficient sparse generalized multiple kernel learning. In: IEEE Transactions on Neural Networks. 2011 ; Vol. 22, No. 3. pp. 433-446.
@article{9d29294606354e24802e259a55b61194,
title = "Efficient sparse generalized multiple kernel learning",
abstract = "Kernel methods have been successfully applied in various applications. To succeed in these applications, it is crucial to learn a good kernel representation, whose objective is to reveal the data similarity precisely. In this paper, we address the problem of multiple kernel learning (MKL), searching for the optimal kernel combination weights through maximizing a generalized performance measure. Most MKL methods employ the L1-norm simplex constraints on the kernel combination weights, which therefore involve a sparse but non-smooth solution for the kernel weights. Despite the success of their efficiency, they tend to discard informative complementary or orthogonal base kernels and yield degenerated generalization performance. Alternatively, imposing the Lp-norm (p < 1) constraint on the kernel weights will keep all the information in the base kernels. This leads to non-sparse solutions and brings the risk of being sensitive to noise and incorporating redundant information. To tackle these problems, we propose a generalized MKL (GMKL) model by introducing an elastic-net-type constraint on the kernel weights. More specifically, it is an MKL model with a constraint on a linear combination of the L1-norm and the squared L2-norm on the kernel weights to seek the optimal kernel combination weights. Therefore, previous MKL problems based on the L1-norm or the L2-norm constraints can be regarded as special cases. Furthermore, our GMKL enjoys the favorable sparsity property on the solution and also facilitates the grouping effect. Moreover, the optimization of our GMKL is a convex optimization problem, where a local solution is the global optimal solution. We further derive a level method to efficiently solve the optimization problem. A series of experiments on both synthetic and real-world datasets have been conducted to show the effectiveness and efficiency of our GMKL.",
keywords = "Grouping effect, Kernel methods, level method, multiple Kernel learning",
author = "Haiqin Yang and Zenglin Xu and Jieping Ye and Irwin King and Lyu, {Michael R.}",
year = "2011",
month = "3",
doi = "10.1109/TNN.2010.2103571",
language = "English (US)",
volume = "22",
pages = "433--446",
journal = "IEEE Transactions on Neural Networks and Learning Systems",
issn = "2162-237X",
publisher = "IEEE Computational Intelligence Society",
number = "3",

}

TY - JOUR

T1 - Efficient sparse generalized multiple kernel learning

AU - Yang, Haiqin

AU - Xu, Zenglin

AU - Ye, Jieping

AU - King, Irwin

AU - Lyu, Michael R.

PY - 2011/3

Y1 - 2011/3

N2 - Kernel methods have been successfully applied in various applications. To succeed in these applications, it is crucial to learn a good kernel representation, whose objective is to reveal the data similarity precisely. In this paper, we address the problem of multiple kernel learning (MKL), searching for the optimal kernel combination weights through maximizing a generalized performance measure. Most MKL methods employ the L1-norm simplex constraints on the kernel combination weights, which therefore involve a sparse but non-smooth solution for the kernel weights. Despite the success of their efficiency, they tend to discard informative complementary or orthogonal base kernels and yield degenerated generalization performance. Alternatively, imposing the Lp-norm (p < 1) constraint on the kernel weights will keep all the information in the base kernels. This leads to non-sparse solutions and brings the risk of being sensitive to noise and incorporating redundant information. To tackle these problems, we propose a generalized MKL (GMKL) model by introducing an elastic-net-type constraint on the kernel weights. More specifically, it is an MKL model with a constraint on a linear combination of the L1-norm and the squared L2-norm on the kernel weights to seek the optimal kernel combination weights. Therefore, previous MKL problems based on the L1-norm or the L2-norm constraints can be regarded as special cases. Furthermore, our GMKL enjoys the favorable sparsity property on the solution and also facilitates the grouping effect. Moreover, the optimization of our GMKL is a convex optimization problem, where a local solution is the global optimal solution. We further derive a level method to efficiently solve the optimization problem. A series of experiments on both synthetic and real-world datasets have been conducted to show the effectiveness and efficiency of our GMKL.

AB - Kernel methods have been successfully applied in various applications. To succeed in these applications, it is crucial to learn a good kernel representation, whose objective is to reveal the data similarity precisely. In this paper, we address the problem of multiple kernel learning (MKL), searching for the optimal kernel combination weights through maximizing a generalized performance measure. Most MKL methods employ the L1-norm simplex constraints on the kernel combination weights, which therefore involve a sparse but non-smooth solution for the kernel weights. Despite the success of their efficiency, they tend to discard informative complementary or orthogonal base kernels and yield degenerated generalization performance. Alternatively, imposing the Lp-norm (p < 1) constraint on the kernel weights will keep all the information in the base kernels. This leads to non-sparse solutions and brings the risk of being sensitive to noise and incorporating redundant information. To tackle these problems, we propose a generalized MKL (GMKL) model by introducing an elastic-net-type constraint on the kernel weights. More specifically, it is an MKL model with a constraint on a linear combination of the L1-norm and the squared L2-norm on the kernel weights to seek the optimal kernel combination weights. Therefore, previous MKL problems based on the L1-norm or the L2-norm constraints can be regarded as special cases. Furthermore, our GMKL enjoys the favorable sparsity property on the solution and also facilitates the grouping effect. Moreover, the optimization of our GMKL is a convex optimization problem, where a local solution is the global optimal solution. We further derive a level method to efficiently solve the optimization problem. A series of experiments on both synthetic and real-world datasets have been conducted to show the effectiveness and efficiency of our GMKL.

KW - Grouping effect

KW - Kernel methods

KW - level method

KW - multiple Kernel learning

UR - http://www.scopus.com/inward/record.url?scp=79952183228&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79952183228&partnerID=8YFLogxK

U2 - 10.1109/TNN.2010.2103571

DO - 10.1109/TNN.2010.2103571

M3 - Article

VL - 22

SP - 433

EP - 446

JO - IEEE Transactions on Neural Networks and Learning Systems

JF - IEEE Transactions on Neural Networks and Learning Systems

SN - 2162-237X

IS - 3

M1 - 5696759

ER -