Improving neural network training based on Jacobian rank deficiency

Guian Zhou, Jennie Si

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Analysis and experimental results obtained in [1] have revealed that many network training problems are ill-conditioned and may not be solved efficiently by the Gauss-Newton method. The Levenberg-Marquardt algorithm has been used successfully in solving nonlinear least squares problems, however only for reasonable size problems due to its significant computation and memory complexities within each iteration. In the present paper we develop a new algorithm in the form of a modified Gauss-Newton which on one hand takes advantage of the Jacobian rank deficiency to reduce computation and memory complexities, and on the other hand, still has similar features to the Levenberg-Marquardt algorithm with better convergence properties than first order methods.

Original languageEnglish (US)
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer Verlag
Pages531-536
Number of pages6
Volume1112 LNCS
ISBN (Print)3540615105, 9783540615101
DOIs
StatePublished - 1996
Event1996 International Conference on Artificial Neural Networks, ICANN 1996 - Bochum, Germany
Duration: Jul 16 1996Jul 19 1996

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume1112 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other1996 International Conference on Artificial Neural Networks, ICANN 1996
CountryGermany
CityBochum
Period7/16/967/19/96

Fingerprint

Levenberg-Marquardt Algorithm
Neural Networks
Neural networks
Nonlinear Least Squares Problem
Gauss-Newton Method
Gauss-Newton
Convergence Properties
Data storage equipment
Newton-Raphson method
First-order
Iteration
Experimental Results
Training
Form

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Zhou, G., & Si, J. (1996). Improving neural network training based on Jacobian rank deficiency. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 1112 LNCS, pp. 531-536). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 1112 LNCS). Springer Verlag. https://doi.org/10.1007/3-540-61510-5_91

Improving neural network training based on Jacobian rank deficiency. / Zhou, Guian; Si, Jennie.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 1112 LNCS Springer Verlag, 1996. p. 531-536 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 1112 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zhou, G & Si, J 1996, Improving neural network training based on Jacobian rank deficiency. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 1112 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 1112 LNCS, Springer Verlag, pp. 531-536, 1996 International Conference on Artificial Neural Networks, ICANN 1996, Bochum, Germany, 7/16/96. https://doi.org/10.1007/3-540-61510-5_91
Zhou G, Si J. Improving neural network training based on Jacobian rank deficiency. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 1112 LNCS. Springer Verlag. 1996. p. 531-536. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/3-540-61510-5_91
Zhou, Guian ; Si, Jennie. / Improving neural network training based on Jacobian rank deficiency. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 1112 LNCS Springer Verlag, 1996. pp. 531-536 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{8e5443e1fe2847ec95c2397c333cd604,
title = "Improving neural network training based on Jacobian rank deficiency",
abstract = "Analysis and experimental results obtained in [1] have revealed that many network training problems are ill-conditioned and may not be solved efficiently by the Gauss-Newton method. The Levenberg-Marquardt algorithm has been used successfully in solving nonlinear least squares problems, however only for reasonable size problems due to its significant computation and memory complexities within each iteration. In the present paper we develop a new algorithm in the form of a modified Gauss-Newton which on one hand takes advantage of the Jacobian rank deficiency to reduce computation and memory complexities, and on the other hand, still has similar features to the Levenberg-Marquardt algorithm with better convergence properties than first order methods.",
author = "Guian Zhou and Jennie Si",
year = "1996",
doi = "10.1007/3-540-61510-5_91",
language = "English (US)",
isbn = "3540615105",
volume = "1112 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "531--536",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - Improving neural network training based on Jacobian rank deficiency

AU - Zhou, Guian

AU - Si, Jennie

PY - 1996

Y1 - 1996

N2 - Analysis and experimental results obtained in [1] have revealed that many network training problems are ill-conditioned and may not be solved efficiently by the Gauss-Newton method. The Levenberg-Marquardt algorithm has been used successfully in solving nonlinear least squares problems, however only for reasonable size problems due to its significant computation and memory complexities within each iteration. In the present paper we develop a new algorithm in the form of a modified Gauss-Newton which on one hand takes advantage of the Jacobian rank deficiency to reduce computation and memory complexities, and on the other hand, still has similar features to the Levenberg-Marquardt algorithm with better convergence properties than first order methods.

AB - Analysis and experimental results obtained in [1] have revealed that many network training problems are ill-conditioned and may not be solved efficiently by the Gauss-Newton method. The Levenberg-Marquardt algorithm has been used successfully in solving nonlinear least squares problems, however only for reasonable size problems due to its significant computation and memory complexities within each iteration. In the present paper we develop a new algorithm in the form of a modified Gauss-Newton which on one hand takes advantage of the Jacobian rank deficiency to reduce computation and memory complexities, and on the other hand, still has similar features to the Levenberg-Marquardt algorithm with better convergence properties than first order methods.

UR - http://www.scopus.com/inward/record.url?scp=84902166496&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84902166496&partnerID=8YFLogxK

U2 - 10.1007/3-540-61510-5_91

DO - 10.1007/3-540-61510-5_91

M3 - Conference contribution

AN - SCOPUS:84902166496

SN - 3540615105

SN - 9783540615101

VL - 1112 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 531

EP - 536

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

PB - Springer Verlag

ER -