Advanced neural-network training algorithm with reduced complexity based on Jacobian deficiency

Guian Zhou, Jennie Si

Research output: Contribution to journalArticle

50 Scopus citations

Abstract

In this paper we introduce an advanced supervised training method for neural networks. It is based on Jacobian rank deficiency and it is formulated, in some sense, in the spirit of the Gauss-Newton algorithm. The Levenberg-Marquardt algorithm, as a modified Gauss-Newton, has been used successfully in solving nonlinear least squares problems including neural-network training. It outperforms (in terms of training accuracy, convergence properties, overall training time, etc.) the basic backpropagation and its variations with variable learning rate significantly, however, with higher computation and memory complexities within each iteration. The new method developed in this paper is aiming at improving convergence properties, while reducing the memory and computation complexities in supervised training of neural networks. Extensive simulation results are provided to demonstrate the superior performance of the new algorithm over the Levenberg-Marquardt algorithm.

Original languageEnglish (US)
Pages (from-to)448-453
Number of pages6
JournalIEEE Transactions on Neural Networks
Volume9
Issue number3
DOIs
StatePublished - Dec 1 1998

Keywords

  • Gauss-Newton method
  • Jacobian rank deficiency
  • Neural-network training
  • Subset updating
  • Trust region algorithms

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Advanced neural-network training algorithm with reduced complexity based on Jacobian deficiency'. Together they form a unique fingerprint.

  • Cite this