Systematic and effective supervised learning mechanism based on Jacobian rank deficiency

Most of neural network applications rely on the fundamental approximation property of feedforward networks. In a realistic problem setting, a mechanism is needed to devise this learning process based on available data, starting from choosing an appropriate set of parameters in order to avoid overfitting, to an efficient learning algorithm measured by computation and memory complexities, as well as the accuracy of the training procedure (measured by the training error), and not to forget testing and cross-validation for generalization. Many of these aspects have been addressed in literature, however individually or ineffectively. In the present paper we develop a comprehensive procedure to address the above issues in a systematic manner. This process is based on a common observation of Jacobian rank deficiency. A new numerical procedure for solving the nonlinear optimization problem in supervised learning is introduced which not only reduces the training time and overall complexity but also achieves good training accuracy and generalization.