Local adaptive techniques in training neural networks

Loading...
Thumbnail Image
Date
2004-05
Authors
Mahat, Norpah
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The issue of changing the learning rates, 11, dynamically during training has been widely investigated and several techniques for learning rate adaptation have been proposed so far in acceleration strategies of neural networks. The use of these strategies aim at finding the proper learning rate that substitute for a small magnitude of the gradient in a flat region and decrease a large weight changes in a highly deep region. The algorithms for these techniques employ heuristics strategies to adapt the learning rates at each iteration and require fine tuning for other learning parameters that help to ensure a minimization of the error function along each weight direction. These adaptive learning strategies can be divided into two categories, Global and Local Adaptive Techniques. This thesis concentrates on Local Adaptive Techniques, namely, Learning Rate Adaptation by Sign Changes, SuperSAB, Delta-Bar-Delta Rule, Quickprop and Rprop. A description and implementation of these techniques will be presented. The effectiveness of the theoretical result is illustrated in three real life applications: Diagnosis of Breast Cancer Problem, Diabetes Diagnosis and Human Face Recognition Problem. Simulations are conducted to compare and evaluate the convergence behavior of these gradient-based training algorithms with various popular training algorithms. These techniques. have been compared and measured in terms of gradient and error function evaluations, and percentage of success. The analysis of the results indicates that the Local Adaptive Techniques accelerate the convergence rate of neural network learning, especially the Rprop and Quickprop techniques. All algorithms had proven their improvement over Backpropagation, the conventional neural network training method in terms of both convergence rate and convergence characteristics.
Description
Keywords
Techniques training , Neural networks
Citation