Local adaptive techniques in training neural networks
Loading...
Date
2004-05
Authors
Mahat, Norpah
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The issue of changing the learning rates, 11, dynamically during training has been
widely investigated and several techniques for learning rate adaptation have been
proposed so far in acceleration strategies of neural networks. The use of these strategies
aim at finding the proper learning rate that substitute for a small magnitude of the
gradient in a flat region and decrease a large weight changes in a highly deep region.
The algorithms for these techniques employ heuristics strategies to adapt the learning
rates at each iteration and require fine tuning for other learning parameters that help to
ensure a minimization of the error function along each weight direction. These adaptive
learning strategies can be divided into two categories, Global and Local Adaptive
Techniques. This thesis concentrates on Local Adaptive Techniques, namely, Learning
Rate Adaptation by Sign Changes, SuperSAB, Delta-Bar-Delta Rule, Quickprop and
Rprop. A description and implementation of these techniques will be presented. The
effectiveness of the theoretical result is illustrated in three real life applications:
Diagnosis of Breast Cancer Problem, Diabetes Diagnosis and Human Face Recognition
Problem. Simulations are conducted to compare and evaluate the convergence behavior
of these gradient-based training algorithms with various popular training algorithms.
These techniques. have been compared and measured in terms of gradient and error
function evaluations, and percentage of success. The analysis of the results indicates
that the Local Adaptive Techniques accelerate the convergence rate of neural network
learning, especially the Rprop and Quickprop techniques. All algorithms had proven
their improvement over Backpropagation, the conventional neural network training
method in terms of both convergence rate and convergence characteristics.
Description
Keywords
Techniques training , Neural networks