Neural Networks: A Systematic Introduction

ISBN-13: 9783540605058 / Angielski / Miękka / 1996 / 502 str.

R. Rojas; Raul Rojas; J. Feldman

Opis

Neural networks are a computing paradigm that is finding increasing attention among computer scientists. In this book, theoretical laws and models previously scattered in the literature are brought together into a general theory of artificial neural nets. Always with a view to biology and starting with the simplest nets, it is shown how the properties of models change when more general computing elements and net topologies are introduced. Each chapter contains examples, numerous illustrations, and a bibliography. The book is aimed at readers who seek an overview of the field or who wish to deepen their knowledge. It is suitable as a basis for university courses in neurocomputing.

Recenzje

"If you want a systematic and thorough overview of neural networks, need a good reference book on this subject, or are giving or taking a course on neural networks, this book is for you." Computing Reviews

Spis treści

1. The Biological Paradigm.- 1.1 Neural computation.- 1.1.1 Natural and artificial neural networks.- 1.1.2 Models of computation.- 1.1.3 Elements of a computing model.- 1.2 Networks of neurons.- 1.2.1 Structure of the neurons.- 1.2.2 Transmission of information.- 1.2.3 Information processing at the neurons and synapses.- 1.2.4 Storage of information — learning.- 1.2.5 The neuron — a self-organizing system.- 1.3 Artificial neural networks.- 1.3.1 Networks of primitive functions.- 1.3.2 Approximation of functions.- 1.3.3 Caveat.- 1.4 Historical and bibliographical remarks.- 2. Threshold Logic.- 2.1 Networks of functions.- 2.1.1 Feed-forward and recurrent networks.- 2.1.2 The computing units.- 2.2 Synthesis of Boolean functions.- 2.2.1 Conjunction, disjunction, negation.- 2.2.2 Geometric interpretation.- 2.2.3 Constructive synthesis.- 2.3 Equivalent networks.- 2.3.1 Weighted and unweighted networks.- 2.3.2 Absolute and relative inhibition.- 2.3.3 Binary signals and pulse coding.- 2.4 Recurrent networks.- 2.4.1 Stored state networks.- 2.4.2 Finite automata.- 2.4.3 Finite automata and recurrent networks.- 2.4.4 A first classification of neural networks.- 2.5 Harmonic analysis of logical functions.- 2.5.1 General expression.- 2.5.2 The Hadamard—Walsh transform.- 2.5.3 Applications of threshold logic.- 2.6 Historical and bibliographical remarks.- 3.Weighted Networks — The Perceptron.- 3.1 Perceptrons and parallel processing.- 3.1.1 Perceptrons as weighted threshold elements.- 3.1.2 Computational limits of the perceptron model.- 3.2 Implementation of logical functions.- 3.2.1 Geometric interpretation.- 3.2.2 The XOR problem.- 3.3 Linearly separable functions.- 3.3.1 Linear separability.- 3.3.2 Duality of input space and weight space.- 3.3.3 The error function in weight space.- 3.3.4 General decision curves.- 3.4 Applications and biological analogy.- 3.4.1 Edge detection with perceptrons.- 3.4.2 The structure of the retina.- 3.4.3 Pyramidal networks and the neocognitron.- 3.4.4 The silicon retina.- 3.5 Historical and bibliographical remarks.- 4. Perceptron Learning.- 4.1 Learning algorithms for neural networks.- 4.1.1 Classes of learning algorithms.- 4.1.2 Vector notation.- 4.1.3 Absolute linear separability.- 4.1.4 The error surface and the search method.- 4.2 Algorithmic learning.- 4.2.1 Geometric visualization.- 4.2.2 Convergence of the algorithm.- 4.2.3 Accelerating convergence.- 4.2.4 The pocket algorithm.- 4.2.5 Complexity of perceptron learning.- 4.3 Linear programming.- 4.3.1 Inner points of polytopes.- 4.3.2 Linear separability as linear optimization.- 4.3.3 Karmarkar’s algorithm.- 4.4 Historical and bibliographical remarks.- 5. Unsupervised Learning and Clustering Algorithms.- 5.1 Competitive learning.- 5.1.1 Generalization of the perceptron problem.- 5.1.2 Unsupervised learning through competition.- 5.2 Convergence analysis.- 5.2.1 The one-dimensional case — energy function.- 5.2.2 Multidimensional case — the classical methods.- 5.2.3 Unsupervised learning as minimization problem.- 5.2.4 Stability of the solutions.- 5.3 Principal component analysis.- 5.3.1 Unsupervised reinforcement learning.- 5.3.2 Convergence of the learning algorithm.- 5.3.3 Multiple principal components.- 5.4 Some applications.- 5.4.1 Pattern recognition.- 5.4.2 Image compression.- 5.5 Historical and bibliographical remarks.- 6. One and Two Layered Networks.- 6.1 Structure and geometric visualization.- 6.1.1 Network architecture.- 6.1.2 The XOR problem revisited.- 6.1.3 Geometric visualization.- 6.2 Counting regions in input and weight space.- 6.2.1 Weight space regions for the XOR problem.- 6.2.2 Bipolar vectors.- 6.2.3 Projection of the solution regions.- 6.2.4 Geometric interpretation.- 6.3 Regions for two layered networks.- 6.3.1 Regions in weight space for the XOR problem.- 6.3.2 Number of regions in general.- 6.3.3 Consequences.- 6.3.4 The Vapnik—Chervonenkis dimension.- 6.3.5 The problem of local minima.- 6.4 Historical and bibliographical remarks.- 7. The Backpropagation Algorithm.- 7.1 Learning as gradient descent.- 7.1.1 Differentiable activation functions.- 7.1.2 Regions in input space.- 7.1.3 Local minima of the error function.- 7.2 General feed-forward networks.- 7.2.1 The learning problem.- 7.2.2 Derivatives of network functions.- 7.2.3 Steps of the backpropagation algorithm.- 7.2.4 Learning with backpropagation.- 7.3 The case of layered networks.- 7.3.1 Extended network.- 7.3.2 Steps of the algorithm.- 7.3.3 Backpropagation in matrix form.- 7.3.4 The locality of backpropagation.- 7.3.5 Error during training.- 7.4 Recurrent networks.- 7.4.1 Backpropagation through time.- 7.4.2 Hidden Markov Models.- 7.4.3 Variational problems.- 7.5 Historical and bibliographical remarks.- 8. Fast Learning Algorithms.- 8.1 Introduction — classical backpropagation.- 8.1.1 Backpropagation with momentum.- 8.1.2 The fractal geometry of backpropagation.- 8.2 Some simple improvements to backpropagation.- 8.2.1 Initial weight selection.- 8.2.2 Clipped derivatives and offset term.- 8.2.3 Reducing the number of floating-point operations.- 8.2.4 Data decorrelation.- 8.3 Adaptive step algorithms.- 8.3.1 Silva and Almeida’s algorithm.- 8.3.2 Delta-bar-delta.- 8.3.3 Rprop.- 8.3.4 The Dynamic Adaption algorithm.- 8.4 Second-order algorithms.- 8.4.1 Quickprop.- 8.4.2 QRprop.- 8.4.3 Second-order backpropagation.- 8.5 Relaxation methods.- 8.5.1 Weight and node perturbation.- 8.5.2 Symmetric and asymmetric relaxation.- 8.5.3 A final thought on taxonomy.- 8.6 Historical and bibliographical remarks.- 9. Statistics and Neural Networks.- 9.1 Linear and nonlinear regression.- 9.1.1 The problem of good generalization.- 9.1.2 Linear regression.- 9.1.3 Nonlinear units.- 9.1.4 Computing the prediction error.- 9.1.5 The jackknife and cross-validation.- 9.1.6 Committees of networks.- 9.2 Multiple regression.- 9.2.1 Visualization of the solution regions.- 9.2.2 Linear equations and the pseudoinverse.- 9.2.3 The hidden layer.- 9.2.4 Computation of the pseudoinverse.- 9.3 Classification networks.- 9.3.1 An application: NETtalk.- 9.3.2 The Bayes property of classifier networks.- 9.3.3 Connectionist speech recognition.- 9.3.4 Autoregressive models for time series analysis.- 9.4 Historical and bibliographical remarks.- 10. The Complexity of Learning.- 10.1 Network functions.- 10.1.1 Learning algorithms for multilayer networks.- 10.1.2 Hilbert’s problem and computability.- 10.1.3 Kolmogorov’s theorem.- 10.2 Function approximation.- 10.2.1 The one-dimensional case.- 10.2.2 The multidimensional case.- 10.3 Complexity of learning problems.- 10.3.1 Complexity classes.- 10.3.2 NP-complete learning problems.- 10.3.3 Complexity of learning with AND-OR networks.- 10.3.4 Simplifications of the network architecture.- 10.3.5 Learning with hints.- 10.4 Historical and bibliographical remarks.- 11. Fuzzy Logic.- 11.1 Fuzzy sets and fuzzy logic.- 11.1.1 Imprecise data and imprecise rules.- 11.1.2 The fuzzy set concept.- 11.1.3 Geometric representation of fuzzy sets.- 11.1.4 Fuzzy set theory, logic operators, and geometry.- 11.1.5 Families of fuzzy operators.- 11.2 Fuzzy inferences.- 11.2.1 Inferences from imprecise data.- 11.2.2 Fuzzy numbers and inverse operation.- 11.3 Control with fuzzy logic.- 11.3.1 Fuzzy controllers.- 11.3.2 Fuzzy networks.- 11.3.3 Function approximation with fuzzy methods.- 11.3.4 The eye as a fuzzy system — color vision.- 11.4 Historical and bibliographical remarks.- 12. Associative Networks.- 12.1 Associative pattern recognition.- 12.1.1 Recurrent networks and types of associative memories.- 12.1.2 Structure of an associative memory.- 12.1.3 The eigenvector automaton.- 12.2 Associative learning.- 12.2.1 Hebbian learning — the correlation matrix.- 12.2.2 Geometric interpretation of Hebbian learning.- 12.2.3 Networks as dynamical systems — some experiments.- 12.2.4 Another visualization.- 12.3 The capacity problem.- 12.4 The pseudoinverse.- 12.4.1 Definition and properties of the pseudoinverse.- 12.4.2 Orthogonal projections.- 12.4.3 Holographic memories.- 12.4.4 Translation invariant pattern recognition.- 12.5 Historical and bibliographical remarks.- 13. The Hopfield Model.- 13.1 Synchronous and asynchronous networks.- 13.1.1 Recursive networks with stochastic dynamics.- 13.1.2 The bidirectional associative memory.- 13.1.3 The energy function.- 13.2 Definition of Hopfield networks.- 13.2.1 Asynchronous networks.- 13.2.2 Examples of the model.- 13.2.3 Isomorphism between the Hopfield and Ising models.- 13.3 Converge to stable states.- 13.3.1 Dynamics of Hopfield networks.- 13.3.2 Covergence proof.- 13.3.3 Hebbian learning.- 13.4 Equivalence of Hopfield and perceptron learning.- 13.4.1 Perceptron learning in Hopfield networks.- 13.4.2 Complexity of learning in Hopfield models.- 13.5 Parallel combinatorics.- 13.5.1 NP-complete problems and massive parallelism.- 13.5.2 The multiflop problem.- 13.5.3 The eight rooks problem.- 13.5.4 The eight queens problem.- 13.5.5 The traveling salesman.- 13.5.6 The limits of Hopfield networks.- 13.6 Implementation of Hopfield networks.- 13.6.1 Electrical implementation.- 13.6.2 Optical implementation.- 13.7 Historical and bibliographical remarks.- 14. Stochastic Networks.- 14.1 Variations of the Hopfield model.- 14.1.1 The continuous model.- 14.2 Stochastic systems.- 14.2.1 Simulated annealing.- 14.2.2 Stochastic neural networks.- 14.2.3 Markov chains.- 14.2.4 The Boltzmann distribution.- 14.2.5 Physical meaning of the Boltzmann distribution.- 14.3 Learning algorithms and applications.- 14.3.1 Boltzmann learning.- 14.3.2 Combinatorial optimization.- 14.4 Historical and bibliographical remarks.- 15. Kohonen Networks.- 15.1 Self-organization.- 15.1.1 Charting input space.- 15.1.2 Topology preserving maps in the brain.- 15.2 Kohonen’s model.- 15.2.1 Learning algorithm.- 15.2.2 Mapping high-dimensional spaces.- 15.3 Analysis of convergence.- 15.3.1 Potential function — the one-dimensional case.- 15.3.2 The two-dimensional case.- 15.3.3 Effect of a unit’s neighborhood.- 15.3.4 Metastable states.- 15.3.5 What dimension for Kohonen networks?.- 15.4 Applications.- 15.4.1 Approximation of functions.- 15.4.2 Inverse kinematics.- 15.5 Historical and bibliographical remarks.- 16. Modular Neural Networks.- 16.1 Constructive algorithms for modular networks.- 16.1.1 Cascade correlation.- 16.1.2 Optimal modules and mixtures of experts.- 16.2 Hybrid networks.- 16.2.1 The ART architectures.- 16.2.2 Maximum entropy.- 16.2.3 Counterpropagation networks.- 16.2.4 Spline networks.- 16.2.5 Radial basis functions.- 16.3 Historical and bibliographical remarks.- 17. Genetic Algorithms.- 17.1 Coding and operators.- 17.1.1 Optimization problems.- 17.1.2 Methods of stochastic optimization.- 17.1.3 Genetic coding.- 17.1.4 Information exchange with genetic operators.- 17.2 Properties of genetic algorithms.- 17.2.1 Convergence analysis.- 17.2.2 Deceptive problems.- 17.2.3 Genetic drift.- 17.2.4 Gradient methods versus genetic algorithms.- 17.3 Neural networks and genetic algorithms.- 17.3.1 The problem of symmetries.- 17.3.2 A numerical experiment.- 17.3.3 Other applications of GAs.- 17.4 Historical and bibliographical remarks.- 18. Hardware for Neural Networks.- 18.1 Taxonomy of neural hardware.- 18.1.1 Performance requirements.- 18.1.2 Types of neurocomputers.- 18.2 Analog neural networks.- 18.2.1 Coding.- 18.2.2 VLSI transistor circuits.- 18.2.3 Transistors with stored charge.- 18.2.4 CCD components.- 18.3 Digital networks.- 18.3.1 Numerical representation of weights and signals.- 18.3.2 Vector and signal processors.- 18.3.3 Systolic arrays.- 18.3.4 One-dimensional structures.- 18.4 Innovative computer architectures.- 18.4.1 VLSI microprocessors for neural networks.- 18.4.2 Optical computers.- 18.4.3 Pulse coded networks.- 18.5 Historical and bibliographical remarks.

Blurb

Artificial neural networks are an alternative computational paradigm with roots in neurobiology which has attracted increasing interest in recent years. This book is a comprehensive introduction to the topic that stresses the systematic development of the underlying theory. Starting from simple threshold elements, more advanced topics are introduced, such as multilayer networks, efficient learning methods, recurrent networks, and self-organization. The various branches of neural network theory are interrelated closely and quite often unexpectedly, so the chapters treat the underlying connection between neural models and offer a unified view of the current state of research in the field.
The book has been written for anyone interested in understanding artificial neural networks or in learning more about them. The only mathematical tools needed are those learned during the first two years at university. The text contains more than 300 figures to stimulate the intuition of the reader and to illustrate the kinds of computation performed by neural networks. Material from the book has been used successfully for courses in Germany, Austria and the United States.