Neural Systems: Fundamentals Through Simulations

Neural and Adaptive Systems: Fundamentals Through Simulations

Preface

This book presents neurocomputing from a different perspective. Throughout the discussion we blend the power of a software simulator with the theory of neurocomputing to constantly reinforce the synergism between the conceptual equations and their behavior in practical neural systems. In our presentation model, which we call an interactive electronic book (i-book) [Principe et al., 2000], the text co-exists with a functional simulator. This is much different than the two common methods of integrating a text and computer software. We do not simply add computer-based examples to the end of the text, we use computer simulations early and often in the presentation. In addition, the i-book is not simply a laboratory manual where exercises are wrapped with text. We have reorganized the theory of neurocomputing into fine-grain conceptual modules, each of which is illustrated with a simple simulation to enhance conceptual understanding throughout the text. At the end of every chapter there is at least one full simulation, which is incrementally built and explained throughout the chapter, that can be used for a project. The simulation thus becomes an essential piece of information delivery, deeply affecting the organization of the textbook, and allowing visualization and reader interaction with the material.

Our intent is to present the concepts at a level that can be understood by senior-level undergraduate students in science and engineering. The link between the hypertext and the simulator creates an interactive learning environment especially appropriate for self-guided study of neural networks and adaptive systems. Our more practical, simulation-driven approach will be particularly beneficial to professionals in the fields of science, engineering, and economics.

We strive to fully cover the major topics in neurocomputing, from theory to applications. Because adaptation is the key to neurocomputing, we give it special attention, covering both the supervised and unsupervised paradigms. We treat neural networks as a nonlinear extension of linear adaptive systems, thus discussing both linear and nonlinear adaptive topologies in detail. In fact, we start with the well-known regression problem to introduce the idea of data fitting and many of the concepts of adaptation. Time, so important for engineering applications, is covered from the perspective of digital signal processing and is integrated with neural topologies. Our hope is to show that adaptive filters and neural networks, normally taught in two different disciplines, can be integrated under the common theme of neural and adaptive systems. Each chapter contains extensive simulation examples (more than 200 simulations overall) to illustrate the concepts, and to provide for further exploration with instructor or student supplied data. We emphasize the design and use of neural and adaptive systems. Readers can use the electronic book in other disciplines, or for senior projects, master theses, or even Ph.D. dissertations.

The textbook is presented in two formats: an electronic version on CD-ROM, consisting of a hypertext document linked with a software simulator, and the paper version duplicating the text material.

Electronic Version

The electronic version is a hypertext document in the Windows help format, so any PC with the Windows NT or Windows 95 (or higher) operating system will be able to install and run the interactive book. The hypertext is linked to NeuroSolutions, a neural network simulator developed by NeuroDimension that is included on the CD-ROM. The simulator is called from the hypertext by clicking on an icon. The simulation examples include fully functional neural networks with explanations of the network topology, and directions for the reader to use and study the relevant aspects of the simulation. A key point is that the simulator is open to experimentation: students can change parameters and topologies, and open new probes to answer “what if” questions, which normally leads toward a much deeper understanding of the concepts. Once the simulations are complete, control passes to the point where the simulator was called. Hence a seamless integration of the text with the simulator is achieved.

Because of this organization, it is very important that the reader (as well as the instructor) learns how to interact with NeuroSolutions. We include a NeuroSolutions tutorial as Appendix B , and we strongly encourage everyone to consult and run the different examples in the tutorial in order to learn how to create, configure, probe, and adequately control the simulator.

The text is geared toward the presentation of the fundamental concepts of adaptive systems and how they can be applied in practice. Derivation of equations is encapsulated in hot links (which we call “know-more” boxes) that can be called from the main text but can be skipped in a first reading pass. Important definitions, equations, and references are accessible in pop-up windows.

The text can be navigated in several different ways. The beginning of each chapter contains hyperlinks to its section headings, and a hyperlink to the preface, as well as the next chapter. At the end of each section there is a hyperlink to the next section, providing a sequential, paper-book style of organization. Hot links to the appendix are called from the text (return to the text with the Back button in the control bar). At the end of each chapter we present hyperlinks to all the examples of the chapter, and a concept map with hot-links to all the chapter section headings and to conceptually related chapters.

Using the Electronic Book

As mentioned above, the electronic text is in the Windows Help file format. For those not familiar with the Windows Help system, this section will briefly introduce the navigation through the book.

Navigation through help files is done by using the help file toolbar and through hyperlinks. The following is an image of the help file toolbar:

By pressing the [image] button you move to the next topic in the sequence. Pressing the [image] moves to the previous topic in the sequence. If you click on a hyperlink (described below), press the [image] button to return to the original position in the text. Pressing the [image] button brings up the table of contents and key word search panel shown below.

Selecting a chapter and clicking "Open" (or double-clicking the chapter) in the table of contents opens all the subsections of that chapter. Clicking display on a subsection moves immediately to that section of the text. Clicking on the "Index" tab of the window allows you to search through the keywords of the text. Clicking on the "Find" tab allows you to search through the text of the book.

There are three types of hyperlinks

  • Pop-up windows that are used for definintions, references, and footnotes,

  • Know More Boxes that contain more detailed information such as derivations, and

  • Hyperlinks to related topics in other sections or chapters. Press the "back" button in the help file toolbar to return to the original position in the text.

Simply click on any of these hyperlinks to see the related information.

Paper Version

The paper version is a carbon copy of the electronic hypertext and it is composed of three fundamental elements: the text, the “know-more” boxes, and the NeuroSolutions examples. The know-more boxes are marked in the left margin with gray diamonds, and they contain further details about the topic (normally derivations). They correspond to the hot-links in the hypertext. They can be skipped in a first reading, but no thorough understanding of the material is possible without studying them. The NeuroSolutions examples are shaded boxes, and they contain the details to run the simulator.

We suggest that even when using the paper version of the textbook, the reader should execute the examples on the computer as soon as they appear in the text because they are intrinsically bound to the material, and enhance the understanding. For convenience, at the end of each chapter in the electronic version we provide a list of the examples with their corresponding hot-links as a quick reference for readers of the paper version. After executing the example, the reader can go back to the example listing by clicking on the Back button of the control bar.

Topics

The material is divided into 11 chapters, 3 appendices, and a glossary. Chapter 1 covers the concept of data fitting with the linear model, and at the same time, gradient-descent learning. We decided to start with the linear regression model because it is at least vaguely familiar to students in science, economics, and engineering. Linear regression exemplifies the central theme of the book, which is adapting parameters from data. The link between least squares and the search for the minimum of the mean square error is established in this chapter. Gradient-descent learning and the elegant LMS algorithm are also presented here. The style of computation and the properties of the iterated solution are covered for single and multiple regression. Newton's method is presented as well. At the end of the first chapter a project using data from the Internet is set up to help students apply what they learned to real- world data.

Chapter 2 presents the concepts of statistical pattern recognition. This chapter formulates classification as the placement of discriminant functions in pattern space to minimize the probability of the classification error. We stress the difference between data fitting and classification but show that the same basic methodology can be used to train both classifiers and regressors. The design of the Bayes’ classifier is introduced. We also present the concept of using high dimensional spaces to simplify the placement of the discriminant function. The concept of linear and quadratic classifiers,and their links to likelihoods are also discussed.

Chapter 3 studies multilayer perceptrons (MLPs). We systematically treat the processing power of the topology and the adaptation of the parameters as two different aspects of the problem. We start very simply, with a single McCulloch-Pitts processing element in two- dimensional space to build up intuition. We define the perceptron and show how to adapt its parameters with gradient-descent learning by using the chain rule. We also present a large-margin learning rule (the Adatron algorithm) linked to the perceptron learning rule. We then introduce the need for more powerful topologies. The study of the one-hidden-layer MLP is first conducted with fixed parameters (in terms of discriminant functions), and then backpropagation is derived and applied. The two-hidden-layer MLP is also studied as a topology and adapted using backpropagation. The method of ordered derivatives, important for practical implementations, is covered, and its implications for constructing general-purpose simulators are explored. Since NeuroSolutions uses a data-flow implementation, it very clearly illustrates the concepts of backpropagation and ordered derivatives. Training embedded adaptive subsystems in larger fixed parameters systems, so often forgotten but so relevant to solving practical problems, is also addressed with examples. The chapter finishes with the interpretation of an MLP as an a posteriori probability estimator.

Chapter 4 delves into the details of how to use MLPs in real-world applications. The practical aspects of initialization, alternative search methods, topology size, and stopping criterion are all addressed. We start by pointing out the stochastic nature of training and its implications. We cover only first-order search methods in detail (gradient descent, momentum, adaptive step sizes, and noise), but we present the full framework. Cross-validation is presented as the preferred method of stopping the training, but other methods are also covered. For good generalization, we demonstrate the use of weight decay as a way to control the size of topologies. Norm selection is also addressed as an extra control of performance. We explain committees as a practical way of controlling the variance of the estimator. After covering all of these aspects, we present two practical applications of MLPs, and we set up a simulation again using the Internet.

Chapter 5 presents the unifying view that regression and classification are special cases of the more general problem of function approximation. Adaptive systems should ultimately be interpreted as parametric function approximators. This chapter is a bit more demanding in terms of mathematics, but the goal is to introduce the problem of function approximation and the use of MLPs, and radial basis functions (RBFs) as universal function approximators. The statistical view of nonlinear regression is provided, and RBFs are also used as classifiers. We finish the chapter by providing a very brief introduction to support vector machines. Two projects, one with financial data and the other with housing prices, are constructed in the simulator.

Chapter 6 covers the principles of Hebbian learning. The estimation of correlation with local rules is stressed and shown to be a universal principle in learning systems. We present principal component analysis (PCA) as a robust and efficient data reduction methodology. The extensions of Hebbian learning to forced- and anti-Hebbian learning are also covered. The linear associative memories and its applications both to hetero-association and autoassociation receive a special place in the presentation.

Chapter 7 is devoted to competition, the other major learning principle. We apply competitive networks to clustering, and develop the Kohonen self-organizing network from the point of view of a topology-preserving map. Modular networks are briefly reviewed, and applications are presented.

Chapter 8 covers the fundamentals of extracting (and quantifying) information in time. The relationship between time signals and vectors is stressed. We motivate filtering as projections in vector spaces. This chapter is a summary of the fundamental concepts of linear signal and systems analysis. The concepts of impulse response, convolution, frequency response, and transfer function as descriptors of system properties are emphasized. This chapter also covers Fourier transforms, and a simple filter design method.

The remaining chapters exploit combinations of previous topics. Chapter 9 brings together the concepts of regression and time processing in the form of adaptive filters. The idea of optimal filters is presented, and the LMS algorithm is utilized for adaptation. We present system identification from the point of view of function approximation. Hebbian learning extended to time produces eigenfilters and Karhunen-Loeve transforms. This chapter contains a showcase of examples to illustrate the exceptional applicability of the simple linear adaptive network. Over 20 practical examples are offered, including

  • System identification

  • Prediction

  • Model-based spectral analysis

  • Noise cancellation

  • Interference cancellation

  • Echo cancellation

  • Inverse modeling

  • Inverse adaptive controls

  • Eigenfiltering

  • PCA in time

  • Subspace spectral analysis

  • Blind source separation

Readers may modify all of these models to solve their own problems.

Chapter 10 brings together time and nonlinear processing elements to yield systems that can be interpreted either as extensions of the MLP to time processing, or as nonlinear extensions of adaptive filters. The topologies covered are called time-lagged-feedforward networks (TLFNs), and they are feedforward combinations of linear filters with nonlinear PEs. The adaptation can still be performed with static backpropagation. The design of short-term memories is covered in detail. Many problems can be solved using these intermediate topologies. We present examples of temporal pattern recognition, nonlinear system identification, and nonlinear prediction.

Chapter 11 teaches the training of recurrent topologies in time, and presents the general case of first-order distributed dynamic systems. We extend static backpropagation to the training of recurrent systems, for both static patterns (fixed point learning) and for trajectory learning. Real-time recurrent learning (RTRL) and backpropagation through time (BPTT) are presented and compared. We discuss applications to nonlinear system identification and control. This chapter covers the Hopfield model, and the concept of computational energy. The hierarchy of neural models is introduced with the Grossberg additive model. The last topic is an extension of first-order dynamics to systems that are locally stable, but globally chaotic, to illustrate some of the challenges ahead in the field of neurocomputing.

Appendix A provides a topical view of matrix computations and probabilities, and is a quick reference to the underlying concepts necessary to understand the text material. Appendix B is the NeuroSolutions tutorial and it is a must for the student (and instructor) who is not familiar with the software package. It contains the basic concepts to construct networks, configure and probe components, and set up the simulations. A description of the data included on the CD-ROM is found in Appendix C . For quick reference, we also include a glossary section with definitions of the terms used in the text for quick reference.

The extensive coverage of topics makes the book impossible to cover in one semester, but permits different organizations of topics depending on the special goals of the instructor. Many possible topic sequences are possible. Chapters 1 through 5 form a short sequence that can be used as an introduction to neural networks covering regression, multi-layer perceptrons, and radial basis functions. A second course could cover chapters 6, 7, 9, 10, and 11, which include unsupervised learning rules, SOMs, and temporal neural networks. Chapters 1, 8, and 9 cover linear adaptive systems and filters and can be used separately or as introduction to the nonlinear systems.

Classroom Experience and the Simulator

The i-book can be used in a normal classroom setting, improving the explanation of the material through the simulations with real data, and providing the students with a way to practice at home what has been demonstrated in class. However, the full potential of the i-book is achieved with some improvements to the conventional classroom format. We would like to share our experience of teaching an elective course for electrical engineering undergraduates at the University of Florida using this electronic book. For further information, please consult the paper [Principe et al., 2000]. The course is taught in a laboratory format where we use a computer projector and an electronic white board, and each student has access to a computer during every lecture. The white board allows the text to be interactive, and examples are executed by touching the board. We cover a concept for 15 minutes and then allow the students 15 minutes to run the simulator and explore the example. Then we conclude and move to the next topic. The text naturally provides conceptual boundaries, with each concept illustrated by an example. The function of the instructor in this environment is as a conductor: explaining the concepts, answering questions, and timing the experiments. Students are allowed two hours of self-study a day in the classroom. Office hours are conducted in the classroom. Projects involve searches for data on the Internet.

The simulator is one of the central tools of learning. We slowly cover the most important parts of the simulator in Chapters 1 and 3, at the same time the material in those chapters is covered. However, we recommend that the instructor be familiar with the simulator before the course begins. The simulator’s user interface is iconic which makes learning easy. Icons are organized into families: five neural component families (Synapse, Axon, search, and supervised and unsupervised criteria) and simulation and visualization functions such as Probes, Controllers, Inputs, Transmitters and Schedulers. Each icon has a corresponding Inspector that changes the component parameters. The mechanics are intuitive, with drag-and-drop functionality. Configuration is a bit more detailed, but the user interface mimics the design steps. Therefore the choices are logical if the material is understood. Students can use other data supplied on the CD-ROM, and even modify the topologies given in the examples up to the limits imposed by the software developer.

The simulator is such an important part of the course presentation that we include a tutorial with the book. The instructor should master the simulator since it facilitates the answers to “what if” questions that make learning more interesting and productive. Students should also learn the simulator so that they can run the examples independently, make modifications for better learning, work out the problems at the end of the chapters, and apply newly acquired knowledge to projects.

References:

Principe J., N. Euliano, C. Lefebvre, “Innovating Adaptive and Neural Systems Instruction with Interactive Electronic Books”, Proceedings of the IEEE, special issue on engineering education, January 2000 (in press).

Go to next section


Chapter 1 - Data Fitting with Linear Models

  1. 1 Introduction

  1. 2 Linear Models

  1. 3 Least Squares

  1. 4 Adaptive Linear Systems

  1. 5 Estimation of the Gradient: The LMS Algorithm

  1. 6 A Methodology for Stable Adaptation

  1. 7 Regression for Multiple Variables

  1. 8 Newton's Method

  1. 9 Analytic versus Iterative Solutions

  1. 10 The Linear Regression Model

  1. 11 Conclusions

  1. 12 Exercises

  1. 13 NeuroSolutions Examples

  1. 14 Concept Map for Chapter 1

The goal of this chapter is to introduce the following concepts:

  • Data fitting and the derivation of the best linear (regression) model

  • Iterative solution of the regression model

  • Steepest descent methods

  • The least mean square (LMS) estimator for the gradient

  • The trade-off between speed of adaptation and solution accuracy

  • Examples using NeuroSolutions

Go to next section


Chapter 2 - Multilayer Perceptrons

Chapter 2 - Pattern Recognition

  2. 1 The Pattern-Recognition Problem

2.2 Statistical Formulation of Classifiers

2.3 Linear and Nonlinear Classifier Machines

2.4 Methods of Training Parametric Classifiers

2.5 Conclusions

2.6 Exercises

2.7 NeuroSolutions Examples

2.8 Concept Map for Chapter 2

The goal of this chapter is to provide the basic understanding of:

  • Statistical pattern recognition

  • Training of classifiers

Go to next section


Chapter 3 - Multilayer Perceptrons

  3. 1 Artificial Neural Networks

3.2 Pattern Recognition Ability of the McCulloch-Pitts PE

3.3 The Perceptron

3.4 One Hidden Layer Multilayer Perceptrons

3.5 MLPs with Two Hidden Layers

3.6 Training Static Networks with the Backpropagation Procedure

3.7 Training Embedded Adaptive Systems

3.8 MLPs as Optimal Classifiers

3.9 Conclusions

3.10 NeuroSolutions Examples

3.11 Exercises

3.12 Concept Map for Chapter 3

The goal of this chapter is to provide the basic understanding of the following concepts:

  • Definition of neural networks

  • McCulloch-Pitts PE

  • Perceptron and its separation surfaces

  • Training the perceptron

  • Multilayer perceptron and its separation surfaces

  • Backpropagation

  • Ordered derivatives and computational complexity

  • Data flow implementation of backpropagation

Go to next section


Chapter 4 - Designing and Training MLPs

  4. 1 Introduction

4.2 Controlling Learning in Practice

4.3 Other Search Procedures

4.4 Stop Criteria

4.5 How Good are MLPs as Learning Machines?

4.6 Error Criterion

4.7 Network Size and Generalization

4.8 Project: Application of the MLP to Real World Data

4.9 Conclusions

4.10 NeuroSolutions Examples

4.11 Exercises

4.12 Concept Map for Chapter 4

In this chapter, we address the more practical aspects of using MLPs, which include

  • Search strategies to find the minimum

  • Alternative cost functions

  • Control of generalization (topologies)

After presenting these aspects from a practical point of view, real-world problems will be solved with the MLP topology.

Go to next section


Chapter 5 - Function Approximation with MLPs, Radial Basis Functions, and Support Vector Machines

  5. 1 Introduction

5.2 Function Approximation

5.3 Choices for the Elementary Functions

5.4 Probabilistic Interpretation of the Mappings

5.5 Training Neural Networks for Function Approximation

5.6 How to Select the Number of Bases

5.7 Applications of Radial Basis Functions

5.8 Support Vector Machines

5.9 Project: Applications of Neural Networks as Function Approximators

5.10 Conclusions

5.11 Exercises

5.12 NeuroSolutions Examples

5.13 Concept Map for Chapter 5

The goal of this chapter is to introduce the following concepts:

  • Function approximation as a unifying perspective of adaptive systems

  • Radial basis function (RBF) networks and applications

  • Understanding the issues linking network size and generalization

  • Support vector machines (SVMs)

Go to next section


Chapter 6 - Hebbian Learning and Principal Component Analysis

  6. 1 Introduction

6.2 Effect of the Hebbian Update

6.3 Oja's Rule

6.4 Principal Component Analysis

6.5 Anti-Hebbian Learning

6.6 Estimating Cross-correlation with Hebbian Networks

6.7 Novelty Filters and Lateral Inhibition

6.8 Linear Associative Memories (LAMs)

6.9 LMS Learning as a Combination of Hebbian Rules

6.10 AutoAssociation

6.11 Nonlinear Associative Memories

6.12 Project: Use of Hebbian Networks for Data Compression and Associative Memories

6.13 Conclusions

6.14 Exercises

6.15 NeuroSolutions Examples

6.16 Concept Map for Chapter 6

The goal of this chapter is to introduce the concepts of:

  • Correlation learning

  • Hebbian rule and its modifications

  • Principal component analysis (PCA)

  • PCA networks

  • Associative memories

Go to next section


Chapter 7 - Competitive and Kohonen Networks

  7. 1 Introduction

7.2 Hard Competition; Winner-Take-All Networks

7.3 Competitive Learning

7.4 Clustering

7.5 Improving Competitive Learning

7.6 Soft Competition

7.7 Kohonen Self-Organizing Networks

7.8 Creating Classifiers from Competitive Networks

7.9 Adaptive Resonance Theory (ART)

7.10 Modular Networks

7.11 Conclusions

7.12 Exercises

7.13 NeuroSolutions Examples

7.14 Concept Map for Chapter 7

The goal of this chapter is to introduce the concepts of:

  • Competitive learning

  • Clustering networks

  • Data representation (vector quantization)

  • Kohonen SOM networks

  • Competitive networks and classification

  • Modular networks

Go to next section


Chapter 8 - Principles of Digital Signal Processing

  8. 1 Time Series and Computers

8.2 Vectors and Discrete Signals

8.3 The Concept of Filtering

8.4 Time Domain Analysis of Linear Systems

8.5 Recurrent Systems and Stability

8.6 Frequency Domain Analysis

8.7 Z Transforms and the System Transfer Function

8.8 Linear Systems in the Frequency Domain

8.9 Frequency Response and Poles and Zeros

8.10 Types of Linear Filters

8.11 Project: Design of Digital Filters

8.12 Conclusions

8.13 Exercises

8.14 NeuroSolutions Examples

8.15 Concept Map for Chapter 8

This chapter covers the basic ideas of:

  • Digital representation of continuous signals

  • Filtering as a projection

  • Time and frequency analysis

  • Fourier transforms

  • Filter frequency response

  • Digital filter design

This chapter is not meant to be an introduction to digital signal processing (DSP), but a survey of the field in terms of fundamental concepts and tools. Many excellent books have been written on the subject of DSP subject, and we strongly encourage the reader to consult them for a thorough treatment (Oppenheim and Schaefer, and Haykin and Van Veen).

Go to next section


Chapter 9 - Adaptive Filters

9.1 Introduction

9.2 The Adaptive Linear Combiner and Linear Regression

9.3 Optimum Filter Weights

9.4 Properties of the Iterative Solution

9.5 Hebbian Networks for Time Processing

9.6 Applications of the Adaptive Linear Combiner

9.7 Applications of Temporal PCA Networks

9.8 Conclusions

9.9 Exercises

9.10 NeuroSolutions Examples

9.11 Concept Map for Chapter 9

The goal of this chapter is to introduce the concepts of:

  • Adaptive linear filtering

  • The adaptive linear combiner

  • Temporal PCA networks

  • Many useful applications of adaptive filters

Go to next section


Chapter 10 - Temporal Processing with Neural Networks

10.1 Static versus Dynamic Systems

10.2 Extracting Information in Time

10.3 The Focused Time Delay Neural Network (TDNN)

10.4 The Memory PE

10.5 The Memory Filter

10.6 Design of the Memory Space

10.7 The Gamma Memory PE

10.8 Time Lagged Feedforward Networks (TLFNs)

10.9 Focused TLFNs Built from RBFs

10.10 Project: Iterative Prediction of Chaotic Time Series

10.11 Conclusions

10.12 Exercises

10.13 NeuroSolutions Examples

10.14 Concept Map for Chapter 10

The goal of this chapter is to introduce the concepts of:

  • Time processing with neural networks

  • Time-lagged feedforward networks (TLFNs)

  • The focused topology

  • Memory PE

  • Generalized feedforward memories

Go to next section


Chapter 11 - Training and Using Recurrent Networks

11.1 Introduction

11 2 Simple Recurrent Topologies

11 3 Adapting the Feedback Parameter

11 4 Unfolding Recurrent Networks in Time

11 5 The Distributed TLFN Topology

11 6 Dynamic Systems

11.7 Recurrent Neural Networks

11.8 Learning Paradigms for Recurrent Systems

11.9 Applications of Recurrent Networks to System Identification and Control

11.10 Hopfield Networks

11.11 Grossberg's Additive Model

11.12 Beyond First Order PEs: Freeman's Model

11.13 Conclusions

11.14 Exercises

11.15 NeuroSolutions Examples

11.16 Concept Map for Chapter 11

The goal of this chapter is to introduce the following concepts:

  • Why backpropagation cannot train recurrent systems

  • Algorithm for backpropagation through time

  • Distributed TLFNs

  • Recurrent networks

  • Hopfield's computational energy

  • Grossberg's additive neural model

  • Applications of recurrent neural networks

Go to next section


Appendix A - Elements of Linear Algebra and Pattern Recognition

The goal of this appendix is to present the fundamental concepts of signals as vectors and to present the tools to analyze and transform vectors. This is followed by a brief introduction to random vectors and pattern recognition based on the Bayes' theory.

Go to next section


Appendix B - NeuroSolutions Tutorial

This appendix is designed to describe the operation of NeuroSolutions without assuming any previous knowledge of neural networks. We do, however, assume a familiarity with PC software and windows.

Although we don't assume previous knowledge of neural networks, the operation of NeuroSolutions will make more sense as you learn more about them. We recommend that you read through the tutorial quickly the first time through without focusing on the details. It is very important that you run the simulations while you read the material. You should then periodically revisit the tutorial as you progress through the text.

The outline of this tutorial is as follows:

  • Introduction to NeuroSolutions

  • Introduction to the Interactive Examples

  • The fundamentals of NeuroSolutions

  • Using Probes in NeuroSolutions

  • Providing Input to Your Networks

  • Training a Network

  • Summary

Go to next section


Appendix C - Data Directory

This appendix provides a description of the structure of the data directory and of the data available in the CD-ROM. The data is made available to illustrate different aspects of neurocomputing with the simulator and can be used for projects or further understanding of the material.

Each data set is contained in a different folder with a readme file. The name of the data folder is illustrative of the data it contains (e.g. data folder spiral contains the data set spiral used for classification). The readme file provides a brief description of the data, how it can be used and its source.

The data directory is structured into five categories: data for classification, data for regression, data for signal processing, data for the exercises and data from the NeuroSolutions examples (see figure)

All the data is in ASCII format,organized in columns, and can be readily utilized in the simulator. Desired response files were created to facilitate running the examples. More data will be made available in the Wiley and NeuroDimension websites.

The Table below presents by category the data sets included in the CD-ROM.

Classification | Regression | Signal Processing | Exercises | Examples
Blood testing | Abalone | Bugs | Ch. 2 | Ch.1
Breast cancer | Atmospheric | EMG | Ch. 3 | Ch. 2
Credit screening | Bodyfat | Noise reduction | Ch. 4 | Ch. 3
Echocardio | Lynx | Sunspots | Ch. 9 | Ch. 4
Glass | MPG | SystemID1 |  | Ch. 5
Images |  | SystemID2 |  | Ch. 6
Iris |  | Yalefaces |  | Ch. 7
Mushroom |  | echo |  | Ch. 8
Music |  |  |  | Ch. 9
OCR |  |  |  | Ch. 10
Parity |  |  |  | Ch. 11
Sleepdata |  |  |  |
Sonar |  |  |  |
Spiral |  |  |  |
Synthetic1 |  |  |  |
Synthetic2 |  |  |  |
Vowel |  |  |  |
Wine |  |  |  |
WNBA |  |  |  |


1.1 Introduction

  1. 1 Introduction

The study of neural and adaptive systems is a unique and growing interdisciplinary field that considers adaptive, distributed, and mostly nonlinear systems - three of the ingredient found in biology. We believe that neural and adaptive systems should be considered another tool in the scientist's and engineer's toolbox. They will complement effectively present engineering design principles and help build the preprocessors to interface with the real world and ensure the optimality needed in complex systems. When applied correctly, a neural or adaptive system may considerably outperform other methods.

Neural and adaptive systems are used in many important engineering applications, such as signal enhancement, noise cancellation, classification of input patterns, system identification, prediction, and control. They are used in many commercial products such as modems, image-processing and recognition systems, speech recognition, front-end signal processors and biomedical instrumentation. We expect that the list we will grow exponentially in the near future.

The leading characteristic of neural and adaptive systems is their adaptivity, which brings a totally new system design style (Figure 1-1). Instead of being built a priori from specification, neural and adaptive systems use external data to automatically set their parameters. This means that neural systems are parametric. It also means that they are made "aware" of their output through a performance feedback loop that includes a cost function. The performance feedback is utilized directly to change the parameters through systematic procedures called learning or training rules, so that the system output improves with respect to the desired goal (i.e., the error decreases through training).

Figure 1-1 Adaptive system's design

The system designer has to specify just a few crucial steps in the overall process: he or she has to decide the system topology, choose a performance criterion, and design the adaptive algorithms. In neural systems the parameters are often modified in a selected set of data called the training set and are fixed during operation. The designer thus has to know how to specify the input and desired response data and when to stop the training phase. In adaptive systems the system parameters are continuously adapted during operation with the current data. We are at a very exciting stage in neural and adaptive system development because

  • We now know some powerful topologies that are able to create universal input-output mappings.

  • We also know how to design general adaptive algorithms to extract information from data and adapt the parameters of the mappers.

  • We are also starting to understand the pre-requisites for generalization, that is, how to guarantee that the performance in the training set extends to the data found during system operation.

Therefore we are in a position to design effective adaptive solutions to moderately difficult real- world problems. Because of the practicality derived from these advances, we believe that the time is right to teach adaptive systems in undergraduate engineering and science curricula.

Throughout this textbook we explain the principles that are necessary to make judicious choices about the design options for neural and adaptive systems. The discussion is slanted toward engineering, both in terminology and in perspective. We are very much interested in the engineering model-based approach and in explaining the mathematical principles at work. We center the explanation on concepts from adaptive signal processing, which are rooted in statistics, pattern recognition, and digital signal processing. Moreover, our study is restricted to model building from data.

  1. 1.1 Engineering Design and Adaptive Systems

Engineering is a discipline that builds physical systems from human dreams, reinventing the physical world around us. In this respect it transcends physics, which has the passive role of explaining the world, and also mathematics, which stops at the edge of physical reality. Engineering design is like a gigantic Lego construction, where each piece is a subsystem grounded in its physical or mathematical principles. The role of the engineer is to first develop the blueprint of the dream through specifications and then to look for the pieces that fit the blueprint. Obviously, the pieces cannot be put together at random, since each has its own principles, so it is mandatory that the scientist or the engineer first learn the principles attached to each piece and then specify the interface. Normally, this study is done using the scientific method. When the system is physical, we use the principles of physics, and when it is software, we use the principles of mathematics. As an example, see the development of the phone system.

The scientific method has been highly successful in engineering, but let us evaluate it in broad terms. First, engineering design requires the availability of a model for each subsystem. Second, when the number of pieces increases the interactions among the subsystems increase exponentially. Fundamental research will continue to provide a steady flux of new physical and mathematical principles (provided the present trend of reduced federal funding for fundamental science is reversed), but the exponential growth of interactions required for larger and more sophisticated systems is harder to control. In fact, at this point in time, we simply do not have a clear vision of how to handle complexity in the long term. But there are two more factors that present major challenges: the autonomous interaction of systems with the environment and the optimality of the design. We will discuss these now.

Humans have traditionally mediated the interaction of engineering systems with the external world. After all, we use technology to reduce our physical constraints, so we have traditionally maintained control of the machines we build. Since the invention of the digital computer, there has been a trend to create machines that interact directly with the external world without a human in the loop. This brings the complexity of the external world directly into engineering design. We are not yet totally prepared for this, because our mathematical and physical theories about the external world are mere approximations - very good approximations in some cases, but rather poor in others. This disturbs the order of engineering design and creates performance problems (the worst subsystem tends to limit the performance of the full system). As an example, see the Mars' pathfinder mission.

System optimality is also a rising concern to save resources and augment the performance/price ratio. We might think that optimally designing each subsystem would bring global optimality, but this is not always true. Optimal design of complex systems is thus a difficult problem that must also take into consideration the particular type of system function, meaning that the complexity of the environment is once again present. We can conclude that the current challenges faced in engineering are the complexity of the systems, the need for optimal performance, and the autonomous interaction with the environment that will require some form of intelligence. These are the challenges for engineering in the 21st century and beyond.

Whenever there is a new challenge, we should consider new solutions. Quite often the difficulty of a task is also linked to the particular method we are using to find the solution. Is building machines by specification the only way to proceed?

Let us look at living creatures from an engineering systems perspective. The cell is the optimal factory, building directly from the environment at the fundamental molecular level what it needs to carry out its function. The animals we observe today interact efficiently with the environment (otherwise they would not have survived); they work very close to optimality in terms of resources (otherwise they would have been replaced in their niche by more efficient animals); and they certainly are complex. Biology has, in fact, already conquered some of the challenges we face in building engineering systems, so it is worthwhile to investigate the principles at work.

Biology has found a set of inductive principles that are particularly well tuned to the interaction with a complex and unpredictable environment. These principles are not known explicitly but are being intensively studied in biology, computational neurosciences, statistics, computer science, and engineering. They involve extraction of information from sensor data (feature extraction), efficient learning from data, creation of invariants and representations, and decision making under uncertainty. In a global sense, autonomous agents have to build and fit models to data through their daily experience; they have to store these models, choose which shall be applied in each circumstance, and assess the likelihood of success for a given task. An implicit optimization principle is at play, since the goal is to do the best with the available information and resources.

From a scientific perspective, biology uses adaptation to build optimal system functionality. The anatomical organization of the animal (the wetware) is specified in the long term by the environment (through evolution), and in the short term it is used as a constraint to extract in real time the information that the animal needs to survive. At the nervous-system level, it is well accepted that the interaction with the environment molds the wetware using a learning-from-examples metaphor.

  1. 1.2 Experimental Model Building

The problem of data fitting is one of the oldest in experimental science. The real world tends to be very complex and unpredictable, and the exact mechanisms that generate the data are often unknown. Moreover, when we collect physical variables, the sensors are not ideal (of finite precision, noisy, with constrained bandwidth, etc.), so the measurements do not represent the real phenomena exactly. One of the quests in science is to estimate the underlying data model.

The importance of inferring a model from the data is to apply mathematical reasoning to the problem. The major advantage of a mathematical model is the ability to understand, explain, predict, and control outcomes in the natural system [Casti]. Figure 1-2 illustrates the data-modeling process. The most important advantage of the existence of a formal equivalent model is the ability to predict the natural system's behavior at a future time and to control its outputs by applying appropriate inputs.

Figure 1-2 Natural systems and formal models

In this chapter we address the issues of fitting data with linear models, which is called the linear regression problem. (see Dunteman) Notice that we have not specified what the data is, because it is really immaterial. We are seeking relationships between the values of the external (observable) variables of the natural system in Figure 1-2. This methodology can therefore be applied either to meteorological data, biological data, financial data, marketing data, engineering data, and so on.

  1. 1.3 Data Collection

The data-collection phase must be carefully planned to ensure that

  • Data will be sufficient

  • Data will capture the fundamental principles at work

  • Data is as free as possible from observation noise.

Table 1-1 Regression Data

Table 1-1 presents a data example with two variables (x, d) in tabular form. The measurement x is assumed error free, and d is contaminated by noise. From Table 1-1 very little can be said about the data, except that there is a positive trend between the variables (i.e. when x increases d also increases). Our brain is somehow able to extract much more information from pictures than from numbers, so data should first be plotted before performing data analysis. Plotting the data allows verification, assures the researcher that the data was collected correctly, and provides a "feel" for the relationships that exist in the data (e.g., natural trends).

Go to next section


1.2 Linear Models

  1. 2 Linear Models

From the simple observation of Figure 1-3, it is obvious that the relationship between the two variables x and d is complex, if we assume that no noise is present. However, there is an approximately linear trend in the data. The deviation from a straight line could be produced by noise, and underlying the apparent complexity could be a very simple (possibly linear) relationship between x and d, that is

(1.1)

Figure 1-3 Plot of x versus d

or more specifically,

(1.2)

where [image] i is the instantaneous error that is added to yi (the linearly fitted value), w is the line slope, and b is the d axis intercept (or bias). Assuming a linear relationship between x and d has the appeal of simplicity. The data-fitting problem can be solved by a linear system with only two free parameters, the slope w and the bias b.

Figure 1-4 Linear regression PE

The system of Figure 1-4 is called the linear processing element (PE), or Adaline (for adaptive linear element), and it is very simple. It is built from two multipliers and one adder. The multiplier w scales the input, and the multiplier b is a bias, which can also be thought of as an extra input connected to the value +1.The parameters (b, w) have different functions in the solution. We will be particularly interested in studying the dependence of the solution on the parameter(s) that multiply the input xi.

NEUROSOLUTIONS EXAMPLE 1.1

The Linear Processing Element in NeuroSolutions

The goal of this book is to demonstrate as many concepts as possible through demonstrations and simulations. NeuroSolutions is a very powerful neural network/adaptive system design and simulation package that we will use for the demonstrations. We highly recommend that you read through the NeuroSolutions tutorial in Appendix B. NeuroSolutions constructs adaptive systems in a Lego style, that is component by component. The components are chosen from palettes, selected with the mouse, and dropped into the large window called the breadboard. This object-oriented methodology allows for the simple creation of adaptive systems by simply dragging and dropping components, connecting them, and then adjusting their parameters. Particularly in the early chapters we will automatically create the adaptive systems for you through a set of macros. This will shield you from the details of NeuroSolutions until you have a better grasp of the fundamentals of adaptive systems and the use of NeuroSolutions.

In this first example, we introduce a few simple components. The first component required in any simulation is an input component, which belongs to the Axon family. Its function is to receive data from the computer file system or from signal generators within the package. In this case, we will add a file input component to the input Axon to read in the data from Table 1-1. The linear PE shown in Figure 1-4 can be constructed with a Synapse and a Bias Axon. The Synapse implements a sum of products and the Bias Axon adds the bias. The output of such system is exactly Eq. 1.1. The Controller manages the system and controls the firing of data through the system. Since Table 1-1 has 12 data points, the Controller is configured to send 12 points through the system.

The purpose of this example is to display the output of the linear PE, which is a line, and to modify its location in the space by entering different slope and bias values. To display the input and regression line, we use the Data Storage component (stores 12 samples) and the Scatter Plot component. The Scatter Plot component allows us to plot the input (x axis) versus the system response (y axis). We also add two Matrix Editor boxes to allow you to change the values of the two parameters: the weight (slope) and bias (y-intercept). After changing these parameters, you use the Control Palette to run the network.

NeuroSolutions Control Palette

The Run button (the green triangle) tells the Controller to send the data through the network. The other buttons are not important now, but will be used and explained later. Now run the NeuroSolutions example by clicking on the yellow NeuroSolutions icon below. It will walk you through the creation of the breadboard and allow you to see how the regression line changes as you change the weight (slope) and bias (y intercept).

NeuroSolutions Example

(Reminder, click on the yellow icon to start the example.)

Go to next section


1.3 Least Squares

  1. 3 Least Squares

We face a problem when trying to fit a straight line to the noisy observations of Table 1-1. A single line will fit any two observations (two points define a line), but it is unlikely that all points will fall on exactly the same line. Since no single line will fit every point, a global property of the points is needed to find the best fit. The problem of fitting a line to noisy data can be formulated as follows: What is the best choice of w, b such that the fitted line passes the closest to all the points?

Least squares solves the problem of fitting a line to data by finding the line for which the sum of the square deviations (or residuals) in the d direction (the noisy variable direction) are minimized. The fitted points in the line will be denoted by [image] i = b + wxi. The residuals are defined as [image] i = di - [image] i. The fitted points [image] i can also be interpreted as approximated values of di estimated by a linear model when the input xi is known:

(1.3)

Figure 1-5 Regression line showing the deviations

This linear model will be called the linear regressor. Estimated quantities will be denoted by the tilde (~) throughout the book. The outputs of the linear system of Figure 1-4 are the fitted points, [image] i = yi in Figure 1-5. To pick the line that best fits the data, we need a criterion to determine which linear estimator is the "best." The average sum of square errors J (also called the mean square error (MSE)) is a widely utilized performance criterion given by

(1.4)

where N is the number of observations. To simplify the notation, we sometimes drop the top index in the sum.

NEUROSOLUTIONS EXAMPLE 1.2

Computing the MSE for the Linear PE

To create a simulation that displays the MSE, we have to add a new component to the Breadboard, the L2 Criterion. The L2 Criterion implements the mean square error Eq. 1.4. The L2 Criterion requires two inputs to compute the MSE – the system output and the desired response. We will attach the L2 Criterion to the output of the linear PE (system output) and attach a File Input component to the L2 Criterion to load in the value of the desired response from Table 1. In order to visualize the MSE, we will place a Matrix Viewer probe over the L2 Criterion (cost access point). This Matrix Viewer simply displays the data from the component that it resides over – in this case, the mean square error.

Run the demonstration and try to set the slope and bias to minimize the mean square error. Compute the error by hand according to Eq. 1.4 and see whether it matches the value displayed.

NeuroSolutions Example

Our goal is to minimize J analytically, which according to Gauss can be done by taking its partial derivative with respect to the unknowns and equating the resulting equations to zero:

(1.5)

which yields, after some manipulation, least square derivation

(1.6)

where an overbar represents the variable's mean value; for example, [image].

This procedure to determine the coefficients of the line is called the least square method. If we apply these equations to the data of Table 1-1, we get the regression equation (best line through the data)

(1.7)

The least square computation for a large data set is time-consuming, even with a computer.

NEUROSOLUTIONS EXAMPLE 1.3

Finding the Minimum Error by Trial and Error

Enter these values for the slope and bias by typing them in the respective Edit boxes. Verify that with these values the error is the smallest. Change the values slightly (in either direction) and see that the MSE increases. Enter a negative slope and see how the error increases a lot. For the negative slope, what is the value of the bias that gives the smallest error? Note that when one of the coefficients is wrong, the value of the other for best performance is also wrong, that is, they are coupled.

It is important to explore the NeuroSolutions breadboards. The best way to accomplish this is to open the Inspector associated with each icon. Select a component with the mouse. Then press the right mouse button, and select properties. The Inspector will appear in the screen. The Inspector has fields that allow us to configure the NeuroSolutions components and tell us what settings are being used. For instance, go to the input Axon and open the Inspector. You will see that it has one input, one output and no weights (go to the Soma level to look at the weights). If you do the same in the Synapse, you will see that it also has a single input and output and one weight which happens to be our slope parameter. The Bias Axon has a single input, a single output and a single weight, which is the system's bias.

The large barrel on the input Axon is a probe that collects data. Since the barrel is placed on the Activity point, it is storing the 12 data samples that are injected into the network. This is exactly what gets displayed in the x axis of the Scatter Plot. The y axis values are sent from the L2 Criterion by the small barrel (a Data Transmitter). So the Scatter Plot is effectively displaying the pairs of points (xi,di). Likewise it is also displaying the output of the system in blue, i.e. the pairs of points (xi,yi).

If you want to know what the component is and what it does, just go to the NeuroSolutions control bar, select the arrow icon with the question mark, and click on the component that you want to know about (this is called context-sensitive help).

NeuroSolutions Example

  1. 3.1 Correlation Coefficient

We have found a way to compute the regression equation, but we still do not have a measure of how successfully the regression line represents the relationship between x and d. The size of the mean square error (MSE) can be used to determine which line best fits the data, but it doesn't necessarily reflect whether a line fits the data tightly because the MSE depends on the magnitude of the data samples. For instance, by simply scaling the data, we can change the MSE without changing how well the data is fit by the regression line. The correlation coefficient (r) solves this problem. By definition, the correlation coefficient between two random variables x and d is

(1.8)

The numerator is the covariance of the two variables (see the Appendix A Section 4.11), and the denominator is the product of the corresponding standard deviation (variance).

The correlation coefficient is confined to the range [-1,1]. When r =1 there is a perfect positive linear correlation between x and d, that is, they covary, which means that they vary by the same amount. When r=-1, there is a perfectly linear negative correlation between x and d, that is, they vary in opposite ways (when x increases, y decreases by the same amount). When r =0 there is no correlation between x and d, i.e. the variables are called uncorrelated. Intermediate values describe partial correlations. In our example r =0.88, which means that the fit of the linear model to the data is reasonably good. Notice that the correlation coefficient is a property of the data, as we can see from Eq. 1.8 (it is independent of the model). However, the value r² also represents the amount of variance in the data captured by the optimal linear regression (correlation coefficient and least squares ).

The method of least squares is very powerful. Estimation theory says that the least square estimator is the best linear unbiased estimator (BLUE), since it has no bias and has minimal variance among all possible linear estimators. Least squares can be generalized to higher-order polynomial curves, such as quadratics, cubics, and so on (generalized least squares). In this case, nonlinear regression models are obtained. More coefficients need to be computed, but the methodology still applies. Regression can also be extended to multiple variables as we will do later in the chapter (1.7 Regression for Multiple Variables). The dependent variable d in multiple variable regression is a function of a vector x = [x1,…,xD] [image] , where T means the transpose and D is the number of inputs. In this book, vectors are denoted by bold letters. In the multivariate case, the regression line becomes a hyperplane in the space x1,x2,...xD.

Go to next section


1.4 Adaptive Linear Systems

  1. 4 Adaptive Linear Systems

  1. 4.1 Least Squares as a Search for the Parameters of a Linear System

The purpose of least squares is to find parameters (b, w) that minimize the difference between the system output yi and the desired response di. Regression is effectively computing the optimal parameters of an interpolating system (linear in this case) which predicts the value of d from the value of x.

Figure 1-6 Regression as a linear system design problem

Figure 1-6 shows graphically the operation of adapting the parameters of the linear system. The system output y is always a linear combination of the input x with the bias, so it has to lie on a straight line of equation y = wx+b. Changing b modifies the y intercept, while changing w modifies the slope. Therefore we conclude that the goal of linear regression is to adjust the position of the line to minimize the average square difference between the y values (on the line) and the cloud of points di (i.e. the criterion J).

The key point is to recognize that the error contains information that can be used to optimally place the line. Figure 1-6 shows this by including a subsystem that accepts the error as input and modifies the parameters of the system. Thus, the error [image] i is fed back to the system and indirectly affects the output through a change in the parameters (b, w). Effectively the system is made "aware" of its performance through the error. With the incorporation of the mechanism that automatically modifies the system parameters, a very powerful linear system can be built that will constantly seek optimal parameters. Such systems are called neural and adaptive systems and are the focus of this book.

  1. 4.2 Neural and Adaptive Systems

Before pursuing the study of adaptive systems, it is important to reflect briefly on the implications of neural and adaptive systems in engineering design. System design usually begins with specifications. First, the problem domain is studied and modeled, specifications are established, and then a system is built to meet the specifications. The key point is that the system is built to meet the current specifications and will always use the designed set of parameters, even if the external conditions change.

Here we are proposing a very different system design approach based on adaptation, which has a biological flavor to it. In the beginning the system parameters may be way off, creating a large error. However, through the feedback from the error, the system can change its parameters to decrease the error as much as possible. The system's "experience" with the data designs the best set of parameters. An adaptive system is more complex because it has to not only accomplish the desired task, but also be equipped with a subsystem that adapts its parameters. But notice that even if the data changes in the future, this design methodology will modify the system parameters so that the best possible performance is obtained. Additionally, the same system can be used for multiple problems.

There are basically two ways to adapt the system parameters: supervised learning and unsupervised learning. The method described until now belongs to supervised learning because there is a desired response. Later on in the book we will find other methods that also adapt the system parameters, but using only an internal rule. Since there is no desired response, these methods are called unsupervised. We will concentrate here on supervised learning methods.

The ingredients of supervised adaptive system design are

• A system (linear in this case) with adaptive parameters

• The existence of a desired or target response d

• An optimality criterion (the MSE in this case) to be minimized

• A method (subsystem) to compute the optimal parameters.

The method of least squares finds the optimal parameters (b, w) analytically. Our goal is to find alternate ways of computing the same parameters using a search procedure.

  1. 4.3 Analysis of the Error in the Space of the Parameters: The Performance Surface

Let us analyze the mean square error (J) as we change the parameters of the system (w and b). Without loss of generality, we are going to assume that b =0 (or equivalently, that the mean of x and d have been removed), such that J becomes a function of the single variable w

(1.9)

If w is treated as the variable and all other parameters are held constant, we can immediately see that J is quadratic on w with the coefficient of w² (i.e. xi²) being always positive. In the space of the possible w values, J is a parabola facing upward (J is always positive since it is a sum of squares). The function J(w) is called the performance surface for the regression problem (Figure 1-7). The performance surface is an important tool that helps us visualize how the adaptation of the weights affects the mean square error.

Figure 1-7 The performance surface for the regression problem

NEUROSOLUTIONS EXAMPLE 1.4

Plotting the Performance Surface

The performance surface is just a plot of the error criterion (J) versus the value of the weights, so what we will do during the simulation is to vary the Synapse weight (which corresponds to the slope parameter of the linear regressor) between two appropriate values. We can imagine that the error will be minimum at an intermediate value of the weight and that it will increase for both lower values and higher values.

To modify the Synapse weight incrementally we attach a Linear Scheduler to the Synapse and place the Matrix Viewer on it so we can see how the weight is changing. To visualize the MSE we will bring another Scatter Plot to the L2 Criterion. This will allow us to plot the cost versus weight (performance surface).

Run the example and see how the slope parameter of the linear PE affects the mean square error of the linear regressor. As we are going to see the input and desired signals tremendously affect the shape of the performance surface. But how can we change the shape of the performance curve without touching the data files? Let us substitute the L2 Criterion by the Lp Criterion. Go to Palettes and open the Error Criteria menu. Click on the Lp Criterion and bring the pointer to the Breadboard. Notice that the pointer changed to a stamper. If you left-click on the L2 Criterion component, the L2 is substituted by the Lp component, which computes a cost given by the p norm

By default the norm is p =5. Run the simulation again. What do you see? Does the location of the minimum change appreciably? What about the shape of the performance surface? Do you now better understand the function of the cost criterion?

NeuroSolutions Example

Using the performance surface, we can develop a geometric method for finding the value of w, here denoted by w*, which minimizes the performance criterion. Previously (Eq. 1.5), we computed w* by setting to zero the derivative of J with respect to w.

The gradient of the performance surface is a vector (with the dimension of w) that always points toward the direction of maximum J change and with a magnitude equal to the slope of the tangent of the performance surface (Figure 1-8). If you visualize the performance surface as a hillside, each point on the hill will have a gradient arrow that points in the direction of steepest ascent at that point, with larger magnitudes for steeper slopes. A ball rolling down the hill will always attempt to roll in the direction opposite to the gradient arrow (the steepest descent). The slope at the bottom is zero, so the gradient is also zero (that is the reason the ball stops there).

In our special case the gradient has just one component along the weight axis w, [image] J = [image] wJ given by

(1.10)

(see the gradient definition and construction ). A graphical way to construct [image] wJ at a point w0 is to first find the level curve (curve of constant J value) that passes through the point (also called the contour plot). Then take the tangent to the level curve at w0. The gradient component [image] wJ is always perpendicular to the contour curve at w0, with a magnitude given by the partial derivative of J with respect to the weight w (Eq. 1.10). For one weight (one-dimensional problem), as in Figure 1-8 the construction is simplified and we have to only find the direction of the gradient on the axis.

Figure 1-8 Performance surface and its gradient

At the bottom of the bowl, the gradient is zero, because the parabola has slope zero at the vertex. Thus, for a parabolic performance surface, computing the gradient and equating it to zero finds the value of the coefficients that minimize the cost, just as we did in Eq.1.6. The important observation is that the analytical solution found by the least squares coincides with the minimum of the performance surface. Substituting the value of w* into Eq.1.9, the minimum value of the error (Jmin) can be computed (properties of the performance surface).

NEUROSOLUTIONS EXAMPLE 1.5

Comparison of Performance Curves for Different Data Sets

In this example we provide two sets of input files and two sets of output files. By changing the input data, we find that the minimum error, its location in the weight space (a simple line for this 1D example), and the shape of the performance surface change. On the other hand, if we change the desired signal, only the minimum value of the performance and its location change, but the overall shape remains the same.

NeuroSolutions Example

  1. 4.4 Search of the Performance Surface with Steepest Descent

Since the performance surface is a paraboloid, which has a single minimum, an alternative procedure to find the best value of the coefficient w is to search the performance surface instead of computing the best coefficient analytically by Eq.1.6. The search for the minimum of a function can be done efficiently using a broad class of methods based on gradient information. The gradient has two main advantages for the search:

• The gradient can be computed locally.

• The gradient always points in the direction of maximum change.

If the goal is to reach the minimum, the search must be in the direction opposite to the gradient. Thus, the overall method of searching can be stated in the following way.

Start the search with an arbitrary initial weight w(0), where the iteration number is denoted by the index in parentheses. Then compute the gradient of the performance surface at w(0), and modify the initial weight proportionally to the negative of the gradient at w(0). This changes the operating point to w(1). Then compute the gradient at the new position w(1), and apply the same procedure again, that is,

(1.11)

where [image] is a small constant and [image] J(k) denotes the gradient of the performance surface at the kth iteration. The constant [image] is used to maintain stability in the search by ensuring that the operating point does not move too far along the performance surface. This search procedure is called the steepest descent method. Figure 1-9 illustrates the search procedure.

Figure 1-9 The search using the gradient information

If we trace the path of the weights from iteration to iteration, intuitively we see that if the constant [image] is small, eventually the best value for the coefficient w* will be found. Whenever w >w*, we decrease w, and whenever w <w*, we increase w.

Go to next section


1.5 Estimation of the Gradient The LMS Algorithm

  1. 5 Estimation of the Gradient The LMS Algorithm

An adaptive system can use the gradient to optimize its parameters. The gradient, however, is usually not known explicitly and thus must be estimated. Traditionally, the difference operator is used to estimate the derivative as outlined in Figure 1-8. A good estimate, however, requires many small perturbations to the operating point to obtain a robust estimation through averaging. The method is straightforward but not very practical.

In the late 1960s Widrow proposed an extremely elegant algorithm to estimate the gradient that revolutionized the application of gradient descent procedures. His idea is very simple: Use the instantaneous value as the estimator for the true quantity. For our problem this means to drop the summation in Eq.1.9 and define the gradient estimate at step k as its instantaneous value. Substituting Eq. 1.4 into Eq.1.10, removing the summation, and then taking the derivative with respect to w yields

(1.12)

What Eq. 1.12 tells us is that an instantaneous estimate of the gradient at iteration k is simply the product of the current input to the weight times the current error. The amazing thing is that the gradient can be estimated with one multiplication per weight. This is the gradient estimate that led to the famous least means square (LMS) algorithm (or LMS rule). The estimate will be noisy, however, since the algorithm uses the error from a single sample instead of summing the error for each point in the data set (e.g., the MSE is estimated by the error for the current sample). But remember that the adaptation process does not find the minimum in one step. Normally, many iterations are required to find the minimum of the performance surface, and during this process the noise in the gradient is being averaged (or filtered) out.

If the estimator of Eq.1.12 is substituted in Eq.1.11, the steepest descent equation becomes

(1.13)

This equation is the LMS algorithm. With the LMS rule one does not need to worry about perturbation and averaging to properly estimate the gradient at each iteration, it is the iterative process that is improving the gradient estimator. The small constant [image] is called the step size , or the learning rate.

NEUROSOLUTIONS EXAMPLE 1.6

Adapting the Linear PE with LMS

Several things have to be added to the previous breadboard of the linear PE to make it learn automatically using the LMS algorithm. The methodology will be explained in more detail later. However, the technique used in NeuroSolutions is called back-propagation. In short, the algorithm passes the input data forward through the network and the error information (desired minus output) backward through another network. The error is propagated through a second layer, which can be obtained from the original network with minor and well- established modifications (more about this later). Thus, at every component there is a local activity (x) and a local error ( [image]) such that the weights of the network can be modified by Eq. 1.13. NeuroSolutions implements this technique by adding two additional layers to the network: the back-propagation layer and the gradient-search layer. These two layers can be automatically added to the breadboard. The backpropagation layer looks like a small version of the network that sits on top of the original network (in red instead of orange). The gradient- search layer sits on top of the back-propagation layer and uses one of the gradient search methods to adjust the weights. In this case, the gradient search layer is a simple "step" layer, which implements the gradient descent rule Eq 1.13 . Notice that only the components that have adjustable weights (the Synapse (w) and Bias Axon (b)) have gradient-search components.

In addition to the two layers, we need an additional controller to manage the back-propagation layer. The Backprop Controller sits above the yellow Controller and sets parameters such as whether we use batch or on-line learning. In this example we use batch learning, that is the learning rule will compute all the weight updates for the training set, add them up, and at the end of the epoch (one presentation of all the training data) update the weights according to Eq 1.13. The value of the step size will be set at 0.01, and the training will use 200 iterations.

When you run the network, watch the regression line move toward the optimal value in the Scatter Plot. When the simulation ends, notice that the weight is approximately 0.139, the bias is approximately 1.33, and the error is approximately 0.033 – all in excellent agreement with the optimal values we computed analytically.

You should explore this breadboard by entering several values of the step size and opening the Inspector to see how each component is configured.

NeuroSolutions Example

  1. 5.1 Batch and Sample-by-Sample Learning

The LMS algorithm was presented in a form in which the weight updates are computed for each input sample, and the weights modified after each sample. This procedure is called sample-by-sample learning, or on-line training. As we have mentioned, the estimate of the gradient is going to be noisy, that is the direction towards the minimum is going to zigzag around the gradient direction.

An alternative solution is to compute the weight update for each input sample and store these values (without changing the weights) during one pass through the training set which is called an epoch. At the end of the epoch, all the weight updates are added together, and only then will the weights be updated with the composite value. This method adapts the weights with a cumulative weight update, so it will follow the gradient more closely. It is called the batch training mode, or batch learning. Batch learning is also an implementation of the steepest-descent procedure. In fact, it provides an estimator for the gradient that is smoother than the LMS. We will see that the agreement between the analytical quantities that describe adaptation and the ones obtained experimentally is excellent with the batch update. See batch versus on-line learning for more detail.

To visualize the differences between these two update methods, we will plot the value of the cost during adaptation (called the learning curve).

NEUROSOLUTIONS EXAMPLE 1.7

Batch versus On-Line Adaptation

It is important to visualize the differences in adaptation for on-line and batch learning. Up to now we have been using the batch mode. In this example we set the Backprop Controller to use on-line training. To display the learning curve, we have to introduce one new component: the Megascope. The Megascope is a probe that acts just like an oscilloscope: it plots a continuous stream of inputs, using the iteration number as the x axis. The Megascope sits on top of the Data Storage component.

The important controls of the Megascope are the scales of the x and y axes. In the scope level of the Inspector, we can select the vertical scale (y axis) and the offset of each channel. Alternatively we can use the auto scale feature. The horizontal scale is selected in the sweep level of the Inspector (number of samples per division). Remember that the number of samples displayed is defined by the Data Storage component.

To create the learning curve, we simply place a Data Storage component over the L2 Criterion at one of the cost access points and then place a Megascope on top of it. In batch learning, the Average Cost is selected, while in on-line learning, the Cost access point is utilized.

We will see that the learning curve is not smooth anymore because we are updating the weights after each example. Since the individual errors vary from sample to sample, our updates will make the learning curve noisy. The learning curve will have a periodic component superimposed on a decaying exponential. The exponential tells us that we are approaching a better overall solution. The periodic features show the error obtained for each input sample, while the envelope is related to the learning curve for the batch mode. Note that the weights never stabilize: the performance curve should otherwise be smooth and converge to a single final value. Since there is more noise in on-line learning, we must decrease the step size to get smoother adaptation. We recommend using a step size 10 times smaller than the step size for batch learning. But the price paid is a longer adaptation time; the system needs more iterations to get to a predefined final error. Experiment with the learning rates to observe this behavior.

NeuroSolutions Example

  1. 5.2 Robustness and System Testing

One of the interesting aspects of the LMS solution is its robustness. From the explanation given (Figure 1-9), no matter what the initial condition for the weights, the solution always converges to basically the same value. We can even add some noise to the desired response and find out that the linear regressor parameters are basically unchanged. This robustness is rather important for real- world problems, where noise is omnipresent.

The group of input samples and desired responses (shown in Table 1-1) used to train the system are called collectively the training set for obvious reasons. It is with their information that the system parameters were adapted. But once the optimal parameters are found, the parameters should be fixed. When the system is utilized for new inputs never encountered before, it will produce for each input a response based on the parameters obtained during training. If the new data comes from the same experiment, the response should resemble the value of the desired response for that particular input value.

Thus we see that the system has the ability to extrapolate responses for new data. This is an important feature since in general we wish the performance obtained in the training set to also apply (generalize) to the new data when the system is deployed. But due to the methodology utilized to derive the parameter values we can never be exactly sure of how well the system will respond to new data.

For this reason it is a good methodology to use a test set to verify the system performance before deploying it in the real-world application. The test set consists of new data not used for training but for which we still know the desired response. It is like the final rehearsal before a play's inauguration. We should also compute the correlation coefficient in the test set. Normally, we will find a slight decrease in performance from the training set. If the performance in the test set is not acceptable, we have to go back to the drawing board. When this happens in regression, the most common problem is that the training data is inadequate – either in quantity or in exhaustive coverage of the experimental conditions. This point will be addressed in more depth in the following chapters.

NEUROSOLUTIONS EXAMPLE 1.8

Robustness of LMS to Noise

The LMS algorithm is very robust. It works from any arbitrary location and even works well with noise added to the desired data. To demonstrate that the system works well even with noisy data, we add one additional component to the Breadboard from the previous example: the Noise component. The Noise component allows uniform, Gaussian, or "user-defined" noise to be added to the input or desired signals. We will add the Noise component to the desired signal and watch as the system moves close to the optimal location even with the noisy data.

NeuroSolutions Example

  1. 5.3 Computing the Correlation Coefficient in Adaptive Systems

The correlation coefficient, r, tells how much of the variance of d is captured by a linear regression on the independent variable x. As such, r is a very powerful quantifier of the modeling result. It has a great advantage with respect to the MSE because it is automatically normalized, while the MSE is not. However, the correlation coefficient is blind to differences in means because it is a ratio of variances (see Eq.1.8), that is, as long as the desired data and input covary r will be small, in spite of the fact that they may be far apart in actual value. Thus we need both quantities (r and MSE) when testing the results of regression.

Although the correlation coefficient can be computed directly from x and d (Eq. 1.8), we would like to estimate r at the output of the linear system to follow the adaptive systems' methodology. From Eq. 1B.4, we can write

(1.14)

Note, however, that y changes during adaptation so we should wait until the system adapts to read the final correlation coefficient (i.e., [image]). During adaptation the numerator of Eq. 1.14 can be larger than the denominator, giving a value for r larger than 1, which is meaningless. We therefore propose to compute a new parameter [image] that is a reasonable proxy for the correlation coefficient, even during adaptation. We subtract a term from the numerator of Eq. 1.14 that becomes zero at the optimal setting but limits [image] such that its value is always between -1 and 1 even during adaptation. We can write computation of correlation coefficient

(1.15)

Note that all these quantities can be computed on-line with the information of the error, the output, and the desired response. Remember, however, that Eq. 1.14 measures the correlation coefficient only when the Adaline has been totally adapted to the data.

NEUROSOLUTIONS EXAMPLE 1.9

Estimating the Correlation Coefficient During Learning

NeuroSolutions does not include a component to compute the correlation coefficient. It does, however, allow you to write your own components. These custom components are called DLLs. A custom component looks just like the component it takes the place of, except that its icon has "DLL" printed on it. In this example, we include a custom component to compute the correlation coefficient. This component looks exactly like an L2 Criterion component, except it has "DLL" printed on it.

Plug in the values of the optimal weights and verify that the formula Eq. 1.14 gives the correct correlation coefficient. Slightly modify w to 0.120 and verify that the correlation coefficient decreases. If you plug in values for w and b that are very far away from the fitted regression, this estimation of r using Eq 1.14 becomes less accurate, but is still bound by -1 and 1. The example also uses LMS to adapt the coefficients. Observe that the correlation coefficient is always between -1 and 1 during adaptation and that the final value corresponds to the computed one.

NeuroSolutions Example

Go to next section


1.6 A Methodology for Stable Adaptation

  1. 6 A Methodology for Stable Adaptation

During adaptation the learning algorithm automatically changes the system parameters using Eq 1.13. This adaptation algorithm has one free parameter (the step size) that must be user selected. To appropriately set the step size, the user should have a good understanding of what is happening inside the system. In this section we quantify the adaptation process and develop visualization tools that will help you understand how well the system is learning.

  1. 6.1 Learning Curve

As is readily apparent from Figure 1-9, when the weights approach the optimum value, the values of J(k) (the MSE at iteration k) will also decrease, approaching its minimum value Jmin. One of the best ways to monitor the convergence of the adaptation process is to plot the error at each iteration. The plot of the MSE across iterations is called the learning curve (Figure 1-10). The learning curve is as important for adaptive systems as the thermometer is to check your health. It is an external, scalar, easy-to-compute indication of how well the system is learning. But similar to body temperature, it is unspecific; that is when the system is not learning, it does not tell us why.

Figure 1-10 The learning curve

Notice that the error approaches the minimum in a one-sided manner (i.e., always larger than Jmin). As we can expect, the rate of decrease of the error depends on the value of the step size [image]. Larger step sizes will take fewer iterations to reach the neighborhood of the minimum, provided that the adaptation converges. However, too large a step size creates a divergent iterative process, and the optimal solution is not obtained. It is interesting to note that we would like as large a step size as possible, because this decreases the convergence time. However, if the step size is increased too much, divergence will result. We therefore must seek a way to find the largest possible step size that guarantees convergence.

NEUROSOLUTIONS EXAMPLE 1.10

The Learning Curve

The goal of this example is to display the learning curve and to show how the learning rate affects its shape. This example plots the mean squared error during adaptation, which we called the learning curve, the thermometer of learning.

When you run the simulation, watch how the error moves towards zero as the regression line moves towards the optimal location. You can also change the learning rates and watch the regression line moving more quickly or slowly toward the optimal location, thus causing the learning curve to be steeper or shallower. The visualization of the regression line contains more information about what the system is doing but is very difficult to compute and display in higher dimensions. The learning curve, however, is an external, scalar quantity that can be easily measured with minimal overhead.

NeuroSolutions Example

  1. 6.2 Weight Tracks

An adaptive system modifies its weights in an effort to find the best solution. The plot of the value of a weight over time is called the weight track. Weight tracks are an important and direct measure of the adaptation process. The problem is that normally our system has many weights, and we don't know what their optimal values are. Nevertheless, the dynamics of learning can be inferred and monitored from the weight tracks.

In the gradient-descent adaptation, adjustments to the weights are governed by two quantities Eq. 1.11: the step size [image] and the value of the gradient at the point. Even for a constant step size, the weight adjustments become smaller and smaller as the adaptation approaches w*, since the slope of the quadratic performance surface is decreasing near the bottom of the performance surface. Thus the weights approach their final values asymptotically (Figure 1-11).

Three cases are depicted in Figure 1-11. If the step size is small, the weight converges monotonically to w*, and the number of iterations to reach the bottom of the bowl may be large. If the step size [image] is increased, the convergence will be faster but still monotonic. After [image] reaches a value called critically damped, the weight will approach w* in an oscillatory fashion ( [image] 2> [image] 1), that is, it will overshoot and undershoot the final solution. The number of iterations necessary to reach the neighborhood of w* will start increasing again. If the step size is too large ( [image] 3> [image] 2), the iterative process will diverge, that is instead of getting closer to the minimum, the search will visit points of larger and larger MSE until there is a numeric overflow. We say that the learning diverged.

Figure 1-11 Weight tracks and plots of the weight values across iterations for three values of [image] .

NEUROSOLUTIONS EXAMPLE 1.11

Weight Tracks

It is very instructive to observe the linear PE parameters during learning and how they change as a function of the step size. Let us install a Mega Scope over the Synapse to visualize the slope parameter of the regressor and over the Bias Axon to visualize the regressor bias. These are called weight tracks. Run the simulation and watch how changing the step sizes affects the way the system approaches its final weights.

The weight tracks are a finer display of how adaptation is progressing, but the problem is that in systems with many weights it becomes impractical to observe all the weight tracks. Why do we say that weight tracks give us a better handle on the adaptation parameters? Enter 0.02 for the step size and see the weight tracks converge monotonically to their minimum value. Now enter 0.035. The weight tracks are oscillating toward the final value which means that the system is already in the underdamped regime (but the learning curve is still monotonically decreasing toward the minimum at a faster rate). We can expect divergence if we increase the weights further. Try 0.038 and see it happen. Relate this behavior to Figure 1-11.

NeuroSolutions Example

  1. 6.3 Largest Step Size for Convergence

As we have just discussed, we would like to choose the largest step size possible for fastest convergence without creating an unstable system. Since adjustment to the weights is a product of the step size and the local gradient of the performance surface, it is clear that the largest step size depends on the shape of the performance surface. We already saw (Eq. 1B.10) that the shape of the performance surface is controlled by the input data. Therefore we can conclude that the maximum step size will be dictated by the input data. But how?

If we rewrite and manipulate the equations that produce the weight values in terms of the first weight w(0), Derivation of the Largest Step Size we get

(1.16)

where

The term (1- [image] [image] ) [image] must be less than or equal to 1 to guarantee weight convergence (and less than 1 to guarantee convergence to zero, giving the solution w(k+1)=w*). This implies that

(1.17)

where [image] is the geometric ratio of the iterative process. Hence, the value of the step size [image] must always be smaller than 2/ [image]. The fastest convergence is obtained with the critically damped step size of 1/ [image] . The closer [image] is to 1/ [image] [image] the faster is the convergence, but faster convergence also means that the iterative process is closer to instability. We can visualize this in Figure 1-11. When [image] is increased, a monotonic (overdamped) convergence to w* is substituted by an alternating (underdamped) convergence that finally degenerates into divergence.

There is a slight practical problem that must be solved. During batch learning the weight updates are added together during an epoch to obtain the new weight. This effectively includes a factor of N in the LMS weight update formula Eq. 1.13. To apply the analysis of the largest step size Eq. 1.17 one has to use a normalized step size

(1.18)

With this modification, even if the number of samples in our experiment changes, the step sizes do not need to be modified. Note that for on-line learning (N =1) we get the LMS rule again. We will always use normalized step sizes but to make the notation simpler, we will drop the subscript n in the normalized step size. An added advantage of using normalized step sizes is that we can switch between on-line updates and batch updates without having to change the step size in the simulations.

The analysis of the largest step size Eq. 1.17 also applies in the mean to the LMS algorithm. However, since LMS uses an instantaneous (noisy) estimate of the gradient, even when [image] obeys Eq 1.17, instability may occur. When the iterative process diverges, the algorithm "forgets" its location in the performance surface; that is the values of the weights change drastically. This means that all the iterations up to that point were wasted. Hence with LMS it is common to include a safety factor of 10 in the step size [image] [image] [image] [image] [image] [image] [image] [image] [image] max) or to use batch training, which reduces the noise in the estimate of the gradient.

  1. 6.4 Time Constant of Adaptation

An alternative view of the adaptive process is to quantify the convergence of w(k) to w* in terms of an exponential decrease. We know that w(k) converges to w* as a geometric Eq.1.16. The envelope of the geometric progression of weight values can be approximated by an exponential decay exp(-t/ [image]), where [image] is the time constant of weight adaptation. A single iteration or epoch can be considered a unit of time. One may want to know approximately how many iterations are needed until the weights converge. The time constant of weight adaptation can be written as

(1.19)

which clearly shows that fast adaptation (small time constant [image]) requires large step sizes. (see derivation of the time constant of adaptation). For all practical purposes the iterative process converges after four time constants.

The steps used to deriv1 the time constant of weight adaptation can be applied also to come up with a closed-form solution to the decrease of the cost across iterations, which is called the time constant of adaptation. Eq. 1.16 tells us how the weights converge to w*. If the equation for the weight recursion is substituted in the equation for the cost (Eq.1B.11) we get

which means that J also approximates Jmin in a geometric progression, with a ratio equal to [image] ². Therefore the time constant of adaptation is

Since the geometric ratio is always positive, J approximates Jmin monotonically (i.e., an exponential decrease). The time constant of adaptation describes the learning time (in number of iterations) needed to adapt the system. Notice that these expressions assume that the adaptation follows the gradient. With the instantaneous estimate used in the LMS, J may oscillate during adaptation since the estimate is noisy. But even in the LMS, J will approach Jmin in a one-sided way (i.e. always greater than or equal to Jmin).

NEUROSOLUTIONS EXAMPLE 1.12

Linear Regression Without Bias

The previous example solved the linear regression problem with one weight and one bias. To compare the equations previously given (which are a function of a single parameter) with the simulations, we have to make a modification in the data set or in the simulation. Shortly we will see how to extend the analysis for multiple weights, but for the time being let us work with the simpler case.

We substitute the Bias Axon by an Axon, a component that simply adds its inputs, so the regression solution becomes y =wx which has to pass through the origin. With this new Breadboard we can compare the numerical results of the simulations directly with all the equations derived in this section, since there is only one free parameter. Batch updates are used throughout.

The optimal value of the slope parameter is computed by Eq. 1.6, which gives w =0.30009, with an average error of 0.23. This solution is different from the value obtained previously (w = 0.139511) for the bias regressor because the regression line is now constrained to pass through the origin. It turns out that this constrained solution is worse than before, as we can see by the error (0.23 versus 0.033). Observing in the scatter plot the output (red points) and the input samples (blue) shows clearly what we are describing.

Computing [image] by Eq. 1.16 yields 54, so according to Eq. 1.17 the maximum step size is [image] [image] =3.6e-2. The critically damped solution is obtained with a step size of 1.8e-2, and adaptation with a step size below this value is overdamped. When we run the simulator in the overdamped case the weights approach the final value monotonically; for the critically damped case, they stabilize quite rapidly; while for the underdamped case they oscillate around the final value, and the convergence takes more iterations. Notice also that the linear regressor "vibrates" around the final position, since the slope parameter is overshooting and undershooting the optimal value.

According to Eq. 1.19 for the critically damped step size [image] [image] =1, so the solution should stabilize in four updates (epochs). This step size yields the fastest convergence. Go to the Controller Inspector and use the epoch button to verify the number of samples until convergence.

NeuroSolutions Example

  1. 6.5 Rattling

Up to now our main focus was the speed of adaptation, that is, how fast the weights approximate w*, or equivalently, how fast J approximates Jmin. Unfortunately, this is only part of the story. For fast convergence we need large step sizes ( [image]). But when the search is close to the minimum w*, where the gradient is small but not zero, the iterative process continues to wander around a neighborhood of the minimum solution without ever stabilizing. This phenomenon is called rattling (Figure 1-12), and the rattling basin increases proportionally to the step size [image]. This means that when the adaptive process is stopped by an external command (such as the number of iterations through the data), the weights may not be exactly at w*. We know they are in the neighborhood of the minimum but not exactly at the optimum.

Figure 1-12 Rattling of the iteration procedure

If we picture the performance surface (Figure 1-12), when the final weights are not at w*, there will be a penalty in performance, that is the final MSE will be higher than Jmin. In the theory of adaptation the difference between the final MSE and the Jmin (normalized by Jmin) is called the misadjustment M.

(1.20)

This means that in search procedures that use gradient descent there is an intrinsic compromise between accuracy of the final solution ( a small misadjustment) and speed of convergence. The parameter that controls this compromise is the step size [image] . High [image] means fast convergence but also large misadjustment, while small [image] means slow convergence but little misadjustment.

NEUROSOLUTIONS EXAMPLE 1.13

Rattling

We observed in Example 1.7 how noisy the learning curve became with the on-line update. This is an external indication that the weights were changing from sample to sample even after the system reached the neighborhood of the optimum. The implication of this random movement in the weights is a penalty in the final MSE. In this example we show and exactly quantify the rattling.

The rattling has important consequences for adaptation, since if we set the step size large for fast convergence, we pay a price of inaccurate coefficients, which is translated into an excess MSE. The rule of thumb for LMS is to use a step size that is 1/10 of the largest possible step size. For a step size close to the largest possible, the MSE for the epoch is effectively smaller than the theoretical minimum, which is impossible. This happens because the parameters are changing so much with each update that the slope is continuously varying. The problem is that when we stop the training we do not know if the final value of the weight is a good approximation to the theoretical regression line.

This shows that for adaptive systems the final MSE is only part of the story. We have to make sure that the system coefficients have stabilized. It is interesting to note that with batch updates there is no rattling, so in the linear case the batch solution is more appropriate. Observe this in the simulations by displaying the MSE for large and small step sizes. We are just paying the small price of storing the individual weight updates. For nonlinear systems the batch is unfortunately no longer always superior to the on-line update as we will see.

NeuroSolutions Example

This example shows that obtaining a small MSE is a necessary but not sufficient condition for stable adaptation. Adaptation also requires that the weights of the model settle to stable values. This second condition is required because the system can be endlessly changed its parameters to fit the present sample. This will always give a small MSE, but from a modeling point of view it is a useless solution because no single model is found to fit the data set.

  1. 6.6 Scheduling the Step Sizes

As we saw in the latest examples, for fast convergence to the neighborhood of the minimum a large step size is desired. However, the solution with a large step size suffers from rattling. One attractive solution is to use a large learning rate in the beginning of training to move quickly toward the location of the optimal weights, but then the learning rate should be decreased to obtain good accuracy on the final weight values. This is called learning rate scheduling. This simple idea can be implemented with a variable step size controlled by

(1.21)

where [image] (0) = [image] 0 is the initial step size, and [image] is a small constant. Note that the step size is being linearly decreased at each iteration. If we have control of the number of iterations we can start with a large step size and decrease it to practically zero toward the end of training. The value of [image] needs to be experimentally determined. Alternatively, we can decrease the step size slowly (in optimization this slow decrease is called annealing) using either a linear, geometric, or logarithmic rule (see more on scheduling step sizes).

NEUROSOLUTIONS EXAMPLE 1.14

Scheduling of Step Sizes

In this demonstration we vary the step size using the scheduling component already introduced in Example 1.4. The scheduler is a component that takes an initial value from the component it is attached to, and changes the value according to a predetermined rule. Here we use the linear rule, and since we want to decrease the step size, the factor [image] is negative. We should set a maximum and a minimum value just to make sure that the parameters are always within the range we want. should be set according to the number of iterations and the initial and final values ( [image] init - [image] N= [image] residual).

Here the important parameter is the minimum (the residual step size is set at 0.001), because after scheduling we want to let the system fine-tune its parameters to find the best approximation of the minimum. However, notice that this implies that the parameter is already in its optimal neighborhood, and this depends on a lot of unknown factors. Thus, if the scheduling is not right, the adaptation may stall in positions far from the minimum.

You should explore the breadboard by entering other values for [image] and the final value and see their impact on the final weight value. You can also use the exponential or the logarithmic schedulers and see how they behave. Which one do you prefer for this case?

NeuroSolutions Example

Go to next section


1.7 Regression for Multiple Variables

  1. 7 Regression for Multiple Variables

Assume that d is now a function of several inputs x1, x2,...xD (independent variables), and the goal is to find the best linear regressor of d on all the inputs (Figure 1-13). For D =2 this corresponds to fitting a plane through the N input samples or a hyperplane in the general case of D dimensions.

Figure 1-13 Fitting a regression plane to a set of samples in 2D space.

As an example, let us assume that we have two variables x1 (speed) and x2 (feed rate) that affect the surface roughness (d) of a machined workpiece. In abstract units the values of x1, x2, d for 15 workpieces are presented in Table1-2.

Table 1-2 Multiple Regression Data

The goal is to find how well one can explain the quality of machining by the two variables x1 and x2, and which is the most important parameter.

As before, we assume that the measurements x are noise free and that d is contaminated by a noise vector [image] [image] with these properties: Gaussian distributed with components that are zero mean, of equal variance [image] ², and uncorrelated with the inputs. The regression equation when D =2 is now

(1.22)

where xi1 is the ith value of x1 (the ith workpiece in the training set). In the general case we write Eq. 1.22 as

(1.23)

where we made w0 =b and xi0 =1 (compare with Eq. 1.3). The goal of the regression problem is to find the coefficients w0, ….wD. To simplify the notation we will put all these values into a vector w = [w0, ….wD ] [image] that minimizes the MSE of [image] i over the N samples. Figure 1-14 shows that the linear PE now has D inputs and one bias.

Figure 1-14 Regression system for multiple inputs

The mean square error (MSE) becomes for this case

(1.24)

The solution to the extreme (minimum) of this equation can be found in exactly the same way as before, that is, by taking the derivatives of J with respect to the unknowns (wk), and equating the result to zero (derivation of normal equations ).

This yields a set of D+1 equations in D+1 unknowns called the famous normal equations

(1.25)

The normal equations can be written much more compactly with matrix notation (see the Appendix A section 3) Let us define

(1.26)

as the autocorrelation of the input samples for indices k and j. As you can see, the autocorrelation measures similarity across the samples of the training set. When k =j, R is just the sum of the squares of the input samples (the variance if the data is zero mean). When k differs from j, R measures the sum of the cross products for every possible combination of the indices. As we did for w, we will also put all these Rkj values into a matrix R, that is,

.

Thus we obtain pairwise information about the structure of the data set.

Let us call

(1.27)

the cross-correlation of the input x for index j and desired response d, which can also be put into a vector p of dimension D+1. Pj measures the similarity between the input x and the desired response d at shift j. Substituting these definitions in Eq. 1.25, the set of normal equations can be written simply

(1.28)

where w is a vector with the D+1 weights wi. w* represents the value of the D+1 weights for the optimal (minimum) solution. denotes the inverse of the autocorrelation matrix (see Appendix A Section 3.11). Eq. 1.28 states that the solution of the multiple regression problem can be computed analytically as the product of the inverse of the autocorrelation matrix of the input samples and the cross-correlation vector between the input and the desired response. The least square solution for this problem yields

It is remarkable that we are able to write an equation that describes the relationship between the two variables when only measured data samples were given. This attests the power of linear regression. But as for the single variable case, we still do not know how accurately the equation fits the data, that is, how much of the variance of the input is actually captured by the regression model. The multiple correlation coefficient rm can also be defined in the multiple dimensional case for a single output, as multiple variable correlation coefficient

(1.29)

and measures the amount of variation explained by the linear regression, normalized by the variance of d. In this expression d is the vector built from the desired responses di, and U is a matrix whose columns are the input data vectors. For this case rm=0.68, so there is a large portion of the variability that is not explained by the linear regression. (Either the process is nonlinear, or there are more variables involved.) We can still approximate the correlation coefficient for the multiple regression case by Eq. 1.14 after the system has adapted.

NEUROSOLUTIONS EXAMPLE 1.15

Multivariable Regression

Moving to multi-dimensional inputs is very simple in NeuroSolutions. You simply change the input and desired files (for the new input data) and change the input Axon to accept two inputs. The rest is automatic. In this example we do all this for you using macros. Note that in the two-dimensional case the regression line is now a regression plane. In NeuroSolutions, there is not currently a good way of showing a plane in three dimensions, so we will not plot our regression plane plot. When we run the network, we will see that the learning curve (one of our only indications of whether the network is training correctly) decreases steadily and that the weights eventually approach the theoretical optimal weights.

The amazing thing about the adaptive system's methodology is that we changed the problem, but the technique to solve it did not change significantly. It is true that we have to dimension the system properly and choose new values for the step size, but the fundamental aspects of the methodology did not change at all.

NeuroSolutions Example

  1. 7.1 Setting the Problem as a Search Procedure

All the concepts previously mentioned for linear regression can be extended to the multiple regression case. The performance surface concept can be extended to multidimensions, making J a paraboloid in D+1 dimensions, facing upwards (Figure 1-15 depicts the two weight case). J now involves matrix computations, but it remains a scalar quantity that is a quadratic function of the weights

(1.30)

where the superscript T means the transpose.

Figure 1-15 The performance surface for two dimensions and its contour plot

The coefficients that minimize the solution are

(1.31)

which gives exactly the same solution as Eq.1.28 (see derivation of the optimal solution). In the space (w1,w2), J is a paraboloid facing upwards. See more performance surface properties.

As in the one dimensional case, the autocorrelation of the input (R) completely specifies the shape of the performance surface Eq.1B.19. However, the location of the performance surface in the space of the weights Eq.1.31 and its minimum value Eq.1B.18 depend also on the desired response.

NEUROSOLUTIONS EXAMPLE 1.16

Checking the LMS Solution with the Optimal Weights

Let us first consider a least square solution with only two weights w1, and w2, since we can still compute it easily by hand. For the data set of Table 1-2, the autocorrelation matrix is Eq. 1.26

To determine the eigenvalues, we solve the equation

which yields [image] 1 =59 and [image] 2 =1.5. From these results we can immediately see that the eigenvalue spread is roughly 40, so the performance surface paraboloid is very skewed (i.e., much narrower in one direction). The performance surface is shown in the following figure. Notice how it is very steep in one direction and very shallow in the other. Thus, if we train the network with gradient descent, we would expect it to move very quickly down the steep slope at first and then to move slowly down the valley toward the optimum.

To compute the optimal solution, we first need to compute the cross-correlation vector Eq. 1.27 as

For the two-dimensional case it is still easy to solve for w1 and w2, by writing from Eq. 1.28

which gives for optimal weights w1 =0.2848 and w2 =0.2180. The minimum J is 0.390 from Eq.1B.18. When we run the simulator with the Axon substituting the Bias Axon (no bias), the network weights approach these values.

NeuroSolutions Example

  1. 7.2 Steepest Descent for Multiple Weights

Gradient techniques can also be used to find the minimum of the performance surface, but now the gradient is a vector with D+1 components

(1.32)

The extension of Eq. 1.11 is

(1.33)

where all quantities are vectors, i.e. w(k) = [w0(k), …wD(k)] [image]. In order to calculate the largest step size [image], we again rewrite the update equation in the form of

(1.34)

where I is the identity matrix, R is the input autocorrelation matrix and [image]. The solution of this equation is cross coupled, that is, the way w converges to w* depends on the behavior of the geometric progression in all the D+1 directions. Therefore, the simple picture of having w(k+1) converge to w* with a single geometric ratio, as in the unidimensional case, has to be modified. One can show that the weights converge with different time constants, each related to an eigenvalue of R ( convergence for multiple weights case ).

  1. 7.3 Step Size Control

As we have seen, the set of values taken by the weight during adaptation is called the weight track. The weight moves in the opposite direction of the gradient at each point, so the weight track depicts the gradient direction at each point of the performance surface visited during adaptation. Therefore the gradient direction tells us about the performance surface shape. In particular, it is useful to construct the contour plot of J since the gradient has to be perpendicular to the lines that link points with the same J value. The contour plot provides a graphical construction for the gradient at each point of the performance surface.

Given a point in a contour, we take the tangent of the contour at the point. The gradient is perpendicular to the tangent, so the weights will move along the gradient line and point in the opposite direction. Likewise, if we run the adaptation algorithm with several initial conditions and we record the value of J at each point, we can determine the contour plots by taking ellipses that pass through the points of equal cost and are perpendicular to the weight tracks.

When the eigenvalues of R are the same (see Appendix A section 3.17), the contour plots are circular and the gradient always points to the center, that is to the minimum. In this case the gradient descent has only a single time constant as in the one-dimensional case. But this is an exceptional condition. In general, the eigenvalues of R will be different. When the eigenvalues are different, the weight track bends because it follows the direction of the gradient at each point, which is perpendicular to the contours (Figure 1-17). The gradient direction no longer points to the minimum, which means that the weight tracks will not be straight lines to the minimum. The adaptation will take longer for two reasons. First, a longer path to the minimum will be taken. Second, the step size must be decreased compared with the circular case. Let us address the step size aspect further.

Figure 1-17 Weight track toward the minimum: (a) equal eigenvalues; (b) unequal eigenvalues

For guaranteed convergence the learning rate in each ith principal direction of the performance surface must be

(1.35)

where [image] i is the corresponding eigenvalue. The worst case condition to guarantee convergence to the optimal w* in all directions is therefore

(1.36)

That is, the step size [image] must be smaller than the inverse of the largest eigenvalue of the autocorrelation matrix. Otherwise the iteration will diverge in one (or more) directions. Since the adaptation is coupled, divergence in one direction will cause the entire system to diverge.

In the early stages of adaptation the convergence is primarily along the direction of the largest eigenvalue, since the weight update along this direction will be bigger. On the other hand, toward the end of adaptation the algorithm will mainly adapt only the weight associated with the smallest eigenvalue (which corresponds to the smallest time constant). The time constant of adaptation is therefore

(1.37)

An implication of this analysis is that when the eigenvalue spread of R is large, there will be very different time constants of adaptation in each direction. This reasoning gives a clear picture of the fundamental constraint of adapting the weights using gradient descent with a single step size [image]: The speed of adaptation is controlled by the smallest eigenvalue, while the largest step size is constrained by the inverse of the largest eigenvalue. This means that if the ratio between the largest and the smallest eigenvalues (the eigenvalue spread) is large, the convergence will be intrinsically slow. This problem cannot be avoided when only a single step size is used in the steepest descent.

The learning curve will approach Jmin in a geometric progression as before. However, there will be many different time constants of adaptation, one for each direction. Initially the learning curve will decrease at the rate of the largest eigenvalue, but toward the end of adaptation the rate of decrease of J is controlled by the time constant of the smallest eigenvalue (estimation of eigenvalue spread).

NEUROSOLUTIONS EXAMPLE 1.17

Visualizing the Weight Tracks and Speed of Adaptation

According to our previous calculations, the largest step size for convergence is Eq. 1.36 3.3e-2. The critically damped mode along the largest eigenvector should be 1.6e-2. The time constant of adaptation for the largest step size is around 20 iterations (epochs for batch), that is, the convergence should take 80 epochs with this step size.

When we run the simulations, the algorithm converges first along the direction of the largest eigenvalue (largest eigenvector direction), and then along the direction of the smallest eigenvector. Since the eigenvalue spread is 40, the steps are much bigger along the largest eigenvector direction. If we look at the figure of Example 1.16, we can see that the weights converge perpendicular to the contour plots since this is the steepest descent path. As we will see, there are two distinct regions in the learning curve: In the beginning it is controlled by the geometric ratio along the largest eigenvector, while toward the end it is controlled by the geometric ratio of the smallest eigenvector.

After running this example and observing the weight tracks let us change the input data file to a data set with a smaller eigenvalue spread. Click the input file icon and bring up its Inspector by clicking the right mouse button. Remove the present input file, and add the file regression2a.asc from the \data\chapters\ch1 folder on the CD-ROM.

The modification was made only in the variable x2, all the rest is the same. Respond to the panel associate by clicking on the close button. In the customize panel, skip the desired signal, and click on close. You have just modified the input data to this example. This new file has a much smaller eigenvalue spread, so we can expect that the weight tracks are basically straight lines to the minimum. Compute the new eigenvalue spread, and adjust the learning rates so that the convergence is as fast as possible.

NeuroSolutions Example

  1. 7.4 The LMS Algorithm for Multiple Weights

It is straightforward to extend the gradient estimation given by the LMS algorithm from one dimension to many dimensions. We just apply the instantaneous gradient estimate Eq.1.12 to each element of Eq.1.33. The LMS for multiple dimensions reads

(1.38)

An interesting feature is that the LMS adaptation rule still uses local computations, i.e. for the i'th weight we can write

(1.39)

Note that although the analysis of the gradient descent techniques became complex, the LMS algorithm itself is still very simple. This is one reason why LMS is so widely used. But since LMS is a steepest-descent algorithm, the analysis and discussions concerning the largest step size for convergence and coupling of modes also apply to the LMS algorithm.

NEUROSOLUTIONS EXAMPLE 1.18

Visualizing Weight Tracks with On-Line Learning

In this example we configure the Backprop Controller to on-line learning to implement the LMS algorithm. Notice that the weight tracks follow basically the same path as before, but now the path is much more irregular due to the sample-by-sample update of the weights. When the eigenvalue spread is very large (the performance surface is very steep in one direction and shallow in others), the problem is difficult for LMS to solve. Any small perturbation in the smallest eigenvector direction gets amplified by the large eigenvalue spread.

NeuroSolutions Example

  1. 7.5 Multiple Regression with Bias

Up to now we have implemented and solved analytically the multiple regression problem without bias. The reason for this is only for simplicity. With two weights we can still easily solve the multiple regression case by hand; however, if the bias is added, we must do the computations with three parameters. The simulations are transparent to these difficulties, since we just substitute the Axon by a Bias Axon. Note that the largest step size between the two cases will differ since the input data was effectively changed if one interprets the bias as a weight connected to an extra constant input of one. Hence the autocorrelation function and its eigenvalue spread changed.

We should state that the use of a bias is called the full least square solution, and it is the recommended way to apply least squares. The reason can be understood easily: When a bias is utilized in the PE, the regression line is not restricted to pass through the origin of the space, and smaller errors are normally achieved. There are two equivalent ways to solve the full least squares problem for D input variables:

  • The input and desired responses need to be modified such that they become zero mean variables (this is called the deviation or z scores). In this case a D weight regression will effectively solve the original problem. The bias b is computed indirectly by

where wi are the optimal weights and the bars represent mean values.

  • Alternatively, the input matrix has to be extended with an extra column of 1s (the first column). This transforms R into a (D+1)x(D+1) matrix, which introduces a D+1 weight in the solution (the bias).

NEUROSOLUTIONS EXAMPLE 1.19

Linear Regression without Bias

We will now substitute a Bias Axon for the Axon in the previous breadboard. This will effectively provide the regression solution without constraining the regression plane to pass through the origin. We see that the weight tracks are very similar in the beginning but that the error continues to drop, and the weights advance towards the w1=0 line. This means that the optimal solution changed. We now have a better solution than before but with increased complexity of the performance surface (four dimensional instead of three) and an increased number of adjustable parameters in our system (two weights and a bias).

NeuroSolutions Example

  1. 7.6 The LMS Algorithm in Practice

We can use some rules of thumb to choose the step size in the LMS algorithm. The step size should be normalized by the variance of the input data estimated by the trace of R.

(1.40)

where [image] 0 = 0.1 to 0.01. This normalization by the input variance was the original rule proposed by Widrow to adapt the Adaline. We can expect the algorithm to converge (settling time) in a number of iterations k given by four times the time constant of adaptation:

(1.41)

The LMS algorithm has a misadjustment that is basically one half the trace of R times

(1.42)

When the eigenvalues are equal, the misadjustment can be approximated by

which allows us to give the following rule of thumb: The misadjustment equals the number of weights divided by the settling time, or equivalently, selecting [image] so that it produces 10% misadjustment means a training duration in iterations of 10 times the number of inputs.

Go to next section


1.8 Newton's Method

  1. 8 Newton's Method

If you are familiar with numerical analysis, you may be asking why we aren't using Newton's method for the search. Newton's method is known to find the roots of quadratic equations in one iteration. The minimum of the performance surface can be equated to the root of the gradient equation Eq.1.32, as is outlined by Eq.1.31. Hence, Newton's method can also be used in search. The adaptive weight equation using the Newton's method is Newton's Derivation

(1.43)

Compare this with Eq.1.33 and note that the gradient information is weighted by the inverse of the correlation matrix of the input and that [image] is equal to 1. This means that Newton's method corrects the direction of the search such that it always points to the minimum, while the gradient descent points to the maximum direction of change. These two directions may or may not coincide (Figure 1-18).

Figure 1-18 Directions of the steepest descent and Newton's method

They coincide when the contour plots are circles, that is, when the largest and the smallest eigenvalue of the correlation matrix are the same. When the ratio of the largest to the smallest eigenvalue (the eigenvalue spread) increases, the slope of the performance surface in the two directions differs more and more. Thus, for large eigenvalue spreads the optimization path taken by gradient descent is normally much longer than the path taken by Newton's method. This implies that Newton's method will be faster than LMS when the input data correlation matrix has a large eigenvalue spread.

Another advantage of Newton's method versus the steepest descent method is in the time constant of adaptation. When the gradient is multiplied by R [image] not only the direction of the gradient is being changed but also the different eigenvalues in each direction are being equalized. What this means is that Newton's method is automatically correcting the time constant of adaptation for each direction such that all the weights converge at the same rate. Hence Newton's method has a single time constant of adaptation, unlike the steepest descent method.

These advantages of the Newton's method should not come as a surprise, because Newton's method uses much more information about the performance surface (the curvature). In fact, to implement Newton's method we need to compute the inverse of the correlation matrix, which takes significantly longer than the single multiplication required by the LMS method and also requires global information. Newton's method is also brittle, that is, if the surface is not exactly quadratic, the method may diverge. This is the reason Newton's method is normally modified to also include a small step size [image] instead of using [image] =1 as in Eq.1.43.

(1.44)

Note that x(k) is a vector and R [image] is a matrix, so the update for one weight influences all the other inputs in the system. This is the reason that the computations are no longer local to each weight. However, they are not difficult if one assumes that the inverse of R is known a priori. The algorithm or Eq. 1.44 is called the LMS/Newton algorithm. The case where R [image] has to be estimated on-line is much more involved and leads to the recursive least squares (RLS) algorithm.

Alternatively, to improve convergence speed with the LMS, we can implement an orthogonalizing transformation of the input correlation function followed by an equalization of the eigenvalues which is called a whitening transformation. (see Appendix A section 3.18). Since Newton's method coincides with the steepest descent for performance surfaces that are symmetric, this preprocessing will make the LMS perform as Newton's method.

NEUROSOLUTIONS EXAMPLE 1.20

Newton's Method

In this example, we implement Newton's method with a custom DLL. For this example, we must compute [image] and apply Eq. 1.44 to the simulator. The autocorrelation function for this example is

so R [image] becomes (see the Appendix, section 3.11)

By applying Newton's method to the learning algorithm, we have essentially compensated for the eigenvalue spread. This means that Newton's method behaves as the steepest descent for a circular performance surface where the steepest descent direction always points directly to the optimal value. Thus, although the calculations are more complicated and more demanding (we need to know [image]), the convergence is much faster (in fact, the algorithm can converge in one epoch). When we run the simulator, notice that no matter where the algorithm starts, it always heads directly toward the optimum.

NeuroSolutions Example

Go to next section


1.9 Analytic versus Iterative Solutions

  1. 9 Analytic versus Iterative Solutions

Selecting a search procedure to find the optimal weights is a drastic conceptual change from the analytic least square solution, albeit an equivalent procedure. In learning systems the iterative solution is the most common for several reasons.

When working with learning systems, the interest is very often in on-line solutions, that is, solutions that can be implemented sample-by-sample. The analytic solution requires data to be available beforehand to compute the autocorrelation matrix R and cross-correlation vector p. Fast computers are required to crank out the solution (the inverse of R and product with p). The method produces a value that immediately gives the best possible performance. But several problems may surface when applying the analytic approach. If the matrix R is ill-conditioned, the computation of [image] may not be very accurate. Moreover, the analytic solution also requires a great deal of computation time (Computation of a matrix inverse is proportional to the square of the number of columns D of the matrix. In the big O notation this means O(D²)).

The iterative solution is not free from shortcomings. We already saw that there is no guarantee that the solution is close to the optimal weight w* when all the input samples are used by the algorithm. This depends on the data and on a judicious selection of the step size [image]. The accuracy of the iterative solution is not directly dependent on the condition number of R, but matrices with large eigenvalue spread produce slow convergence because the gradient descent adaptation is coupled. As we said previously, the slowest mode controls the speed of adaptation, while the largest step size is constrained by the largest eigenvalue.

The great appeal of the iterative approach to optimization is that very efficient algorithms exist to estimate the gradient (e.g., the LMS algorithm). Only two multiplications per weight are necessary, so the computation scales proportional to the number of weights D (i.e., O(D) time). Moreover, the method can be readily extended to nonlinear systems, while the analytic approach for most cases of practical relevance cannot be computed.

Go to next section


 2.1 The Pattern Recognition Problem

The human ability to find patterns in the external world is ubiquitous. It is at the core of our ability to respond in a more systematic and reliable manner to external stimuli. Humans do it effortlessly, but the mathematics underlying the analysis and design of pattern-recognition machines are still in their infancy. In the 1930's R.A. Fisher laid out the mathematical principles of statistical pattern recognition, which is one of the most rigorous ways to formulate the problem.

A real-world example will elucidate the principles of statistical pattern recognition at work: Assume that body temperature is used as an indicator of the health of a patient. Experience shows that in the healthy state the body regulates its temperature near 37° Celsius (98.6° F; the low end of normality will not be considered for the sake of simplicity). With viral or bacterial infections the body temperature rises. Any measurement can be thought of as a point in a space called the pattern space, or the input space (one dimensional in our example). So if we plot the temperature of individuals on a line (Figure 2-1), we can verify that the region close to 37°C is assigned to healthy individuals and the higher-temperature region is assigned to sick individuals. This natural distribution of points leads to the definition of classes or category regions in pattern space. The goal of pattern recognition is to build machines, called classifiers, that will automatically assign measurements to classes.

Figure 2-1 The sick/healthy problem in pattern space

A natural way to make the class assignment is to define the boundary temperature between sick and healthy individuals. This boundary is called the decision surface. The decision surface is not trivially determined for many real-world problems. If we start measuring the temperature of healthy subjects with a thermometer, we will soon find out that individual temperatures vary from subject to subject, and change for the same subject depending on the hour of the day, the subject's state (e.g. at rest or after exercise), and so on. The same variability occurs in sick individuals (aggravated by the seriousness and type of illness), and there may be overlap between the temperature of sick and healthy individuals. Thus, we immediately see that the central problem in pattern recognition is to define the shape and placement of the boundary so that the class-assignment errors are minimized.

2.1.1 Can Regression be Used for Pattern Recognition?

In Chapter 1 we presented a methodology that builds adaptive machines with the goal of fiting hyperplanes to data points. A legitimate question is to ask whether regression can be used to solve the problem of separating data into classes. The answer is no because the goals are very different.

  • In regression, both the input data and desired response are experimental variables (normally real numbers) created by a single unknown, underlying mechanism.

  • The goal in regression is to find the parameters of the best linear approximation to the input and the desired response pairs.

The regression problem is therefore one of representing the relationship between the input and the desired response.

In classification the issue is very different. We accept a priori that different input data may be generated by different mechanisms and that the goal is to separate the data as well as possible into classes. The desired response is a set of arbitrary labels (a different integer is normally assigned to each one of the classes), so every element of a class will share the same label. Class assignments are mutually exclusive, so a classifier needs a nonlinear mechanism such as an all-or-nothing switch. At a high level of abstraction, both the classification and the regression problems seek systems that transform inputs into desired responses, but the details of the mappings are rather different in the two cases.

We can nevertheless use the machinery utilized in linear regression, the Adaline and the LMS rule, as pieces to build pattern classifiers. Let us see how we can do this in NeuroSolutions and what the results are.

NEUROSOLUTIONS EXAMPLE 2.1

Comparing Regression and Classification

Suppose we are given the healthy and sick data, and we arbitrarily assign the value 1 as the desired system response to the healthy class, and the –1 desired response to the sick class. With these assignments we can train the Adaline of Chapter I to fit the input/desired response pairs.

The important question is to find out what the solution means. Notice that for equal numbers of sick and healthy cases, the regression line intersects the temperature line at the mean temperature of the overall data set (healthy and sick cases), which is the centroid of the observations. The regression line is not directly useful for classification. However, one can place a threshold function at the output of the Adaline such that when its output is positive the response will be 1 (healthy), and when it is negative, the response is –1 (sick).

Now we have a classifier, but this does not change the fact that the placement of the regression line was dictated by the linear fit of the data and not by the requirement to separate the two classes as well as possible to minimize the classification errors. With the arrangement of an Adaline followed by a threshold, we have created our first classifier. But how can we improve its performance, estimate the optimal error rate, and extend it to multiple classes?

NeuroSolutions Example

The Adaline can be applied for classification when the system topology is extended with a threshold as a decision device. However, there is no guarantee of good performance, because the coefficients are being adapted to fit (in the least square sense) the temperature data to the labels 1 and -1, and not to minimize the classification error. This is an especially simple example with only two classes. For the multiple-class case the results become even more fragile. The conclusion is that we need a new methodology to study and design accurate classifiers. The algorithms we developed in Chapter 1, however, will be the basis for much of our future work. All the concepts learning curves, rattling, step sizes, and so on will be applicable.

Go to next section


 2.2 Parametric Classifiers

2.2.1 Optimal Decision Boundary Based on Statistical Models of Data

The healthy/sick classification problem can be modeled in the following way: Assume that temperature is a random variable (i.e., a quantity governed by probabilistic laws) generated by two different phenomena, health and sickness, and further assume a probability density function (pdf) for each phenomenon (usually a Gaussian distribution). From the temperature measurements we can obtain the statistical parameters needed to fit the assumed pdf to the data (for Gaussians only the mean and variance need to be estimated; - see Appendix A). Statistical decision theory proposes very general principles to construct the optimal classifier. Fisher showed that the optimal classifier chooses the class ci that maximizes the a posteriori probability P(ci|x) that the given sample x belongs to the class; that is,

(2.1)

The problem is that the a posteriori probability cannot be measured directly. However, using Bayes' rule,

(2.2)

one can compute the a posteriori probability from P(ci) the prior probability of the classes, multiplied by p(x|ci), the likelihood that the data x was produced by class ci and normalized by the probability of P(x). Both P(ci) and the likelihood can be estimated from the collected data and the assumption of the pdf. P(x) is a normalizing factor that can be left out in most classification cases. understanding Bayes rule

For our example i =1, 2 (healthy, and sick), and P(ci) can be estimated from the demographics, season, and so on. Figure 2-1 shows data collected from 100 cases. The likelihoods p(x|ci) can be estimated assuming a Gaussian distribution,

(2.3)

and estimating the means [image] i [image] and standard deviations [image] i of the distributions for sick and healthy individuals from the data. Using the sample mean and variance (N is the number of measurements)

(2.4)

for this data set gives the results of Table 2.1

Table 2-1 Statistical Measures for Figure 2-1 Data

The separation boundary, that is, the temperature x = T for which the two a posteriori probabilities are identical, can be computed for the one-dimensional case with simple algebra. In this case the optimal threshold is T = 37 C (Figure 2-2). Bayesian threshold

(a)

(b)

Figure 2-2(a) Sampled data distributions, (b) Bayes threshold T

It is rather easy to optimally classify healthy/sick cases using this methodology. Given a temperature x from an individual, we compute Eq. 2.2 for both classes and assign the label healthy or sick according to the one that produces the largest value (see Eq. 2.1). Alternatively, we can compare the measurement to T and decide immediately to use the label healthy if x < T , or sick if x > T. Notice that to the left of T the scaled likelihood of class healthy is larger than for the class sick, so measurements that fall in this area are more likely produced by healthy subjects and should be assigned to the healthy class. Similarly, the measurements that fall toward the right have a higher likelihood of being produced by sick cases.

The class assignment is not error free. In fact, the tail of the healthy likelihood extends to the right of the intercept point, and the tail of the sick likelihood extends to the left of T. The error in the classification is given by the sum of the areas under these tails, so the smaller the overlap the better the classification accuracy. The maximum a posteriori probability assignment (Eq. 2.1) minimizes this probability of error (minimum error rate) and is therefore optimal.

Metric for Classification

There are important conclusions to be taken from the preceeding example. For a problem with given class variances, if we increase the distance between the class means, the overlap will decrease and the classification becomes more accurate. This is reminiscent of the distance in Euclidean space when we think of the class centers as two points in space. However, we cannot just look at the class means to estimate the classification error, since the error depends on the overlap between the class likelihoods. The tails of the Gaussians are controlled by the class variance, so we can have cases where the means are very far apart, but the variances are so large that the overlap between likelihoods is still high. Inversely, the class means can be close to each other, but if the class variances are very small the classification can still be done with small error.

Hence, separability between Gaussian-distributed classes is a function of both the mean and the variance of each class. As we saw in the Bayesian threshold the placement of the decision surface is determined by the class distance normalized by the class variances. We can encapsulate this idea by saying that the metric for classification is not Euclidean, but also involves the dispersion (variance) of each class. If we closely analyze the exponent for the Gaussian distribution (Eq. 2.3), we can immediately see that the value of the function depends not only on [image] but also on [image]. The value of p(x) depends on the distance of x from the mean normalized by the variance. This distance is called the Mahalanobis distance.

Following this simple principle of estimating a posteriori probabilities, an optimal classifier can be built that is able to use temperature to discriminate between healthy and sick subjects. Once again, optimal does not mean that the process will be error free, only that the system will minimize the number of mistakes when the variable temperature is utilized.

2.2.2 Discriminant Functions

Assume we have N measurements x1, x2, xN, where each measurement xk is a vector with D components

(2.5)

and can be visualized as a point in the D-dimensional pattern space. Following Eq. 2.1, the class assignment by Bayes' rule is based on a comparison of likelihoods scaled by the corresponding a priori probability. Alternatively, the measurement xk will be assigned to class i if

(2.6)

Each scaled likelihood can be thought of as a discriminant function g(x), that is, a function that assigns a "score" to every point in the input space. Each class has its individual scoring function, yielding higher values for the points that belong to the class. Discriminant functions intersect in the input space defining a decision surface, where the scores are equal (Figure 2-3). Thus, decision surfaces partition the input space into regions where one of the discriminants is larger than the others. Each region is then assigned to the class associated with the largest discriminant.

Figure 2-3 Discriminant functions and the decision surface.

In this view the optimal classifier just compares discriminant functions (one per class) and chooses the class according to the discriminant gi(x) that provides the largest value for the measurement xk (Eq. 2.6). A block diagram for the general case of C classes is presented in Figure 2-4.

Figure2-4 General parametric classifier for c classes

The blocks labeled gi(x) compute the discriminants from the input data, and the block labeled maximum selects the largest value according to Eq. 2.1 or Eq. 2.6. Studying how the optimal classifier works, we arrive at the conclusion that the classifier system creates decision regions bounded by the intersection of discriminant functions. After this brief introduction, we realize that machines that implement discriminant functions can be used as pattern classifiers. parametric and nonparametric classifiers

2.2.3 A Two-Dimensional Pattern-Recognition Example

The temperature example is too simple (the input is one-dimensional) to illustrate the full method of deriving decision boundaries based on statistical models of the data, the variety of separation surfaces, and the details and difficulty of the design. We will treat here the two-dimensional case, because we can still use pictures to help guide our reasoning. The method, of course, can be generalized to any number of dimensions.

Let us consider the following problem: Suppose that we wishe to classify males and females in a population by using measurements of their height and weight. Since we selected two variables, this is a two-dimensional problem. We are going to assume for simplicity that the distributions of height and weight are multivariate Gaussians and that the probability of occurrence of each sex is ½. Figure 2-5 shows the scatter plot of measurements for the two classes.

Figure 2-5 Scatter plot of the weight and height data with the optimal decision surface

The goal is to determine the placement of the decision surface for optimal classification. According to our previous discussion of the one-dimensional, this is achieved by estimating the means and standard deviations of the likelihoods from measurements performed in the population. Then the decision boundary is found by solving for gi(x) = gj(x), where i and j are the classes male and female. The difference from our previous example is that here we have a two-dimensional input space.

One can show [Duda and Hart] that for our two-dimensional, two-class case (normally distributed) with equal a priori probabilities, the classification will be a function of a normalized distance between the class centers, called the Mahalanobis distance

(2.7)

where [image] is the class center vector and [image] is the covariance matrix of the input data in two-dimensional space. Notice that instead of the class variances, in the multidimensional case we have to compute the covariance matrix that is built from the class variances along each input space dimension. For this case the discriminant function is given by (derivation of quadratic discriminant)

(2.8)

Our example is a two-dimensional problem (D = 2) with two classes (i = 1, 2). Since the classes are equiprobable, the last term in Eq. 2.8 will be the same for each discriminant and can be dropped (likewise for the term D / 2 log(2 [image])). When the discriminants for each class are equated to find the decision surface, we see that in fact its placement is going to be a function of the Mahalanobis distance and class covariances. As in the one-dimensional case, for classification, what matters is the distance among the cluster means normalized by the respective covariance. This is the metric for classification, and it is beautifully encapsulated in the Mahalanobis distance.

Table 2.2 shows the estimates for the class covariances and means considering 100 and 1,000 samples per class. We will solve the problem with 1,000 samples first by computing the discriminants for each class.

Table 2-2 Data Statistics

In this case the decision surface is given by the Bayes classifier

which is an equation for a quadratic curve in the input space (Figure 2-5).

2.2.4 Decision Surfaces of Optimal Classifiers

It can be shown (Fukunaga) that the optimal classifier for Gaussian-distributed classes is always a quadratic. There are three cases of interest:

  • Covariance matrices are diagonal and equal.

  • Covariance matrices for each class are equal.

  • The general case.

For the two first cases, both the optimal discriminants and the decision surface are linear (Figure 2-6). These plots tell us how the density of samples decreases away from the center of the cluster (the mean of the Gaussian). The optimal discriminant function depends on each cluster shape, and it is in principle a quadratic. When the cluster shapes are circularly symmetric with the same variance, there is no difference in the radial direction for optimal classification, so the discriminant defaults to a linear function (a hyperplane). The decision surface is also linear and is perpendicular to the line that joins the two cluster centers. For the same a priori probability the decision surface is the perpendicular bisector of the line joining the class means.

Even when the shapes of the clusters are skewed equally (the contour plots of each cluster are ellipses with equal axes), there is no information to be explored in the radial direction, so the optimal discriminants are still linear. But now the decision surface is a line that is slanted with respect to the line joining the two cluster means.

Figure 2-6 Contour plots for the data clusters that lead to linear discriminant functions

Figure 2-7 shows the contours of each data cluster and the discriminant for the case of arbitrary covariance matrices. Notice the large repertoire of decision surfaces for the two-class case when we assume Gaussian distributions. The important point is that the shape of the discriminants is highly dependent on the covariance matrix of each class. Knowing the shape of one data cluster is not enough to predict the shape of the optimal discriminant. We need knowledge about BOTH data clusters to find the optimal discriminant. shapes of 2D discriminants

Figure 2-7 The general case of arbitrary covariance matrices

Eq. 2.8 shows that the optimal discriminant for Gaussian-distributed classes is a quadratic function. This points out that the relationship between cluster distributions and discriminants for optimal classification is not unique, i.e. there are many possible functional forms for the optimal discriminant (Gaussians, quadratics, and hyperplanes for this case). Observe that the parameters of the discriminant functions are a direct function of the parameters of the class likelihoods, so once the parameters of Eq. 2.8 are estimated, we can immediately determine the optimal classifier. For the female-male example, Figure 2-8 shows the modeled distribution of the two classes.

Figure 2-8 Modeled distribution of Figure 2-5 data

Discriminant Sensitivity to the Size of the Data

We have developed a strategy that is able to construct optimal decision surfaces from the data under the assumptions that the pdf of each class is Gaussian. This is a powerful procedure, but it is based on assumptions about the pdf of the input and also requires enough data to estimate the parameters of the discriminant functions with little error. The ultimate quality of the results will depend upon how valid these assumptions are for our problem. We illustrate here the effect of the training data size on the estimation of the discriminant function parameters.

Let us assume that we had only 100 samples for the height/weight example (50 males and 50 females). We extracted these samples randomly from the larger data file and computed the means and covariances for each class, as shown in Table 2.2. Just by inspection you can see that the parameters changed quite a bit. For instance, the covariances are no longer diagonal matrices, and the elements also have different values. Note also that the mean estimates are more accurate than the covariance estimates. When we build the optimal discriminant function from these parameters (Eq. 2.8) the shape and position in the input space is going to be different from the "ideal" case of 1,000 samples.

Figure 2-9 shows the differences in shape and placement of the optimal decision surface for the two data sets. In this case the differences are significant, producing different classification accuracy, but the decision surfaces still have the same overall shape. But remember that this is a simple problem in two-dimensions (7 parameters to be estimated with 50 samples per class). In higher-dimensional spaces the number of parameters to be estimated may be of the same order of magnitude of the data samples, and in this case catastrophic differences may occur. So what can we do to design classifiers that are less sensitive to the a priori assumptions and the estimation of parameters?

The answer is not clear-cut, but it is related to the simplicity of the functional form of the discriminant. We should use discriminant functions that have fewer parameters and that can be robustly estimated from the amount of data that we have for training. These simpler discriminants may be suboptimal for the problem, but experience shows that many times they perform better than the optimal discriminant. This seems a paradox, but the reason can be found in the brittleness of the parameter estimation. Even if we use the quadratic discriminant (which is optimal for Gaussian-distributed classes), the classifier may perform poorly if its discriminant functions are not shaped and positioned accurately in the input space.

Figure 2-9 Comparisons of decision surfaces

Go to next section


 2.3 Linear and Nonlinear Classifier Machines

2.3.1 The Linear Machines

Thus far we have encountered three types of discriminant functions: the linear, the quadratic and the Gaussian. Let us compare them in terms of the number of free parameters for D-dimensional data. The linear discriminant function given by

(2.9)

has a number of parameters that increases linearly with the dimension D of the space. The discriminant function with the next higher degree polynomial, the quadratic, has a square dependence on the dimensionality of the space (i.e., it has approximately D² parameters) as we can see in Eq. 2.8. The Gaussian gave rise to a quadratic discriminant by taking the logarithm. Although quadratics are the optimal discriminants for Gaussian-distributed clusters, it may not be feasible to properly estimate all these parameters in large-dimensional spaces unless we have a tremendous amount of data.

Notice that Eq. 2.9 is a parametric equation for a hyperplane in D-dimensions, which we already encountered in linear regression (although there it had the function of modeling the input-output relationship). The hyperplane can be rotated and translated (an affine transformation) by changing the values of the free parameters wi and b, respectively. The system that implements the discriminant of Eq. 2.9 is depicted in Figure 2-10 and is called a linear machine. Notice that the pattern-recognizer block is simpler (compared with Figure 2-4) because the unspecified functions g(x) become simply a sum of products. We have seen that the linear machine is even optimal for Gaussian distributed classes with equal variances, which is a case of practical relevance (for example data transmission in stationary, noisy channels).

Figure 2-10 Linear classifier for c classes

It is ironic that in our classifier design methodology we are again considering linear functions for discrimination. It seems that we are back at the point where we started this chapter, the construction of a classifier based on the linear regressor followed by a threshold. But notice that now we have a much better idea of what we are seeking. The pattern-recognition theory tells us that we may use linear discriminants, but we use one per class, not a regression line linking all the input data with the class labels. We also now know that the linear discriminant may not be optimal but may be the best we can do because of the practical limitation of insufficient data to properly estimate the parameters of the optimal discriminant functions. The pattern-recognition theory thus gave us the insight to seek better solutions.

It is important to stress that the linear discriminant is less powerful than the quadratic discriminant. A linear discriminant primarily utilizes differences in means for classification. If two classes have the same mean, the linear classifier will always produce poor results. Examine Figure 2-11. A linear separation surface will always misclassify approximately half of the other class. However, the quadratic discriminant does a much better job because it can utilize the differences in the covariances.

Figure 2-11 Comparison of the discriminant power of the linear and quadratic classifiers

2.3.2 Kernel-Based Machines

A more sophisticated learning machine architecture is obtained by implementing a nonlinear mapping from the input to another space, followed by a linear discriminant function (Figure 2-12). See Nilsson. The rationale of this architecture is motivated by Cover's theorem. Cover basically states that any pattern-recognition problem is linearly separable in a sufficiently high dimensionality space, so the goal is to map the input space to another space, called the feature space [image] [image] [image] using nonlinear transformations. Let us assume that the mapping from the input space x = [x1,…xD] to the higher dimensional [image] space is a one-to-one mapping operated by a kernel function family,

applied to the input. For instance, we can construct a quadratic mapping in this way by equating the first D components of K to [image], the next D(D-1)/2 components to all pairs, xixj i [image] j , and the last D components to xi The feature space [image] in this case is of size M=[D(D+3)]/2.

Figure 2-12 A kernel-based classifier

There is great flexibility in choosing the family of functions K(x). They need to be nonlinear, such as Gaussians, polynomials, trigonometric polynomials, and so on. Then in [image] space we can construct a linear discriminant function as

As before the problem is to select the set of weight vector w in [image] space that classifies the problem with minimum error. The general architecture for the kernel classifier is to build a kernel processor (which computes [image] [image] [image] x [image] ) followed by a linear machine. In the previous example, we actually constructed a quadratic discriminator. The major advantage of the kernel-based machine is that it decouples the capacity of the machine (the number of free parameters) from the size of the input space. size of feature space

Recently Vapnik has shown that if K(x) are symmetric functions that obey the Mercer condition (i.e. K(x) represents an inner product in the feature space), the solution for the discriminant function problem is greatly simplified. The Mercer condition basically states that the weights can be computed without ever solving the problem in the higher dimensional space [image], which gives rise to a new classifier called the Support Vector Machine (SVM). We will study the SVM later.

2.3.3 Classifiers for the Two Class Case

The general classifier can be simplified for the two-class case, since only two discriminant functions are necessary. It is sufficient to subtract the two discriminant functions and assign the classes based on the sign of a single, new discriminant function. For instance, for the sick-healthy classification

(2.10)

which leads to the block diagram in Figure 2-13.

Figure 2-13 Classifier for a two-class problem

Note that gnew(x) divides the space into two regions that are assigned to each class. For this reason, this surface obeys the definition of a decision surface , and its dimension is one less than the original data space dimension. For the 1-D case it is a threshold (a point) at 37°C. But will be a line (1-D surface) in 2-D space, and so on.

It is important at this point to go back to our NeuroSolutions Example 2.1 where we built a classifier from an Adaline followed by a threshold function, and compare that solution with Figure 2-13. We can conclude that the Adaline is effectively implementing the discriminant function gnew(x). Due to the particular way that we defined the labels (1,-1), the regression line will be positive in the region of the temperatures for healthy individuals and negative toward the temperatures of sick individuals. The sign of the regression thus effectively implements gnew(x). There are several problems with this solution:

  • First, there is no rigorous way to choose the values of the desired response, and they tremendously affect the placement of the regression line (try 0.9, and -0.1 instead of [image] and see how the threshold changes).

  • Second, it is not easy to generalize the scheme for multiple classes (the sign information can be used only for the two-class case). As we saw, classification requires a discriminant function per class.

  • Third, the way that the Adaline was adapted has little to do with minimizing the classification error. The error for training comes from the difference between the Adaline output (before the threshold) and the class labels (Figure 2-14). We are using a nonlinear system, but the information to adapt it is still derived from the linear part.

Only under very restricted conditions will this scheme yield the optimum classifier. In the following chapter we will learn to implement a classifier in which the classification error (the error after the threshold) is used to train the network.

Figure 2-14 Schematic training of the Adaline with threshold

The Adaline followed by a threshold, as shown in Example 2.1, was applied in the 1960s by Widrow and Hoff for classification purposes.

Go to next section


 2.4 Methods of Training Parametric Classifiers

The methods that we present in this book assume that there is little information available to help us make principled decisions regarding the parameter values of the discriminant functions. Therefore, the parameters must be estimated from the available data. One must first collect sufficient data that covers all the possible cases of interest, then use this data to select the parameters that produce the smallest possible error. This is called training the classifier, and we found a very similar methodology in Chapter 1.

The accuracy of a classifier is dictated by the location and shape of the decision boundary in pattern space. Since the decision boundary is obtained by the intersection of discriminant functions, there are two fundamental issues in designing accurate parametric classifiers (i.e., classifiers that accept a functional form for their discriminant functions):

  • The placement of the discriminant function in pattern space

    • The functional form of the discriminant function

There are two different ways to use the data for training parametric classifiers (Figure 2-15 – do not confuse parametric classifiers with parametric training).

Figure 2-15 Parametric and nonparametric training of a classifier

In parametric training each pattern category is described by some known functional form for which its parameters are unknown. The decision surface can then be analytically defined as a function of these unknown parameters. The method of designing classifiers based on statistical models of the data belongs to parametric training. We describe the data clusters by class likelihoods, and the discriminants and decision surfaces can be determined when these parameters are estimated from the data. For instance, for Gaussian-distributed pattern categories, we need to estimate the mean vector, the covariance (normally using the sample mean and the sample covariance) and the class probabilities to apply Eq. 2.1. Unfortunately, due to the analytic way in which discriminants are related to the likelihoods, only a handful of distributions have been studied, and the Gaussian is almost always utilized.

In nonparametric training the free parameters of the classifier's discriminant functions are directly estimated from the input data. Assumptions about the data distribution are never needed in nonparametric training. Very frequently, nonparametric training utilizes iterative algorithms to find the best position of the discriminant functions. However, the designer still has to directly address the two fundamental issues of parametric classifier design, that is, the functional form of discriminant functions and their placement in pattern space.

2.4.1 Parametric versus Nonparametric Training

Let us raise an important issue. Is there any advantage in using nonparametric training? The optimal discriminant function depends on the distribution of the data in pattern space. When the boundary is defined by statistical data modeling (parametric training), optimal classification is achieved by the choice of good data models and appropriate estimation of their parameters. This looks like a perfectly fine methodology to design classifiers. Therefore, in principle, there seems to be no advantage in nonparametric training, which starts the classifier design process by selecting the discriminant functional form "out of the blue." In reality, there are some problems with parametric training for the following reasons:

  • Poor choice of likelihood models. When we select a data model (e.g. the Gaussian distribution), we may be mistaken, so the classifier may not be the optimal classifier after all. Conversely, estimating the form of the pdf from finite training sets is an ill-posed problem, so this selection is always problematic.

  • Too many parameters for the optimal discriminant. We saw earlier that the performance of the quadratic classifier depends on the quality of the parameter estimation. When we have few data points, we may not get a sufficiently good estimation of the parameters, and classification accuracy suffers. As a rule of thumb we should have 10 data samples for each free parameter in the classifier. If this is not the case we should avoid using the quadratic discriminator. One can say that most real-world problems are data bound; that is, for the number of free parameters in the optimal classifier, there is not enough data to properly estimate its discriminant function parameters.

Very often we are forced to trade optimality for robustness in the estimation of parameters. In spite of the fact that the quadratic classifier is optimal, sometimes the data can be classified with enough accuracy by simpler discriminants (such as the linear), as we showed in the shapes of 2-D discriminants. These discriminants have fewer parameters and are less sensitive to estimation errors (take a look at Table 2.2 and compare the estimations for the means and variances), so they should be used instead of the quadratic discriminants when the data is not enough to estimate the parameters accurately.

This raises the question of using the data to directly estimate the parameters of the discriminant functions, that is choosing a nonparametric training approach. What we gain is classifiers that are insensitive to the assumption on the pdf of the data clusters. We can also control in a more direct way the number of free parameters of the classifier.

2.4.2 Issues in Nonparametric Training

The difficulties that are brought about by nonparametric training are twofold:

  • Deciding the shape of the discriminant function for each class

  • Finding ways to adapt the parameters of the classifier

Shape of Discriminant Functions

The central problem in nonparametric training of parametric classifiers can be re-stated as the selection of an appropriate functional form for the discriminant function that meets these two criteria:

  • It produces small classification error

  • It has as few parameters as possible to enable robust estimation from the available data

The linear-machine decision boundaries are always convex because they are built from the superposition of linear discriminant functions. This is very restrictive, and solving realistic problems may thus require more versatile machines. The kernel-based classifiers are one option. They form convex boundaries in the high dimensional [image] space, but not necessarily in the input space. Another class of more versatile machines in called semi-parametric classifiers because they still work with parametric discriminant functions but are able to implement a larger class of discriminant shapes (eventually any shape, which makes them universal approximators). Semiparametric classifiers are very promising because they are an excellent compromise between versatility and the number of trainable parameters. Artificial neural networks are one of the most exciting types of semiparametric classifiers and will be the main subject of our study.

Adaptation of Classifier Parameters

We can use the ideas of iterated training algorithms to adapt the classifier parameters. In Chapter 1 we trained the Adaline parameters directly from the data, so the linear regressor was effectively nonparametrically trained. We can also foresee the use of the gradient descent method explained in Chapter 1 to adjust the parameters of the discriminant functions so that a measure of the misclassifications (output error) is minimized. Gradient descent will be extended in the next chapter for classifiers.

Go to next section


 2.5 Conclusions

In this short chapter we covered the fundamental principles of pattern recognition. We started by briefly reviewing the problem of pattern recognition from a statistical perspective. We provided the concepts and definitions to understand the role of a pattern recognizer. We covered Bayes' classifier and showed that the optimal classifier is quadratic. Sometimes suboptimal classifiers perform better when the data is scarce and the input space is large, in particular, when the input is projected into a large feature space, as done in kernel classifiers.

The linear classifier was also reviewed, and a possible implementation was provided. These concepts are going to be very important when we discuss artificial neural networks in the next chapter.

Go to next section


 2.6 Exercises

2.1 Modify the desired response to (1,0) in NeuroSolutions Example 2.1 (use the "normalization" function of the "stream" page of the "file" inspector). Notice how the threshold changes. Elaborate on the dependence of linear machines in the actual values of the desired response. Is this good or bad?

2.2 In the chapter 2 folder of the data directory (CDROM) we provide data that can be used for classification (prob2.2 train and test). The last column is the class label. This data exists in a 2-D space such that we can still visualize the discriminant functions. It is also a 2-class problem for simplicity. In this problem you will design a linear classifier, assuming first that each class can be modeled by a Gaussian distribution. You should first find the parameters for each Gaussian, and then design the classifier. Plot the separation surface over the data.

2.3 For the data in Problem 2, design the optimal Bayes' classifier and compare performance with the linear classifier by determining the percent of correct classifications. Plot also the separation surface and compare it with Problem 2. Do you think that for this data set the Bayes' classifier is optimal?

2.4 If you want to use a linear classifier but train it nonparametrically can you do it? What is missing? Why is it that LMS cannot be used?

2.5 Compare the performance of the Bayes' classifier developed in Problem 3 when you use only the first 10 percent of the data for training.

2.6 Estimate the probability of error of the Bayes' classifier developed in Problem 3.

2.7 State in your own words the reason why the metric of classification is not Euclidean.

2.8 In the ch2 folder of the data directory (CD-ROM) we provide data that can be used for classification (prob2.8 train and test). The last column is the class label. In this case the problem is a bit more difficult, with three classes in a higher (five) dimensional space. Develop a linear classifier and test the results. You may want to use Matlab or a similar program. Is it appropriate to develop a Bayes classifier to improve performance?

Go to next section


 2.7 NeuroSolutions Examples

2.1 Comparing Regression and Classification

Go to next section


 2.8 Concept Map For Chapter 2

Go to Next Chapter


 3.1 Artificial Neural Networks

There are many definitions of artificial neural networks (ANNs). We will use a pragmatic definition that emphasizes the key features of the technology. ANNs are distributed, adaptive, generally nonlinear learning machines built from many different processing elements (PEs). Each PE receives connections from other PEs and/or itself. The interconnectivity defines the topology. The signals flowing on the connections are scaled by adjustable parameters called weights, wij. The PEs sum all these contributions and produce an output that is a nonlinear (static) function of the sum. The PEs' outputs become either system outputs or are sent to the same or other PEs. Figure 3-1 shows an example of an ANN. Note that a weight is associated with every connection.

Figure 3-1 An artificial neural network

The ANN builds discriminant functions from its PEs. The ANN topology determines the number and shape of the discriminant functions. The shapes of the discriminant functions change with the topology, so ANNs are considered semiparametric classifiers. One of the central advantages of ANNs is that they are sufficiently powerful to create arbitrary discriminant functions so ANNs can achieve optimal classification.

The placement of the discriminant functions is controlled by the network weights. Following the ideas of nonparametric training, the weights are adjusted directly from the training data without any assumptions about the data's statistical distribution. Hence one of the central issues in neural network design is to utilize systematic procedures (a training algorithm) to modify the weights so that as accurate a classification as possible is achieved. The accuracy is quantified by an error criterion.

There is a style in training an ANN (Figure 3-2). First, data is presented, and an output is computed. An error is obtained by comparing the output with a desired response and it is used to modify the weights with a training algorithm. This procedure is repeated using all the data in the training set until a convergence criterion is met. Thus, in ANNs (and in adaptive systems in general) the designer does not have to specify the parameters of the system. They are automatically extracted from the input data and the desired response by means of the training algorithm.

Figure 3-2 General principles of adaptive system's training

The two central issues in neural network design (semiparametric classifiers) are the selection of the shape and number of the discriminant functions and their placement in pattern space such that the classification error is minimized. We will address all these issues in this chapter in a systematic manner. The function of the PE is explained, both in terms of discriminant function capability and learning. Once this is understood, we will start putting PEs together in feedforward neural topologies with many layers. We will discuss both the mapping capabilities and training algorithms for each of the network configurations.

Go to next section


 3.2 Pattern Recognition Ability of the McCulloch-Pitts PE

The McCulloch-Pitts (M-P) processing element is simply a sum-of-products followed by a threshold nonlinearity (Figure 3-3). Its input-output equation is

(3.1)

where D is the number of inputs, xi are the inputs to the PE, wi are the weights and b is a bias term. The activation function ƒ is a threshold function defined by

(3.2)

which is commonly called the signum function. Note that the M-P PE is composed of the adaptive linear element (Adaline) studied in Chapter 1 followed by a nonlinearity.

We will now study the pattern-recognition ability of the M-P PE. The study will use the interpretation of a single discriminant function as given by Eq.2.9. Note that such a system is able to separate only two classes (one class associated with the +1 and the other with the -1 response). Figure 3-3 represents the network we are going to build in NeuroSolutions.

Figure 3-3 Two input, one output (2-1) McCulloch-Pitts PE

NEUROSOLUTIONS EXAMPLE 3.1

McCulloch and Pitts PE for Classification

The McCulloch-Pitts PE is created by the concatenation of a Synapse and an Axon. The Synapse contains the weights wi, and performs the sum-of-products. The Synapse Inspector shows that the element has 2 inputs and one output. The number of inputs xi is set by the Input Axon. The soma level of the Inspector shows that the element has two weights. The number of outputs is set by the component to its right (the Threshold Axon) and is one in this case.

The Threshold Axon adds its own bias b to the sum of products and computes a static nonlinearity. The shape of the nonlinearity is drawn on the icon of the Axon, which is a step for the Threshold Axon. So this M-P PE maps 2-D patterns to the values {-1, 1}. Basically, the M-P PE is like the Adaline we built in Chapter I, but now the Bias Axon (which is linear) is enhanced with a nonlinearity. This network is very simple, but we can call upon our geometric intuition to understand the input-output map of the M-P PE.

In this example we use two new components, the Threshold Axon and the Function Generator. The Function Generator is a component that is typically used for input and can create common signals such as sine waves, ramps, impulse trains, and so on.

NeuroSolutions Example

The question that we want to raise now is. What is the discriminant function created by this neural network? Using Eq. 3.2 the output of the processing unit is

(3.3)

We can recognize that the output is controlled by the value of w1x1 + w2x2 + b, which is the equation of a plane in two dimensions. This is the discriminant function g(x1, x2) used by the M-P PE, and we see that it is the output of the Adaline. When the threshold operates on this output, it divides the space into two half planes, one with a positive value (+1) and another with a negative value (-1). This is exactly what we need to implement a classifier for the two-class case (see Section 2.3.3). The equation for the decision surface reads

(3.4)

which can be readily recognized as a line with slope

(3.5)

passing through the point (0, -b/w2) of the plane (x2-intercept), or alternatively, at a distance -b/|w| from the origin, where [image]. For this reason b is called a bias.

Can we visualize the response of this system to inputs? If the system was linear, linear system theory could be applied to arrive at a closed-form solution for the input-output map (the transfer function). But for nonlinear systems the concept of a transfer function does not apply. Eq. 3.3 provides the answer for the M-P PE, but this is a very simple case where the output has just 2 values (-1,1). In general, the output is difficult to obtain analytically, so we will resort to an exhaustive calculation of the input-output map, that is, values from every location of the input space are injected into the network, and the corresponding output is computed. Let us restrict our attention to a square region of the input space between -1,1 (x1,x2 [image] [ [image] 1,1]) for now.

NeuroSolutions has a probe component that will compute and display this input-output map of Eq. 3.3. It is called the discriminant probe. Its function is to scan the input field (sequentially generating x1,x2 values), compute the corresponding output, and display it as a gray-scale image. Negative values are displayed as black, positive values as white. The discriminant function is a plane, and its intersection with the [image] plane is a line (the decision surface) that is given by Eq. 3.4. This is the line we see in the discriminant probe between the white and the black regions, and it represents the decision surface.

Before actually starting the simulation, let us raise the question: what do you expect to see? Eq. 3.4 dictates the position of the dividing line between the 1 and -1 responses. Using the values of NeuroSolutions Example 3.2, the decision surface is a line described by the equation

(3.6)

with slope m=-1 and passing through the point x2=0.277 (see vector interpretation of the decision surface). The dividing line (i.e., the decision surface for the two-class case) passes through the point x2=-0.277 and the slope is -1, corresponding to the angle 135 [image] (second quadrant). The position of the decision surface allows us to imagine the location of the discriminant function (Figure 3-4).

Figure 3-4 Linear discriminant function in two dimensions for the two class case

Let us now observe the simulation.

NEUROSOLUTIONS EXAMPLE 3.2

Discriminant Probe to Visualize the Decision Surface

This example brings the Discriminant Probe to the Breadboard. The Discriminant Probe is a pair of DLLs which force data through the network and display the system response, giving us an image of the input output map of the system. In this case we use it to show the discriminant line (separation surface) created by the M-P PE as given by Eq. 3.4. One component of the Discriminant Probe is placed on the Input Axon to send the input coordinates through the network, and the other component is placed on the Output Axon to display the system response.

You are free to experiment with the M-P PE. We suggest that you modify the Synapse weight values and the Threshold Axon bias by placing the cursor in the Matrix Editor field with the mouse and typing in the new values. For every combination you should first guess the solution by computing the slope and y intersect according to Eq. 3.4, and then finding out if you are correct by running the example. To run the example you need to press the Start button on the toolbar controller.

NeuroSolutions Example

3.2.1 Sigmoidal Nonlinearities

This simple example shows that the decision surface between the 1 and -1 responses created by the M-P PE is a line in 2-D space (a function that is linear in the parameters). The same conceptual picture works for higher-dimensional spaces, where the straight line becomes a hyperplane in a space of dimension one less than the input space dimension (but unfortunately we lose our visual intuition).

Notice also how crisp the decision surface is, since a hard threshold acts on the output of the discriminant function. However, other nonlinearities can be utilized in conjunction with the M-P PE. Let us now smooth out the threshold, yielding a sigmoid shape for the nonlinearity. The most common nonlinearities are the logistic and the hyperbolic tangent (tanh) functions of Figure 3-5.

Figure 3-5 Common nonlinearities in neurocomputing

is a slope parameter and normally is set to 1. The major difference between the two sigmoidal nonlinearities is the range of their output values. The logistic function produces values between [0,1], while the hyperbolic tangent produces values between [-1,1]. An alternate interpretation of this PE substitution is to think that the discriminant function has been generalized to

(3.7)

which is sometimes called a ridge function. The ridge function is no longer a hyperplane, since it saturates at 0 (or –1) and +1. However, the intersection of ridge functions can still be approximated in most of the input space by the intersections of their arguments. In fact, the argument of the ridge function is still linear in the input variables, and as long as the function f is steep, the decision surface is still approximately linear (see decision surfaces of the sigmoid PEs). Thus, the previous interpretation for threshold PEs still holds approximately for sigmoid PEs.

The other big advantage of the Tanh Axon (and also of the Sigmoid Axon) is that the nonlinearity is smooth, which means that the derivative of the map exists. This point is going to be very important later for adaptation. We conclude that both the Tanh Axon or the Sigmoid Axon can be substituted for the Threshold Axon with some practical advantages.

We refer to the combination of the Synapse and the Tanh Axon (or the Sigmoid Axon) as the modified McCulloch-Pitts PE because they all respond to the full input space in basically the same functional form (a sum of products followed by a global nonlinearity).

NEUROSOLUTIONS EXAMPLE 3.3

Behavior of the Sigmoid PEs

This breadboard substitutes the Threshold Axon with a Tanh Axon and then a Sigmoid Axon. The difference between the Threshold Axon and the Tanh Axon is that there is a smooth transition between the values of -1 and 1. We can visualize the PE nonlinearity in the component's corresponding Inspector.

The net effect of this modification in the PE input-output function is that the crisp separation between the two regions (positive and negative values of g(.)) is substituted by a smooth transition region between the values of -1 and +1. Hence the name ridge function. However, the orientation of the separation surface is still defined by the ratio of w1 and w2, and its vertical placement is still controlled by the bias of the Axon. A new feature of this nonlinearity is that the absolute values of the weights control the width of the gray region. By increasing the values of the weights while leaving the ratio of the weights constant, we can make the separation surface become crisper -- eventually approximating the one of the Threshold Axon.

The logistic nonlinearity is similar to the tanh; however, the range is between 0 and 1.The tanh is an antisymmetric function (y intercept is zero), while the logistic is not (y intercept is 0.5). The Tanh Axon and Logistic Axon are normally interchangeable, with the final selection determined by the desired range of the output (either [-1,1] or [0,1]).

NeuroSolutions Example

The output of the logistic function varies from 0 to 1. It is interesting to note that under some conditions, the logistic function allows a very powerful interpretation of the output of the PE as a posteriori probabilities for Gaussian-distributed input classes (see probabilistic interpretation of sigmoid outputs). The tanh is closely related to the logistic function by a linear transformation in the input and output spaces, so neural networks that use either of these can be made equivalent by changing weights and biases.

3.2.2 Classification Implies the Control of the Discriminant Function Location

Remember that the ratio of the weights controls the slope (orientation) of the separation surface, and the PE bias controls the x2 intercept (Figure 3-4). So how can the M-P PE system be used to distinguish two classes of patterns?

The discriminant function placement should be controlled so that the system outputs the value 1 for one of the classes and -1 (or 0) for the other, that is, the discriminant function must be moved around in the input space to yield the minimum number of errors. As external observers, we can do this easily by looking at the data clusters in 2-D and placing the separation line between them. But in higher-dimensional spaces we cannot visualize the data clusters, so we need to follow some type of step-by-step procedure.

NEUROSOLUTIONS EXAMPLE 3.4

Classification as the Control of the Decision Surface

The input file placed on the Input Axon links NeuroSolutions to the computer file system. We created a file with 8 points according to the following table. The third column represents the class membership. Since our network is built with a logistic function that ranges from 0 to 1, these values can also be interpreted as the desired network response.

We attached the L2 Criterion (Mean Square Error) at the output of the Logisitic Axon. This PE computes the square difference between the system output and the desired response for each input pattern. The output file works as the desired response, and in this case contains the values used in the "class" column of the table. Note that we have a Matrix Viewer on top of the L2 Criterion to provide a numerical indication of the power of the error (MSE), that is, the difference between the machine output and the output file value. All the eight input samples are sent through the network to display the average error over the entire data set.

One other component deserves to be mentioned. The Scatter Plot probe is placed at the output of the Input Axon and will show the x, y locations of the input data. Since we are interested in displaying all the training patterns, the data buffer size is set to 8. The purpose of this example is to relate the position of the decision surface and the MSE, so experiment with several weights to obtain perfect classification of all the patterns.

As you adjust the network weights, note that the sign of the response may be correct, but the error is still not zero. This is the problem of using a saturating nonlinearity. We have to use very large weights to obtain a response close to 1 and 0. This example shows that acceptable solutions for classification do not require the error to be exactly zero.

Let us now experiment with a different set of data. On the book's CD-ROM, in the "InteractiveBook\Examples\chapter3\McPitts3" subdirectory, there is another file called mcpitts_data1.asc. Go to the Input File component and, with the Inspector open, select the "File List" page. Click the Remove button to deselect the present file and click on the Add button. This will put you in the Win95 Open File panel. Select the file, and NeuroSolutions will ask if it is an ASCII column training file (click the Close button to finalize the selection). Another panel pops up, allowing you to select which columns are used in the experiment. Since the file has three columns, two for inputs and one for the desired response, and we are selecting the input data, we have to skip the third column. Select the third row, click the Skip button, and close the panel. We just completed the selection of the new file. You have to do the same thing for the desired signal file, but now you should skip the first two columns.

NeuroSolutions Example

The procedure we outlined in the example can be automated by the machine itself, without any outside help, by providing feedback to the machine on how it is doing. As in regression, the feedback comes in the form of an error ( [image] p) between the desired response (dp) and the actual output (yp). Note that when the error is zero, the machine output is equal to the desired response. Once the error is defined, the obvious thing to do is to minimize either the error directly or a function of the error that is normally called the cost function or the criterion.

There are many possible definitions of the error (we will treat error criterion in Chapter 4), but in neurocomputing one commonly uses the mean square error (or power), that is, the sum of the square difference between the desired response and the actual output. We already used this criterion in Chapter 1. For ease of explanation we copy it here,

(3.8)

where p is the pattern index. The goal of the classifier is to minimize this cost function by changing its free parameters. This search for the weights to meet a desired response or internal constraint is the essence of any connectionist computation. The central problem to be solved on the road to machine-based classifiers is how to automate the process of minimizing the error so that the machine can independently make these weight changes, without need for hidden agents or external observers.

3.2.3 The First Learning Algorithm for a Nonlinear Machine

As a historical note, Rosenblatt proposed the following procedure to directly minimize the error by changing the weights of the M-P PE: Get an example from class 1 and examine the output. If the output is correct do nothing. If the response is incorrect, tweak the weights and bias until the response becomes correct. Get the next example and repeat the procedure, until all the patterns are correctly classified. This procedure is basically the perceptron learning algorithm, which can be put into the following equation:

(3.9)

where [image] is the step size, y is the M-P PE output, and d is the desired response.

It is important at this stage to compare Eq. 3.9 with the LMS algorithm we used to train the Adaline in the first example of Chapter 2. Note that the functional form is the same, that is, the old weights are incrementally modified proportionally to the product of the error and the input, but there is a significant difference. We cannot say that this corresponds to gradient descent since the system has a discontinuous nonlinearity. In the perceptron learning algorithm, y(n) is the output of the nonlinear system. The algorithm is directly minimizing the difference between the response of the M-P PE and the desired response, instead of minimizing the difference between the Adaline output and the desired response.

This subtle modification has tremendous impact on the performance of the system. For one thing, the M-P PE learns only when its output is wrong. In fact, when y(n) = d(n), the weights remain the same. The net effect is that the final values of the weights are no longer equal to the linear regression result, because the nonlinearity is brought into the weight update rule.

Another way of phrasing this is to say that the weight update became much more selective, effectively gated by the system performance. Notice that the LMS update is also a function of the error to a certain degree. Larger errors have more effect on the weight update than small errors, but all patterns affect the final weights – implementing a "smooth gate." In the perceptron the net effect is that the placement of the discriminant function is no longer controlled smoothly by all the input samples as in the Adaline, only by the ones that are important for placing the discriminant function in a way that explicitly minimizes the output error.

NEUROSOLUTIONS EXAMPLE 3.5

Perceptron Learning Rule

In this example we use the perceptron learning algorithm to classify the two previous data sets: the two-class problem from the previous example and the healthy/sick patient data. With the perceptron learning algorithm the system will learn automatically from the data without the need for us to select weights as we did in this chapter up to now.

This example uses two data sets that produce different results because one (the two-class problem with 8 patterns) is linearly separable, while the other is not. The perceptron learning algorithm trains only when the system response is incorrect; therefore, the weights will converge if all the inputs are classified correctly. Another side effect of training only on the errors is that if the problem is not linearly separable, the training will never stop. This can cause abrupt changes in the weights of the system near the end of the training, which may affect the classification performance. Since the perceptron learning algorithm does not search for the best answer, only a satisfactory answer, the network may not perform well on data that was not included in its training set. Note that there are many final weight values that produce an error of zero, that is, they exactly solve the linearly separable problem.

The network is now learning by itself, so we again have to add the backpropagation and gradient descent planes that we used at the end of Chapter 1. At the end of this chapter we will finally explain the details of these two additional planes. We also have to control the learning rate or step size to make sure that the system will converge to the optimum. You should experiment with other learning rates and observe the learning curve as our thermometer for learning.

NeuroSolutions Example

Let us assume that the patterns are linearly separable, that is, a linear discriminant function exists that produces zero classification error. The solution of the perceptron learning rule is a weight vector w* such that

(3.10)

where n is an index that runs over the training set data. This equation can be written in simpler form as [image]. Knowledge of the M-P PE tells us that for each input pattern (each n) there is a weight vector that produces the partitioning given by the desired response. The optimal weight vector is the vector that simultaneously meets all of these desired partitions. The solution in two dimensions is a line of equation [image] (i.e., the optimal weight vector w* has to be orthogonal to every data vector x). The weight update equation moves the weights directly towards this solution when it corrects w(n) by ± [image] x(n) depending on the sign of the error (see perceptron learning algorithm). This learning rule takes a finite number of steps to reach the optimal solution for linearly separable patterns. There are two major problems with this solution. First, as soon as the last sample is correctly classified, the discriminant function stalls, producing many different final decision surface placements that may not perform well in the test set. Second, this learning rule converges only if the classes are linearly separable. Otherwise the solution will oscillate.

3.2.4 The Delta Rule

We presented in Chapter 1 a systematic step-by-step procedure based on gradient descent (the LMS rule) that was able to modify the weights of the Adaline so that the minimum of the performance surface was reached. The algorithm (which should have memorized by now) is beautifully simple. It adds to the present weight a quantity proportional to the product of the error and the activation available at the PE (just two multiplications per weight), that is,

(3.11)

We are going to reexamine the LMS algorithm using a different concept that is central to learning in neural networks. This concept is an old result from calculus that is called the chain rule. Basically, the chain rule tells how to compute the partial derivative of a variable with respect to another when a functional form links the two. Let us assume that y = f(x), and the goal is to compute [image] y/ [image] x, the sensitivity of y with respect to x. As long as f is differentiable, then

(3.12)

Note that the value of Eq. 3.12 computes how much a change in x is reflected in y, that is, how sensitive y is to a change in x. When we work with sensitivities the calculations progress from the dependent variable to the independent variable. Keep this in mind.

One can show that the LMS rule is equivalent to the chain rule in the computation of the sensitivity of the cost function J with respect to the unknowns. derivation of LMS with the chain rule

Our goal is to extend the LMS concept to the M-P PE, which is a nonlinear system. How can we compute the sensitivity through a nonlinearity?

First, let us examine the problem. Figure 3-6 details the modified M-P PE, where we show multiple weights connected to its input. Note that the output of the modified M-P PE is a nonlinear function of the weights (Eq. 3.1).

Figure 3-6 Illustration of the sensitivity computation through a nonlinear PE

But we can still compute the partial derivative of the output PE with respect to its weights using the chain rule (Eq. 3.12). First compute the partial derivative of the output with respect to the intermediate signal net, and then compute the partial derivative of net with respect to the weight wi, that is,

(3.13)

where f [image] (.) is the partial derivative of the static nonlinearity. derivation of sensitivity through nonlinearity

This is another application of the famous chain rule, but now the chain rule is applied to the topology. As long as the PE nonlinearity is smooth (differentiable) we can compute how much a change in the weight [image] wi affects the output y, or from the point of view of the sensitivity, how sensitive the output y is to a change in a particular weight [image] wi. Note that we compute this output sensitivity by a product of partial derivatives through intermediate points in the topology. For the nonlinear PE there is only one intermediate point, net, but we really do not care how many of these intermediate points there are. The chain rule can be applied as many times as necessary.

In practice, we have an error at the output (the difference between the desired response and the actual output), and we want to adjust all the PE weights so that the error is minimized in a statistical sense. The obvious idea is to distribute the adjustments according to the sensitivity of the output to each weight. Why is this obvious? If we want to minimize the output error, we should make larger changes in the weights that most affect the output value, which is measured by the sensitivity. This is what the gradient descent does, hence the LMS rule.

To modify the weight, we actually propagate back the output error to intermediate points in the topology and scale it along the way as prescribed by Eq. 3.13 according to the elemental transfer functions that we find, as shown in Figure 3-6. This methodology is very powerful, because we do not need to know explicitly the error at intermediate places, such as net. The chain rule automatically derives the error contribution for us. This observation is going to be crucial for adapting more complicated topologies and will result in the backpropagation algorithm.

Let us now complete the formulas to adapt the modified M-P PE weights. Note that now we have two indices; the index i for the weight, and the index p for the pattern, and n for the iteration number. The mean square error is rewritten

(3.14)

By applying the chain rule twice, once for the output and again for the variable net, we get

(3.15)

The application of the gradient descent gives (compare with Eq. 3.11)

(3.16)

This rule is called the delta rule and is a direct extension of the LMS rule to nonlinear systems with smooth nonlinearities. Note that the derivative of the nonlinearity is computed at the operating point netp(n) for the corresponding input pattern. Note also that the delta rule is local to the pattern and to the weight, that is, it only requires knowledge of that specific pattern, the PE error, and its input. optimizing linear and nonlinear systems

Since the smooth nonlinearities discussed so far exponentially approach the values of -1 (or 0) and 1, the multiplication by the derivative reduces appreciably the error in most of the operating range since the derivative is bell shaped around netP=0. error attenuation

The derivative of the logistic function and the tanh are respectively

(3.17)

Our goal is to visualize the movement of the discriminant function during the adjustment of the PE weights using the delta rule. Let us go back to NeuroSolutions and automate the placement and display of the discriminant surface for this simple example.

NEUROSOLUTIONS EXAMPLE 3.6

Delta Rule to Adapt the M-P PE

We now use the delta rule to train the breadboard from Example 3.5. We use the same two-class data set with 8 points. When we run the network, observe the discriminant function move. The output mean square error (MSE) decreases close to zero when the discriminant function is placed between the two classes of points. The output of the network toward the end of learning mimics the desired response. Notice that at the beginning of the run the discriminant line changes its orientation quickly, and then at the end there is a period of fine-tuning. These are the two basic phases of learning; the discovery phase (what is the direction of the minimum?), and the convergence phase (fine tune to get to the minimum). In our case the convergence phase corresponds to increasing both the weights and bias (while keeping the ratio constant) in order to get a sharper cutoff between the two classes.

A small exercise to test your knowledge is to guess the network weights upon visualization of the discriminant function. You should at least be able to correctly guess the signs of the weights and their approximate ratio. Note that the final MSE is much smaller than the value obtained when we entered the parameters for the M-P PE by hand. Also note that there is no analytic solution for the optimal weights as we had in the linear case (the least squares method). Iterative solutions are therefore very appealing and practical when dealing with nonlinear systems.

It is instructive to change the learning rates and see how the solution behaves. You should be able to do this by now, even if the breadboard is not prepared for it (go to the Step icon, rightclick the mouse button to open the Inspector, and then enter other values for the step size). Try very large step sizes (300) for both parameters and see what happens. In this case the data is linearly separable, and it is very difficult to make the system weights diverge. The nonlinearity is always keeping the output between 0 and 1, no matter what the value of the weights are, so divergence of a single PE nonlinear system is not apparent from the value of the output.

An interesting challenge is to find a set of learning rates that will prevent the system from finding the correct discriminant line.

NeuroSolutions Example

3.2.5 Implications of the PE Nonlinearity

Several key aspects changed in the performance surface with the introduction of the nonlinearity. The nice, parabolic performance surface of the least squares problem is lost. Why is that? Note that the performance surface describes how the cost changes with the weights. But the performance depends on the topology of the network through the output error, so when nonlinear processing elements are used to solve a given problem the "performance - weights" relationship becomes nonlinear, and there is no guarantee of a single minimum. The performance surface may have several minima. The minimum that produces the smallest error in the search space is called the global minimum. The others are called local minima. Alternatively, we say that the performance surface is nonconvex. This affects the search scheme because gradient descent uses local information to search the performance surface. In the immediate neighborhood, local minima are indistinguishable from the global minimum, so the gradient search algorithm may be caught in these suboptimal performance points "thinking" it has reached the global minimum (Figure 3-7).

Figure 3-7 Nonconvex performance surface with gradients depicted

Other conditions where the gradient is basically zero are called saddle points (or flat spots). Since the weight update is ultimately produced by the gradient, when the gradient becomes very small, the weights do not change much and the adaptation may stall. Suboptimal performance may then result, since the designer may think that the best performance has been reached.

Fortunately, the noisy estimate produced by the LMS rule, which we called a nuisance before (rattling, see Chapter 1), becomes an advantage. In fact, the noisy gradient increases the chance of escaping both local minima and flat spots. It is obvious that the control of the adaptation algorithm becomes much more delicate in nonconvex performance surfaces. If the gradient search becomes less robust, a fair question is: Why nonlinear PEs?

Let us go to NeuroSolutions to verify this behavior.

NEUROSOLUTIONS EXAMPLE 3.7

Comparing a Linear and Nonlinear PE for Classification

Let us use the same problem as in the previous example, and simply change the PE from a Logistic Axon to a Bias Axon, which simply adds the contributions of the weights plus a bias. Running the simulations, we see that the discrimination probe shows a separation line passing between the classes of points, as in the nonlinear case. The minimum error is 0.011276 and cannot be decreased further, since the output must be a linear combination of the inputs, that is, all the possible outputs must exist in a plane. This should be compared with the nonlinear solution that is able to decrease the error further by operating nonlinearly on the inputs (a zero error is possible). This effectively corresponds to "bending" the regression plane to better fit the 0 and 1 responses in the two half planes.

It is also interesting to verify that the linear system is much more sensitive to the learning rates than its nonlinear counterpart. Explore the breadboard by changing the learning rates, and observe the different speeds at which the regression plane moves to its final position.

NeuroSolutions Example

Go to next section


 3.3 The Perceptron

Rosenblatt's perceptron is a pattern-recognition machine that was invented in the 1950s for optical character recognition. The perceptron has multiple inputs fully connected to an output layer with multiple McCulloch-Pitts PEs (Figure 3-8). Each input xj is multiplied by an adjustable constant wij (the weight) before being fed to the ith processing element in the output layer, yielding

(3.18)

where bi is the bias for each PE. The number of outputs is normally determined by the number of classes in the data. These PEs add the individual scaled contributions and respond to the entire input space.

Figure 3-8 The perceptron with D inputs and M outputs (D-M)

After studying the function of each M-P PE, we are ready to understand the pattern-recognition power of the perceptron. The M-P PE is restricted to classifying only two classes. The more general problem is to classify the input as one of M classes (see Figure 3-9). The topology has to be modified to include a layer of M M-P PEs so that each one is able to create its own linear discriminant function in a D-dimensional space. The advantage of having multiple PEs versus the single M-P PE is the ability to tune each PE to respond maximally to a region in the input space. Each PE is essentially deciding if the input is in class i or not in class i (e.g. all other classes).

One of Rosenblatt's theoretical achievements was the demonstration that the perceptron could be trained with Eq. 3.9 to recognize linearly separable patterns in a finite number of steps. This showed not only that these adaptive devices produce useful pattern classification by tweaking parameters, but that there are systematic algorithms to change the weights that converge in a finite number of steps. These algorithms are called learning rules. The perceptron also had a remarkable property: It was able to generalize. Hence, we can say that the perceptron started the field of learning theory. The recent interest in large margin classifiers shows that the perceptron and its algorithm are still at the center stage in designing practical classifiers.

3.3.1 Decision Surfaces of the Perceptron

An M-output perceptron can divide the pattern space into M distinct regions. Suppose that the ith and jth regions share a common boundary. The decision surface is a segment of a linear surface of equation gi(x) [image] gj(x) = 0. There are M(M-1)/2 such equations and so the decision surfaces of a perceptron are segments of at most the same number of hyperplanes. The hyperplanes that contribute to the definition of the decision boundary must be contiguous (the others are called redundant.) The decision regions of the perceptron are always convex regions, because during training we require that one and only one of the outputs be positive. When a PE responds maximally to an input pattern, it means that the input is inside the region represented by the PE. Hence each PE identifies patterns that belong to a class. The perceptron is therefore a physical implementation of the linear pattern-recognition machine presented in Figure 2-12 .

Figure 3-9 Decision surface for a three output perceptron

NEUROSOLUTIONS EXAMPLE 3.8

Decision Boundaries of the Perceptron

This example shows the discriminant function and the decision boundary created by the perceptron for a three-class problem in 2-D space. The data that created the example is shown in Figure 3-9. What we want to stress is the convex shape of each decision region. This is a characteristic of a single-layer network. Notice that there are areas in the space that have no class assigned to them.

NeuroSolutions Example

3.3.2 Delta Rule Applied to the Perceptron

What changes in the delta rule when we go from a single PE to the perceptron? Not much, except that now there are several (M) outputs, so the cost of Eq. 3.8. must be computed not only as a sum over the training set but also as a summation of each output PE. Thus J becomes

(3.19)

where p is the index over the patterns and i over the output PEs. We can rewrite Eq.3.15 to adapt the j weight of the ith PE as

(3.20)

and the update rule for the weights would be the same as Eq. 3.16. Note that the gradient of the cost with respect to the weight wij is computed by multiplying the partial derivative of the cost with respect to the PE state [image] scaled by the derivative of the nonlinearity of the PE and the input activation. Let us define the local error [image] i for the ith PE as

(3.21)

We can then conclude that the weight update using the delta rule is

(3.22)

which are local quantities available at the weight, that is, the activation xjp that reaches the weight wij from the input and the local error [image] ip propagated from the cost. The amazing thing about this algorithm is that it is local to the weight. Only the local error [image] i and the local activation xj are needed to update a particular weight. This means that it is immaterial how many PEs the net has and how complex their interconnection is. The training algorithm can concentrate on each PE individually and work only with the local error and local activation.

NEUROSOLUTIONS EXAMPLE 3.9

The Perceptron for Character Recognition

This NeuroSolutions breadboard implements Rosenblatt's perceptron with two minor modifications: Instead of using threshold PEs, it implements the classifier with tanh nonlinearities. This allows us to use the delta rule to train the machine instead of the perceptron learning algorithm developed by Rosenblatt.

The task is character recognition. We created 8-pixel-by-8-pixel black-and-white images of the 10 digits (0-9). We have placed an Image Viewer on the Input Axon so you can view the images. The output layer is made up of 10 PEs, one for each of the digits. Since we are going to use the delta rule, we placed an L2 Criterion at the output and included an output file with the desired response, which is a value of 1 for the corresponding digit and 0 for the others. We will place a Bar Graph probe over the output file to show the net response for each input pattern.

Training will be done using on-line learning, that is, the weights are modified after the presentation of each pattern. Learning rates are appropriately set for the task (more about this later). We will place a Mega Scope at the cost access of the L2 Criterion to display the change of the MSE versus the iteration number, producing our learning curve.

What do you expect to see? In the beginning the weights are random, so any combination of outputs can appear, which produces a large error. But after a few iterations the weights are modified to track the desired response. In the Bar Chart, the largest response scrolls from top to bottom meaning that only one large output is present and that it matches the digits (presented in sequence 0-9). When the network is trained, the output follows the desired response.

It is remarkable that 640 weights (the Synapse Inspector displays the number of weights) are trained so quickly. In practice, however, the problem of digit recognition is much more difficult because the characters differ from person to person; they can appear concatenated, and misaligned.

It is interesting to find out how robust the obtained solution is. This can be measured by altering the input with the weights fixed and finding the network response. We implement this modification by placing a Noise generator over the input file. The network does not classify noisy inputs very well. The inputs are very different from the clean, noiseless images used for training. An interesting question is. Can we improve the robustness of the classification if we include noise during the training? This can be easily checked with the setup. The initial condition is the weights obtained without noise. We will continue train the network with the new, noisy data and see whether it can improve its performance.

We see that the error decreases further, albeit in an erratic way. This means that the network has learned to cope with small amounts of noise. The outputs look cleaner, and we can even increase the noise variance and still obtain reasonable results. This experiment shows that adding a small amount of noise to clean data sets may improve the performance when noise is present. However, the method requires tight control of the experimentation to produce good results.

We ask you to single-step through the training to watch the input and output of the system for each data point. This is done by clicking the step exemplar button on the NeuroSolutions Control Palette. At the end of a run the network may not allow you to single-step the network since you have completed the experiment. You can fix this situation by either clicking on the Zero Counter button or by increasing the number of training epochs in the Controller Inspector panel.

Remember that this is a live breadboard, so you can experiment with the breadboard at will. We recommend that you change the learning rates, the noise source variance, use other probes such as the Hinton probe to display the weight values, and so on.

NeuroSolutions Example

3.3.3 Large-Margin Perceptrons

We have seen that the perceptron learning rule is very efficient but not very effective because the movement of the discriminant function stalls as soon as the last sample is classified without error. This normally leaves the separation surface very close to the last sample properly classified. Obviously, this solves the problem in the training set, but we also feel that it may not produce the best possible classification for data in the test set. This is the reason we would like to modify the perceptron training so that the location of the decision surface is placed in the valley between classes and at equal distances from the class boundaries. For this we have to introduce and define the concept of margin. Suppose we have a set of data and labels S = {(x1,d1)…,(xn,dN)}, with d = {-1,+1}, and we have a linear discriminant function defined by (w, b). We define the margin of the hyperplane to the sample set S as

(3.23)

where <.> means the inner product of x and w. The margin is finding the output of the network that is closest to zero. A network output close to zero indicates uncertainty, so large margins are desirable. We can show that the margin is related to the inverse of the L2 norm of the hyperplane's weight vector w, that is, y = 2 / [image] w [image] ² (see Vapnik).

We define the optimal hyperplane as the hyperplane that maximizes the margin between the two classes (Figure 3-10). As shown in this figure, from all the possible hyperplanes that separate the data, the optimal one is halfway between the samples that are closest to the boundary between the classes.

Figure 3-10 Hyperplane with largest margin.

Vapnik showed that the optimal hyperplane provides the smallest bound on the Vapnik-Chervonenkis (VC) dimension, which is one of the best things we can do to guarantee a low error rate in the test set. The issue is how to find this optimal hyperplane. We see from Figure 3-10 that we first have to find the points that are closest to the boundary (called the support vectors), and then place the discriminant function midway.

3.3.4 The Adatron Algorithm

Here we give a very simple algorithm known as the Adatron algorithm to find the parameters of the discriminant function that possesses the largest margin. This algorithm is sequential and is guaranteed to find the optimal solution with an exponentially fast rate of convergence.

To explain the procedure, we have to write the discriminant function of the M-P PE in terms of the data-dependent representation,

(3.24)

where <.> is the inner product, N is the number of samples, and [image] i are a set of multipliers, one for each sample. We consider the input space augmented by one dimension with a constant value of 1 to provide the bias. Let us see how to construct a topology that creates this data-dependent representation (Figure 3-11). Notice that we can implement the inner product by creating a linear network with N linear PEs where the input-layer weights are exactly the values of the training set samples, i.e. wij = xji where xji means the ith coordinate of the jth sample. The [image] i are then weights connecting the ith first-layer PEs to the output. This system creates an output that is the same as the perceptron. Note that once the training data is given, the first-layer weights are immediately fixed.

Figure 3-11 Data dependent representation for the perceptron

In this representation the perceptron learning algorithm of Eq. 3.9 would update [image] i instead of the weights when there is an error (i.e. when sign(g(xi)) ? di) and it becomes

where xi is the input of the perceptron at iteration n and [image] is the step size. The Adatron algorithm, however, chooses the alphas so that the following quadratic form is optimized:

(3.25)

This in general is a difficult optimization problem that nevertheless has a simple solution, provided that all the inner-products among the input data are calculate beforehand (as specified by the Adatron algorithm). The algorithm is as follows: Define

and choose a common starting multiplier (e.g., [image] i=0.1), learning rate [image], and a small threshold (e.g., t = 0.01). Note that we can compute g(xi) locally at each PE if dj is available at the input layer (from the data file) or, equivalently, if the weights to the hidden layer are stored after multiplication by the corresponding desired response. Then, while M > t, choose a pattern xi, and calculate an update [image] ai = [image] (1 [image] g(xi) and perform the update

Notice that for each input and corresponding desired response we can compute [image] [image] i locally, so the Adatron algorithm adheres with the local implementation constraint common in neurocomputation.

The Adatron algorithm is applied to a M-P PE, that is, it is able to only discriminate between two classes. If the problem has multiple classes, it must be solved as a sequence of two-class decisions. The algorithm resembles the perceptron learning algorithm given earlier, except in the form of the updates.

It is very useful to compare the Adatron algorithm with the delta rule previously described, to contrast their differences. In the delta rule the positioning of the boundary is primarily controlled by the samples that produce outputs that differ from the desired values of 1 or -1. These samples tend to exist in the boundary between the classes (Figure 3-12), so the MSE is largely determined by the samples that are close to the boundary between classes. However, due to the fact that J (Eq. 3.19) is a continuous function of the error, all the samples contribute somewhat to J. Therefore the MSE is a function of the full data distribution, and the location of the boundary will be affected by the shape of the data clusters.

Figure 3-12 Differences in the distribution across samples in the Adatron and delta rule

In the Adatron algorithm a very different behavior happens. During the adaptation, most of the alphas go to zero, and the location of the boundary is solely determined from a few samples close to the boundary, which are called the support vectors. The adaptation algorithm thus is insensitive to the overall shape of the data clusters and concentrates on the local neighborhood of the boundary to set the location of the hyperplane. Depending on the data distributions this may provide different boundaries. In particular, Vapnik has shown that the large-margin classifier controls generalization, while the delta rule does not (Vapnik). Let us see how the Adatron works for the problem we solved in section 3.2.3.

NEUROSOLUTIONS EXAMPLE 3.10

The Adatron Algorithm

We are going to implement in NeuroSolutions the Adatron algorithm and compare it with both the perceptron learning rule of Eq. 3.9 and the delta rule of Eq. 3.22 to solve the same classification problem. Notice that the Adatron outperforms the perceptron learning rule, providing always a decision surface midway between the classes. In that respect its behavior is similar to the delta rule. We will see later that the Adatron learning rule has some advantages with respect to MSE learning (such as the delta rule).

NeuroSolutions Example

3.3.5 Limitations of the Perceptron

As with the M-P PE, the perceptron can only solve linearly separable problems The prototype problem that is not linearly separable and thus cannot be solved by perceptrons is the exclusive-or function (XOR). The exclusive-or truth table is presented in Figure 3-13.

Figure 3-13 The XOR problem in pattern space

No matter where we place the half plane that includes the 1 responses, it will always include one of the -1 responses, so this is an example of nonlinearly separable patterns. The parity function extends the XOR to higher dimensions. NeuroSolutions can be used to demonstrate the xor problem.

NEUROSOLUTIONS EXAMPLE 3.11

Perceptron and the XOR Problem

This example shows what happens when the perceptron tries to solve the XOR problem. We start with a modified M-P PE, built with the tanh nonlinearity. The network has two inputs and one output. Training is done with the delta rule in batch mode. The Discriminant probe shows the separation surface, the Mega Scope displays the learning curve to observe how learning progresses. There are two important points about the XOR problem. First, there is no way for the M-P PE to solve the problem. The XOR is not linearly separable. Second, there are many different local minima in the performance surface, the least interesting of which is the one where all the weights quickly approach zero and the network output is thus always zero. This will produce a discriminant plot that is all gray. The other more interesting minima are when the discriminant line separates one of the four points from the other 3; this is the best the network can do. Notice that both of these minima produce the same mean squared error. Why is this so, when one network correctly classifies 3 of four points and the other produces no output at all?

With batch-mode training the weights are updated after an entire epoch. This produces a smoother weight track, but in this example, the smooth weight track very often tends to lead directly to the all-zeros (uninteresting) solution. If we change the training to on-line, the network will find other solutions. Why? Because the on-line training is much noisier than the batch-mode training, thus it has a tendency to bounce out of local minima. Imagine doing gradient descent on the performance surface in Figure 3-7; a smooth track will lead to a single minima, whereas a noisy track will bounce around and end up in various locations.

This example clearly shows that this problem has multiple minima, and depending upon the update rule some of the solutions are preferred. But none is able to correctly classify all the patterns. Notice that the inclusion of nonlinearity, even in a one-layer network, produced a performance surface that is nonconvex.

NeuroSolutions Example

Minsky showed that the perceptron was not a general-purpose processing device, because the possible decision regions that the machine can create are convex, formed through the intersection of hyperplanes (an extension of a plane to more than two dimensions).

Go to next section


 3.4 One Hidden Layer Multilayer Perceptrons

Multilayer perceptrons (MLPs) extend the perceptron with hidden layers, that is, layers of processing elements that are not connected directly to the external world. multilayer perceptrons

Figure 3-14 A multilayer perceptron with one hidden layer (D-K-M)

Figure 3-14 shows a one-hidden-layer MLP with D inputs, K hidden PEs and M outputs (MLP(D-K-M)). Normally the PEs in MLPs are nonlinear sigmoid PEs. Let us analyze in terms of discriminant functions the extra processing power of a hidden layer of nonlinear PEs.

We extend the perceptron given in Figure 3-8 by cascading one extra layer of processing elements. The hidden layer has two processing elements as shown in Figure 3-15. We start our study of the MLP with threshold processing elements. To facilitate the explanation we assume that the outputs are between 0 and 1. The goal here is to find the discriminant functions produced by one-hidden-layer MLPs.

Figure 3-15 A one hidden layer (2-2-1) perceptron

Conceptually, the one-hidden-layer MLP is a cascade of perceptrons. It is straightforward to see from this interpretation that the two PEs in the hidden layer create two linear discriminant functions in the (x1, x2) space. Let us label the output of each hidden processing element as x3 and x4. Each of these variables will be positive "above" their descriminant line with a slope given by the ratio of the respective local weights.

In the space (x3, x4), the output PE is also a perceptron. It will also construct a linear discriminant function, that tis, it will have a positive response above a straight line with slope given by -w6/w5. The problem is that we are interested in finding the overall positive response in the input space (x1,x2). This is a straightforward (but messy) problem in the composition of functions, since we know the (nonlinear) parametric relation between x3 and (x1,x2), and between x4 and (x1,x2) It is instructive to write the overall input-output map as

(3.26)

3.4.1 Discriminant Functions of the MLP

The multilayer perceptron constructs input-output mappings that are a nested composition of nonlinearities, that is, they are of the form

(3.27)

where the number of function compositions is given by the number of network layers. The resulting map is very flexible and powerful, but it is also hard to analyze. Our goal now is to find out what type of discriminant function can be created with the mapping of Eq. 3.27).

Let us assume that the output layer weights are w5=w6=1. Each expression inside the brackets creates one linear discriminant function, yielding a function with a positive value across a half plane (a step function). The location of the transition in the input space is controlled by the discriminant function, so the expression inside the brackets is the addition of two step functions, g1 and g2, with a bias term b3. In the region of the input space where both functions g1 and g2 are positive, the value of y will be the largest. The output y will have an intermediate value in a subspace where either one of the g functions is positive (but not both); finally, there is an area in the input space where y is equal to the bias b3 because each one of the functions g1 and g2 is zero.

The shapes of these areas are controlled by the placement of the original discriminant functions (which in turn are controlled by the values of w1,w2,w3,and w4 of Eq. 3.26). Notice also that the bias value b3 is added to the result of the hidden-layer partition. Its value will dictate whether only the peak value of y is positive, whether the peak and one of the plateaus are positive, whether all are positive or whether all are zero. Hence the role of the bias at the output layer is substantially different from the simple control of the y intercept as in the hidden layer PEs. The bias reveals different details of the composition of functions, effectively changing the overall assignment of values to the partition created by the hidden layer. The output weights w5 and w6 enhance the flexibility (can give different weights to the output of each hidden PE) and further change the mixture of the hidden PE activations.

The interplay among all these parameters becomes quite involved, but we have the feeling that the discriminant function of the one-hidden-layer perceptron is much more flexible than that of the perceptron. Let us go to NeuroSolutions and create a breadboard to help us get familiar with the discriminant function of the one-hidden-layer MLP. Remember that the discriminant probe provides a way to visualize the shape of the overall discriminant function by gray coding the input space with the value of the output.

NEUROSOLUTIONS EXAMPLE 3.12

One-Hidden-Layer MLP in 2-D Space

This breadboard implements the one-hidden-layer perceptron shown in Figure 3-15 with two Treshold Axons in the hidden layer but a Tanh Axon at the output. We have the Discriminant probe attached to the output of the network, thus it will show the combination of the discriminant lines created by the hidden layer-PEs. We will see how the MLP separates the input space into multiple areas with different values (depending on the values of the weights); this is often called a tessellation. Notice that since the tessellation is made up additively, a change in sign of one of the weights changes all the regions, not only the ones that belong to the PE whose weights were modified. This tight coupling between the PEs is what makes the MLP so powerful, because to obtain a given tessellation, all the PEs have to contribute to it. In other words, the training procedure for the MLP is not greedy. Another interesting aspect of the MLP is that different weight combinations can lead to the same tessellation.

NeuroSolutions Example

Figure 3-16 shows the tessellation obtained with the one-hidden-layer MLP for the parameters of the example. The conclusion is that by adding an extra layer to the perceptron we have qualitatively changed the shape of its discriminant function. The decision regions are no longer restricted to be convex, because the network has a more powerful composition mechanism.

Figure 3-16 Tessellation of a one-hidden-layer MLP (2-1) in 2-D space

There are several important conclusions to be drawn from this example.

• First, the maximum number of distinct regions of the input space is controlled by the number of hidden PEs (2 [image] for K >> D). An alternative statement is that each hidden PE creates a linear discriminant function.

• Second, the PEs on the output layer have the ability to combine some of the regions created by the hidden PEs, by either a multiplicative or additive effect. This creates decision regions that are no longer convex.

• Third, there is more than one weight combination that achieves a particular arrangement of decision regions.

Can we solve the XOR problem with the MLP (2,2,1)? The answer is YES if the discriminant functions are modified according to the goal of creating a slanted (by 45 degree) bright strip that passes through the origin (see Figure 3-13). Let us do this by hand in NeuroSolutions.

NEUROSOLUTIONS EXAMPLE 3.13

Solving the XOR Problem by Hand with the One-Hidden-Layer MLP

The input-desired response table for the XOR function is presented in Figure 3-13. Note that to solve this problem we have to define a "bright" region that includes two of the outputs only. This can be accomplished if we place two discriminant functions at 45 degrees that are parallel to each other and positive toward the center. We therefore need only two discriminant functions, that is, two hidden PEs. When you have finished the demonstration, try to find two other solutions to the XOR problem. Remember that the assignment of 1 or -1 to a given class is arbitrary, so reversing the assignments gives one more solution. The other solution is obtained with lines at 135 degrees. The important aspect of this problem is to observe how the single-layer MLP globally constructs the solution. Changing a single weight changes the overall decision surface.

NeuroSolutions Example

We conclude that classification with the MLP is still accomplished by adequately controlling the position of the discriminant functions according to the input data and the desired response. This is the same principle used for linear regression and for the perceptron. What we gain with the MLP is much more freedom in the possible shpates of discriminant functions. We will soon discuss an algorithm that will allow the machine to automatically discover the position of the discriminant that correctly classifies the training data.

3.4.2 Mapping Capabilities of the One-Hidden-Layer MLP

An important consequence of the mapping capabilities of the one-hidden-layer MLP is that it can construct a bump in the input space, that is, a single, limited-extent region of large values surrounded by a region of low values. Notice that the discriminant function of the low values is no longer convex, so the bump cannot be implemented by a perceptron. The simplest bump is triangular and it is obtained with 3 hidden PEs.

NEUROSOLUTIONS EXAMPLE 3.14

Creating a "Bump" with the One-Hidden-Layer MLP

In this example we create a bump in a two-dimensional space using a one-hidden-layer perceptron. The goal is to create a bright triangular region around the origin of the input space. In this breadboard we need three tanh PEs in the hidden layer, since a triangular region requires the combination of 3 linear discriminant functions. When you have successfully solved the problem, experiment with the breadboard, for instance resize the bump and make the boundaries crispier. This experience will come in handy in the future.

NeuroSolutions Example

This example shows that the one-hidden-layer perceptron is able to create a limited-extent (local) bump in the input space. This feature is going to be very important for function approximation in general and for classification in particular. But one-hidden-layer MLPs can also create other types of nonconvex regions, and even disjoint regions. mapping capabilities of the one-hidden-layer MLP

Another important conclusion is that if the user knows the placement of the discriminant functions, then it is possible to synthesize the solution directly. The geometric picture that we developed also applies to higher-dimensional spaces, but we lose the ability to visualize the solutions.

3.4.3 Training the One Hidden Layer MLP

The perceptron and the multilayer perceptron are trained with error-correction learning, which means that the desired response for the system must be known. In pattern recognition this is normally the case, since we have part of our input data labeled, that is, we know which data belong to which class. We already know how to train the perceptron (Eq. 3.22). If we want to utilize the delta rule, we must know how to compute the error at each PE, which requires the availability of a desired response for each network PE. An explicit error is not available in the hidden-layer PEs of the MLP. This is known as the credit assignment problem and represented the stumbling block that ended the first connectionist era around 1970.

The resurgence of interest in neural networks can be traced to the discovery of a method to train MLPs, that is best known as backpropagation or the generalized delta rule. The method was reinvented many times (inventors of backpropagation) and it was known in control theory since the late 19600s, but the connectionist version is much more efficient and can be considered a contribution of connectionism to the theory of gradient-descent learning.

We already covered one of the fundamental concepts required to extend the delta rule to MLPs when we discussed and applied the chain rule to the M-P PE. Remember that we showed that the chain rule is a systematic procedure to propagate sensitivities across an undetermined number of internal (hidden) points of a topology. As long as we have an analytic expression that relates the variables along the forward path, we can compute sensitivities to the same intermediate points in the path. If we accept that the output error should be scaled by the network weights as dictated by gradient descent, then we can propagate the output error to any point along the path and update the weights with the scaled error. In other words, we are effectively substituting the availability of the desired signal at the intermediate points in the path by the propagated and scaled output error. This worked for the M-P PE and works in general. How can this help us train the MLP?

Let us examine Figure 3-17, which depicts a piece of a one-hidden-layer MLP. The goal is to adapt the weights that are connected to the hidden ith PE. Notice that we do not have a desired response at this point, since the PE is not connected to the external world (i.e., it is hidden). To adapt its weights we are going to apply the following methodology:

  • Instead of using an explicit desired response to compute the error at the ith PE, we use an error derived from the output layer.

  • This error is the propagated and scaled output error.

  • The sensitivity is automatically computed by the chain rule.

Figure 3-17 Detail of a hidden layer network

The weight update using backpropagation (backprop derivation) is

(3.28)

Let us re-interpret Eq. 3.28 using the definition of the local error (Eq. 3.21). The summation in Eq. 3.28 is a sum of local errors [image] k at each network output PE, scaled by the weights connecting the output PEs to the ith PE. Thus the term in parenthesis in Eq. 3.28 effectively computes the total error reaching the ith PE from the output layer (which can be thought of as the ith PE's contribution to the output error). When we pass it through the ith PE nonlinearity, we have its local error, which can be written as

Thus there is a unifying link in all the gradient-descent algorithms presented so far. All the weights in gradient descent learning are updated by multiplying the local error ( [image] i(n)) by the local activation (xj(n)) according to Widrow's estimation of the instantaneous gradient first shown in the LMS rule:

(3.29)

What differs is the calculation of the local error, depending on whether the PE is linear or nonlinear and if the weight is attached to an output PE or a hidden-layer PE.

Case 1: If the PE is linear and it is at the output layer,

since f [image] (.) has a constant value of 1 This is the case found in the LMS rule (Eq. 3.11).

Case 2: If the PE is nonlinear and it is at the output, the delta rule (Eq. 3.21) used to train the perceptron is exactly Eq. 3.29 with [image] i substituted by

(3.30)

where [image] i is the error associated with the ith output. We can update the output weights of the MLP with the delta rule since we know the desired response at the output.

Case 3: If the PE is nonlinear and hidden, the local error is computed by summing all the contributions of the local errors in the output layer, scaled by the corresponding weights, which yields

(3.31)

Structurally, the weight update equations do not change, since learning is still using gradient descent. Everyone should remember Eqs. 3.30 and 3.31 to apply gradient descent learning to MLPs. Let us apply the backpropagation algorithm to train the one-hidden-layer MLP to solve the XOR problem. multilayer linear networks

NEUROSOLUTIONS EXAMPLE 3.15

Solving the XOR with Backpropagation

In this example we use the same breadboard from the previous example, set the number of hidden PEs to 2, and add the learning planes so that we can implement the backpropagation algorithm to train the network to solve the XOR problem. When you finish the demonstration, try changing the step size (learning rate) by selecting the Gradient Descent component, opening the Inspector, and typing new values. Notice how this changes the dynamics of learning, exemplified by the learning curve. Another thing that you might notice is that the learning curve has plateaus sometimes where the error remains basically the same. This corresponds to almost flat regions where the gradient is very small. We can imagine that the performance surface is no longer the simple "bowl" we found in linear regression. Sometimes instead of flat regions one gets local minima, and the search gets stuck because a local minimum and the global minimum are indistinguishable with the local gradient. This is one of the added difficulties of working with nonlinear systems.

One other thing to observe is that although the classification is successful, that is, the MSE is very small, the weights of the system have very different values from run to run. This is due to the fact that the system starts with random initial conditions and there are many possible solutions to solve the XOR problem. During adaptation the search takes the system's state along different paths and one of the solutions (we do not know which) is reached. But for classification anyone of them is OK.

NeuroSolutions Example

Notice that there are many different solutions to the XOR problem. When the network weights are seeded randomly, different solutions can be found. However, the most important conclusion is that the network can automatically find the placement of the discriminant functions that are so difficult for us to create by hand.

The other example that we will run at this point is the bump. The goal is to show that backpropagation is able to train a one-hidden-layer MLP to discover a triangularly shaped decision region. To train such an MLP, we need to construct a training set that will tell the network where its response must be 1 and where it should be 0. Here we selected 20 points in the plane. Ten points are organized in a triangle and the desired response for these samples is 1. The remaining 10 samples are placed around the first cluster, and the desired network response is 0.

NEUROSOLUTIONS EXAMPLE 3.16

Solving the Bump with Backpropagation

In this example we use the same MLP that we used previously to construct the bump by hand (3 hidden PEs) and add the learning dynamics. We have placed a Scatter Plot at the net input to help you visualize the correspondence between the samples and the discriminant function found through training. In particular, pay attention to the relationship between the discriminant function being created and the evolution of the learning curve. Depending on the initial conditions, the bright area can appear in the center, or more frequently it appears in one corner and finds its way toward the center, until it finally creates a central bright spot. Notice that the learning curves are different for each case. Sometimes you will find an initial decrease in the error followed by a long period where the error basically is constant until the final solution appears. You should link the learning curve behavior to the positioning of the discriminant function.

At other times the discriminant functions form a wedge and stay there forever. This is a local minimum. It is important to figure out why this behavior occurs. It is obvious that this solution misclassifies one sample (see the Scatter Plot), and it is also obvious that there is one extra discriminant function that does not appear in the plot. The reason is that it has parameters very close to one of the other discriminants (i.e., they are superimposed on each other). This happens when the initial conditions or the gradient-descent path produce weight updates that create hidden-layer weights that are very similar. In such cases the error produced by misclassifying a single sample is not enough to pull the hidden weights apart, and the system rests in this solution. When this happens, open the Matrix Viewers and observe the weight values.

NeuroSolutions Example

3.4.4 The Effect of the Number of Hidden PEs

One of the central issues in neurocomputing is to appropriately set the number of hidden PEs. There are two extreme cases: Either the network has too many hidden PEs to do the job, or it has too few. Understanding each case is important, because correctly setting the number of PEs is still a difficult task at our present state of knowledge. Let us start with an overdimensioned hidden layer.

NEUROSOLUTIONS EXAMPLE 3.17

Effect of the Number of Hidden PEs

In this example we experiment with the number of hidden-layer PEs and how it affects the output and learning dynamics of the network. It is obvious that if the problem can be solved with three hidden PEs, it can also be solved with 6. What happens to the positioning of the other 3 PEs' discriminant functions?

The chances of getting the correct classification in the training set increase, and the system normally requires fewer iterations to reach the solution. However, the computational burden is increased, which slows down the training. But the big problem is that the redundant PEs may have detrimental effects on the test-set performance (data that the system has not seen before), because the MLP may memorize the training data. This may create spurious areas that produce in-class responses in regions of the input space that do not contain any training data (and therefore are "don't care" regions given the training data). Putting it another way, the machine may not perform very well on data with which it was not trained. This does not happen in this case because of the symmetry of the training data, but it can happen in practice.

You should use a Matrix Viewer to observe the values of the weights from the output layer to the input. You will find that some of the output weights are small, so they are not important for the mapping. This is a benign case. The problem occurs when some of the weights are large and make the system respond to regions of the input space void of training samples. This is when the system makes mistakes in the test data. The input-layer weights also appear duplicated (i.e., several superimposed discriminant surfaces). This is computationally inefficient but does not significantly affect the performance in the test set.

NeuroSolutions Example

The other important condition to study is when the network does not have enough hidden PEs to solve the problem correctly. We saw as similar situation when we tried to solve the XOR problem with the perceptron (which effectively has zero hidden PEs). The machine does not know whether the problem is solvable or not, so it will try to do its best, classifying most of the samples correctly.

In general, the learning algorithm first finds the correct shape and placement for the discriminant functions to correctly classify the majority of the samples and then slowly moves them to classify the areas with fewer samples. The reason for this behavior is rooted in the form of the cost function (a sum of the errors over all the training patterns) that is guiding the automatic placement of the discriminant function. If the learning machine does not have enough degrees of freedom, the error will stabilize at a high value, since the system cannot solve the problem. Sometimes oscillations can occur when the learning rates are large. The oscillations correspond to sudden change in weights between two suboptimal solutions.

NEUROSOLUTIONS EXAMPLE 3.18

Fewer Hidden PEs than Required

Let us now train the MLP to create a bump, but in this case let us reduce the number of hidden PEs to 2. This MLP cannot solve the problem exactly, since at least 3 PEs are required. So what will the training do? The machine does not know whether it has enough degrees of freedom or not. It blindly changes the weights to decrease the MSE. So the solutions found will tend to correctly classify the majority of samples.

For some initial conditions we may observe an oscillation, where the error abruptly goes up and then down. During these periods the weights change rapidly, and the discriminant function oscillates between two positions. This means that the weights are driven to values that make the discriminant suddenly increase the number of errors, so the weights have to move back. One nice way to observe this effect is to place a MegaScope on top of the weights and see the weights change with each iteration.

NeuroSolutions Example

Go to next section


 3.5 MLPs with Two Hidden Layers

3.5.1 Discriminant Functions of the Two-Hidden-Layer MLP

What are the discriminant functions of two-hidden-layer perceptrons? Such a network has three levels of function composition:

We can once again study the problem by finding the decision regions created by the one-hidden-layer MLP (as in the previous section) and their composition created by the added output perceptron. This becomes more complex, but it is important to understand the basic capabilities of the overall discriminant functions.

From the previous discussion, we know that the one-hidden-layer MLP can create local regions in the input space. Another layer with several PEs can be thought of as combining bumps in disjoint regions of the space. This is a very important property, because in the theory of function approximation (we touch on this subject in Chapter 5) there are well-established theorems that state that a linear combination of localized bumps can approximate any reasonable function. Therefore an MLP with two hidden layers is also a universal approximator, that is, it can realize any input-output map, just as the one-hidden-layer MLP can.

A word of caution is in order at this point. These theorems are existence theorems (the number of PEs is unconstrained), so they do not address the engineering question, "How many PEs and layers does the MLP need to solve my problem?". This is still an open question for which experimentation is necessary. But it is extremely important to know that theoretically an MLP is a universal approximator and that one or two hidden layers are all it takes to reach this arbitrary mapping capability.

One should associate the number of PEs in the first hidden layer with the number of linear discriminant functions in the input space. One will need in general 2D hidden PEs in the first hidden layer (D is the number of dimensions in the input space) and a modified M-P PE in the second hidden layer to form a single bump. The number of PEs in the second hidden layer creates the number of bumps in the input space that are needed for the approximation. The output layer simply combines these bumps to produce the desired input-output map. This constructive reasoning shows that two-hidden-layer MLPs can approximate any function.

This view is valuable to understand the function of MLPs for classification and to appreciate the difficulty of selecting an appropriate topology. If we have some a priori knowledge of our data clusters we can judiciously set the size of the network as in the following example.

NEUROSOLUTIONS EXAMPLE 3.19

Creating a Halloween Mask by Hand with a Two-Hidden-Layer MLP

In this example we set the weights of a two-hidden-layer MLP by hand to create a Halloween mask in the input space as illustrated in Figure 3-18. This mask is a simple example of a general input-output function. It can be thought of as a classification problem where one of the classes is scattered (multimodal) in the input space (the location of the bright regions) and the other class is in the dark region.

Figure 3-18 Example of input-output mapping

We design the MLP by hand, since we know the placement of the discriminants. Observe that we have 4 distinct regions in the input space. Therefore we need 4 PEs in the second hidden layer, one to build each feature in the input space. Looking at the figure, we see that we need 13 discriminant functions: 3 for the nose, 4 for one eye, 2 for the other (we use 2 of the first), and 4 for the mouth. Figure 3-19 shows the topology of the MLP.

Figure 3-19 MLP topology (2-13-4-1)

This is a long example, but you should get a good feel for how each layer's weights and bias affect the final discriminant function or tessellation. After creating the mask, change the weights and see whether you can predict how they will affect the output.

NeuroSolutions Example

This example illustrates that when we know the decision regions, we can design the discriminant functions directly. Unfortunately, this is very rarely the case for two main reasons: The structure of our data sets may be unknown, but even if they are known, we are not able to visualize them in higher-dimensional spaces so that we can specify the placements of the discriminant functions. We therefore need to resort to training for the adjustment of the weights.

3.5.2 Training Two-Hidden-Layer MLPs with Backpropagation

The beauty of training using the backpropagation algorithm is that it is a systematic, step-by-step procedure that can be applied independent of the topology of the network and the input dimensionality. Larger, more complex networks may require more training time, but backpropagation is still fully capable of training them.

We can expect very similar behavior between the training of the one-hidden-layer and two-hidden-layer MLPs, that is, there are some solutions that take a long time to reach, if they are reached at all, due to the existence of local minima and saddle points.

NEUROSOLUTIONS EXAMPLE 3.20

Comparison of the Perceptron and MLPs

In this example we summarize the network topologies we have discussed so far by trying to solve the male/female classification problem using a single perceptron, one hidden layer MLP, and two hidden layer MLP. The male/female classification problem was discussed in Chapter 2 and involves trying to determine whether a subject is male or female based on height and weight. Please refer to Figures 2-5 and 2-8. As you can tell, there is no way to correctly classify all the data points. Remember that the optimal discriminant is quadratic. How will this affect the best choice of network topologies?

We have added a new probe in this example, the Confusion probe. The Confusion probe generates a confusion matrix, which is a simple methodology to display the classification results of a network. The confusion matrix is defined by labeling the desired classification on the rows and the predicted classifications on the columns (or vice-versa). For each exemplar, a 1 is added to the cell entry defined by (desired classification, predicted classification). Since we want the predicted classification to be the same as the desired classification, the ideal situation is to have all the exemplars end up on the diagonal cells of the matrix. Two confusion matrices follow:

Example 1

Example 2

In example 1 we have perfect classification. Every male subject was classified by the network as male, and every female subject was classified as female. There were no males classified as females or vice versa. In example 2 we have imperfect classification. We have 9 females classified incorrectly by the network as males and 5 males classified as females.

One of the interesting things about this example is that the error sometimes increases during adaptation. This clearly shows that the weight tracks are not moving exactly along the gradient-descent direction. This may happen because we are using a local estimate of the gradient that may be noisy. The second thing to note is the relationship between the number of classification errors and the MSE. They are related in a coarse manner, that is, in the beginning of training both are high and they decrease. However, the MSE may be decreasing, but the number of misclassifications can be stable or even increase. This is due to the fact that the MSE is sensitive to the differences between the desired responses and the actual output, while the number of mistakes is a digital quantity that looks only at the largest output (here the largest output is assigned to the class).

NeuroSolutions Example

In the previous example the one-hidden-layer MLP and the two-hidden-layer performed at the same level of accuracy. But sometimes there are problems for which the two-hidden-layer MLP has an advantage. The fact is that the two-hidden-layer machines can form all of the discriminant functions of the one-hidden-layer MLP and many others, so they are more versatile, although we know that asymptotically a very large one hidden layer MLP would approximate the same performance. As a rule of thumb, we should always start our experiments with the simpler topology, since the two-hidden-layer MLP normally trains more slowly than the one-hidden-layer MLP due to the attenuation of errors through the nonlinearities.

NEUROSOLUTIONS EXAMPLE 3.21

Solving the Bow Tie with MLPs

This example solves a problem that a one-hidden-layer MLP can only approximate, but that a two-hidden-layer machine can solve exactly. The class distribution in 2-D space looks like a bow tie, so we will call it the bow tie problem. We created the two-class problem with the distribution shown in the figure below.

Let us experiment with the three topologies for this case (the perceptron and the one-hidden and the two-hidden-layer perceptrons). Modify the number of hidden-layer PEs and watch the confusion matrix. You will see that the two-hidden-layer MLP gives a lower error for this problem, but if you increase the number of hidden layer PEs, the one-hidden-layer MLP performance differential disappears.

NeuroSolutions Exaumple

Go to next section


 3.6 Training Static Networks with the Backpropagation Procedure

We saw that a two-hidden-layer MLP can be trained with the backpropagation algorithm derived initally for the one-hidden-layer MLP. The backpropagation algorithm can be applied without any modification to the-two-hidden-layer MLP, or for that matter to Any feedforward topology. This is because Eq. 3.31 applies locally to each hidden PE, no matter where it is placed in the topology.

The only fact to remember is that we have to start computing the local errors from the output of the network toward the input, that is, there is an intrinsic flow in the backpropagation algorithm. First, one data sample is sent through the network to find an output and compute an error (Figure 3-20). Then we calculate the injected error at the output layer and reflect it to the input of the output PE (the [image] k in Eq. 3.29). Then the errors in the previous layer can be computed by Eq.3.31, and so on, until we reach the input layer. During the backpropagation of errors the old weights are used. Once all the local errors are found, Eq. 3.16 is used to compute the output weight updates, and Eq. 3.28 is used to compute the hidden-layer weight updates.

Figure 3-20 Chaining of operations in the backpropagation algorithm

Although these general rules for computing the gradients seem pretty reasonable from our presentation, we have mostly focused on the local computations of the gradients, and we do not have a clear idea of the possible limitations of the technique, if any. To answer this question, a more rigorous approach to studying gradient calculations in distributed architectures seems necessary.

3.6.1 Gradient Computation and Ordered Networks

We will briefly study the gradient computations in ordered networks. An ordered network is a network where the state variables can be computed one at a time in a specified order. An MLP is such a network. Normally, each PE (and weight) is numbered, starting from the input (left) to the output (right) as in Figure 3-21.

Figure 3-21 Simple three-PE ordered network

For simplicity, assume the PEs are linear and that we want to compute the total partial derivative of [image]. In ordered networks there are two contributions to this derivative: an explicit or direct dependence and an implicit or indirect dependence through the network. This total partial derivative is called an ordered derivative. The direct dependence will be denoted with a superscript [image]. In the network of Figure 3-21, we have

(3.32)

The output of a high-numbered PE (ith) in an ordered network with N PEs requires the computation of the output of All lower-numbered PEs (jth), that is,

(3.33)

where f(.) is the static nonlinearity, and xi is the PE input (if the PE is an input PE). Note that the summation index enforces i > j, which implies a feedforward topology. This is exactly what happens in the MLP. The intrinsic dependence in the computations can be captured in an ordered list L:

(3.34)

where wij are the weights and yi the PE activations (the state variables). What this means is that variable yi depends only on the variables yj that are located to its left in the ordered list, that is, i > j. Since the activations y are a function of the weights, these have to appear first in the list.

Let us define the performance function J(y1,...yN) for this network as Eq. 3.19. Werbos proved that in ordered networks the ordered partial derivatives of J with respect to the states y can be computed by

(3.35)

This expression states that the ordered derivatives can be composed from the explicit influence (first term) and the implicit effect through the topology (the sum). Note that the gradient computation must be ordered from high indices to low indices, that is, in the reverse order of the ordered list. This is a clean mathematical proof that the computation of the gradients must be done in the way we described, that is, from the output layers to the input. Hence the name backpropagation for the procedure.

Since the weights are first in the list, the sensitivity with respect to the weights is computed across all the network states

(3.36)

Note the similarity with backpropagation: Eq. 3.35 is computing the backpropagated error, while Eq. 3.36 composes it with the local sensitivity of the state with respect to the weight and yields the local contribution for the gradient (see rederivation of backpropagation with ordered derivatives). What we have gained with this analysis is insight on the requirements to apply backpropagation and on the characteristics of the method. We will address two aspects: the computational complexity of the method, and efficient implementations.

3.6.2 Computation Complexity

The backpropagation algorithm derived for neural networks is a very powerful and unique method that can also be applied to the larger field of gradient descent learning. To fully appreciate this point, we have to ask the question, How were gradients computed traditionally? The fields of control theory and digital signal processing have addressed the same problem long ago.

They used what is called the direct differentiation method to compute gradients. The equations are simple to derive for the MLP. Suppose that we want to minimize the cost given by Eq. 3.19 with the forward structure of Eq. 3.33 with N PEs. Applying the chain rule to Eq. 3.19 we get

(3.37)

Let us define the gradient variable

(3.38)

where the superscript k refers to the [image] state variable. The gradient variable can be computed by differentiating the state equation Eq. 3.33 to yield

(3.39)

where f [image] denotes the derivative and [image] ik is the kronecker delta that is equal to one only when k = i and zero otherwise. This expression applies to feedforward topologies such as the MLP.

This method is straightforward to implement, but it is computationally demanding. It basically says that one needs to compute the sensitivity [image] of every state to every weight (Eq. 3.38). Since in an N PE fully connected MLP we have M > N weights, this gives MN quantities. Eq. 3.39 also shows that for each gradient variable we need a constant number of multiplications, so, the end result is a computational complexity proportional to MN that we will denote by O(MN). The storage required for the algorithm is dominated by the storage of the gradient variables, which is O(MN).

Now let us analyze the backpropagation procedure. The backpropagation procedure is O(N²), since for an N PE net we have to compute N errors [image], and each needs N multiplications (Eq. 3.35). This is the same complexity of the forward path (Eq. 3.33), so the asymptotic complexity of the backpropagation algorithm is O(N²). This should be compared with O(MN) for the direct method. Whenever the network has more weights than PEs (which is normally the case with MLPs), the direct procedure is more expensive computationally. In terms of storage the backpropagation algorithm is also more efficient than the direct computation.

The savings come from the use of the topology to compute the weight updates. However, this also brings a shortcoming that connectionism has not fully coped with, namely, the need to rederive the learning equations for each new topology. We will show later how to implement a topology-independent learning algorithm in the simulations if the principles embodied in the ordered derivatives are fully exploited.

3.6.3 Interconnection Algorithm for Backpropagation

The local nature of the backpropagation algorithm has very deep and important implications for simulation in digital computers. Unfortunately, it is not widely used in neurocomputing, so we discuss here a local implementation of backpropagation that requires knowledge only of the topology and of the PE input-output map. We call this implementation the interconnection algorithm because it uses the PE interconnection to specify the flow of the calculations.

For convenience, let us rewrite the activation equation and the error equations:

Analyzing these equations closely, we can conclude that the expression that computes the local error in the internal PEs is tightly coupled with the original network topology. In the forward pass the network works with the input data x and produces activations y. The backward equation works with the injected error [image] [image] and produces errors e. Furthermore, note that the equations become essentially the same if wij is substituted by wji. This indicates that once the topology is known, both the forward and backward equations can be automatically computed. Figure 3-22 presents this similarity in more detail. In the top part we show the original neural network around the ith PE, while in the bottom part of the figure we are implementing a network that realizes the backward equation. The network of Figure 3-22b is called the dual (or transpose) of the network of Figure 3-22(a), and is constructed with the following rules.

At the ith PE the flow of activations (xi) in the original neural topology is from left to right, while in the topology that computes the error ( [image] i) it is from right to left, that is, inputs become outputs and outputs become inputs. Note also that the summing junctions in Figure 3-22.a become splitting nodes in Figure 3-22b, and splitting nodes become summing junctions. The weights keep the same values. The incoming error in the dual network is multiplied by f [image] (neti) to produce [image] i.

Figure 3-22 a) Original and (b) dual networks

The conclusion of this observation is that we can imagine that the injected error [image] i created by the difference between the network output and the desired response flows into the dual topology. Collecting the local error at each PE in the dual network is equivalent to deriving and computing the messy Eq. 3.31. Note also that in the dual topology the value of f [image] (.) is computed at neti, the activation level of the corresponding PE, that is, it corresponds to a linearization of the forward network at the operating point. This leads to the following alternative description of backpropagation (Figure 3-23).

The procedure starts by inputting data to the neural network and obtaining the local activation at every PE (step 1). Then the network output is computed and compared with the desired response to obtain the output error, using the appropriate error criterion (step 2). This error is injected through the dual network, and a local error is obtained at each PE (step 3). Note that the error in the dual network is scaled at each node by the derivative of the nonlinearity at the operating point (given by the activation in the original ANN). Now that we have the local error and the local activation we can again apply Eq. 3.29 to compute all weight updates (step 4).

Figure 3-23 The algorithm for backpropagation with the dual network

The beauty of this arrangement that we call the interconnection algorithm for backpropagation is twofold: first, the data flow has been separated from the local computations, which brings flexibility for the simulations. Later chapters demonstrate that this gradient-descent methodology is also valid for training recurrent networks and networks through time, that is, the interconnection algorithm is a general implementation of gradient-descent learning.

Second, the local error is available as a signal at the PEs of the dual topology, which means that we do not need to write equations to compute the local error, the biggest problem when simulating arbitrary neural networks with backpropagation. Only the topology of the network needs to be specified by the user. The flow of errors through the dual network topology is doing the backpropagation computations for us effortlessly. Implementing the backpropagation algorithm with the dual network is much more versatile than directly coding the previous equations, since the dual network can be programmed very simply from the user's specified topology. NeuroSolutions uses the interconnection algorithm presented in Figure 3-23. Next, we specify the operations required locally to implement the interconnection algorithm.

3.6.4 Specification of the Local Operations

The interconnection algorithm can be programmed in a data-flow fashion that is called a data flow machine. Analyzing Figure 3-23 we can also see that the types of PE enters only in the choice of the local computation (f(.) and its derivative in the dual). What this means in practice is that the flow of signals through the network (i.e., the activations and the errors) can be decoupled from the operations of the PEs. This provides a very powerful way to simulate neural networks, because different PEs can be interchanged at will in the topology simply by specifying new local operations that each building block must execute. We therefore do not need to derive learning equations for each topology. Only two items need to be specified:

  • The network topology (preferably in a graphical form) by numbering PEs in a left-to-right manner. A computer program can easily produce an ordered list of interconnections.

  • The PE and the dual PE input output relations, which we call the local maps.

To be part of the data-flow machine, the ith PE must be programmed to do basically two things when it is activated (fired) by the interconnection algorithm:

  • Propagate an activation forward (forward equation)

(3.40)

  • Propagate an error backward (backward equation)

(3.41)

Both Eq 3.40 and 3.41 are required because the ith PE is part of a larger network. Thus, the PE needs (xi, [image] I) not only to update its weights (with the gradient-descent rule), but also to pass xi forward to continue the chain and to pass [image] i backward so that the dual PEs in the preceding layer can perform adaptation of their own parameters. Figure 3-24 shows the mechanics of the method.

Finally, the weight updates are computed from (xi, [image] i) with the gradient search method of choice. Until now we discussed only the straight gradient-descent algorithm so this simply means [image] wij = xj [image] [image]. But in general for first-order search methods the update is a function h(.) of the activation and error

(3.42)

as we will discuss in Chapter 4. Different PE types (logistic, tanh, linear, quasi-linear, etc.) simply require different local maps for the PE and its dual. This means that PEs can be effectively organized into families of components using an object-oriented programming approach, as exploited in NeuroSolutions. Moreover, each PE can be associated with an icon that leads to a very effective graphical user interface to construct neural networks. Remember that the forward PE and its dual share the same weight values wij.

Figure 3-24 Communications among PEs in the topology

NEUROSOLUTIONS EXAMPLE 3.22

Data-flow Implementation of Backpropagation

This explanation needs to be compared with the iconic representation of the breadboard that implements the MLP. Note that the breadboard has three planes: the large-bubbles layer (the forward network, also called the forward plane) that is computing the forward activation (Eq. 3.40); the smaller-bubbles layer (the dual network, also called the backplane) that is computing local errors (Eq. 3.41), and the gradient-descent components (the search plane) that use the local activations and local sensitivities to compute the weight updates (Eq. 3.42).

The data flow is implemented by the controllers that normally sit at the top left of the network. To implement backpropagation, there are two controllers: the Forward Static Controller (which sends input data forward through the network), and the Back Static Controller (which sends sensitivities through the dual network). The alternation between these controllers implements the data flow necessary to update the weights using backpropagation. These controllers are crucial to decide what type of backpropagation is implemented (static, fixed-point learning, or backpropagation through time), as we will see later.

The data flow still needs to be further specified (remember that backpropagation implicitly imposes a dataflow). You can send one sample at a time through the network, compute the corresponding local error, and perform the adaptation. This is called the on-line or pattern-mode learning. An alternative is to fire all the samples of the training set. Store all the PE activations locally. Compute the network outputs for every input. Compare and compute the errors at the network output for each input desired response pair, and send all the output errors in sequence through the dual network. Now you can compute the weight update for every sample, sum them up, and only then change the weights. This is called the batch-learning mode, where the weights are always updated with the information contained in the full training set.

To demonstrate how easy it is to construct the dual network, the Back Controller has a switch to automatically construct the backpropagation plane. If the Remove button is pressed, the backpropagation plane is deleted. The network now has only the forward dynamics, so it can not learn. This is the way the network should be used for testing. Freeing the back plane is preferable to setting the learning rates to zero because it is more efficient (no sensitivities are ever calculated).

Now if the Allocate button is pressed, the back plane and the gradient plane are automatically constructed. The network will be able to learn again. Notice that with the arrangement of the simulations in planes, the user only has to worry about the construction of the forward plane, that is, the network topology. Once this is done, NeuroSolutions has the information to automatically construct the learning dynamics. The biggest advantage of this arrangement is the ability to create user-defined topologies, free of the conventional constraints. NeuroSolutions is always able to compute the dual network and train the weights with backpropagation. No learning equations are ever explicitly written.

NeuroSolutions Example

Go to next section


 3.7 Training Embedded Adaptive Systems

Backpropagation is able to compute gradients in ordered networks with smooth nonlinearities. The algorithm however, can be applied to any system built from modules that are differentiable (not necessarily adaptive) and that can be ordered. This is indeed a very large class of systems that includes macroeconomical, life science, and engineering models. The beauty of the procedure is that we can propagate sensitivities up and down the modules with the goal of finding optimal coefficients that meet a given external criterion.

This is to say that we can mix adaptive and fixed-parameter subsystems and with backpropagation seek an overall optimal operating point. This is particularly important when we have a priori knowledge about the problem and we want to design subsystems that include that knowledge but have other subsystems that are adaptive. Even if only some of the modules are adaptive, we can propagate sensitivities through the ones with fixed parameters to optimally train the overall system. We use this property later in control applications.

Here we would like to treat the case of Figure 3-25, where an adaptive submodule of a system needs to be adapted, given a desired external response. Notice that the adaptive module is internal, that is, it is not in direct contact with either the input or the desired signal. Can we optimally set the parameters of this system? The answer is "yes", provided that submodule 2 is differentiable and the overall network is ordered.

Figure 3-25 Adapting an embedded adaptive system

Using our known tools of the chain rule and ordered derivatives, we can see that this adaptation problem can be easily solved, taking into consideration that

(3.43)

The first term is the injected error, while the last is the term that backpropagation computes for us. If we know the functional form of the input-output relation for module 2, we can calculate the effect of passing the injected error through module 2. We still have a small problem in computing the weight update using gradient descent, since we have to know the input to the MLP, but this is straightforward if we know the input-output relation of module 1. Of course, if the modules are themselves nonlinear, they may complicate the training because they attenuate errors and activations, but this does not affect the applicability of the method.

This analysis shows that we can adapt easily the parameters of an embedded adaptive system in larger systems built from fixed coefficients and differentiable input-output relationships. We can design the subsystem to optimize the overall system performance. This is a very important aspect of backpropagation that is today still largely unexplored.

NEUROSOLUTIONS EXAMPLE 3.23

Training Embedded Neural Networks with Backpropagation

In this example we adapt an embedded adaptive system (MLP) in a larger system built from a front end fuzzy module with an output that goes through a nonlinear subsystem before it is compared with the desired response.

The goal of this example is to create an intelligent "bell-ringer" machine at a fair. The contestant uses a sledgehammer to hit a lever that projects an object up a column in the hope of ringing the bell at the top. Our goal is to make the device more fair by adjusting the height of the bell based upon the size and weight of the contestant. But since we do not have a scale or a measuring device, we would like to use the quantifiers light, medium and heavy for the weight, and short, medium and tall for the height. We want to train an MLP to move the bell based upon the "fuzzy" inputs to a position where the average person of that height and weight can ring it. The difficulty is that the pneumatic machinery used to move the bell has a discontinuous and nonlinear transfer function. The height of the bell is proportional to the cube of the control input (MLP output). In addition, if the control input is above 0.3, the machinery is warped out of calibration and the bell height must also be adjusted by adding 0.16 plus the square root of the already cubed input. These operations are shown in Figure 3-26. Given a set of heights, weights, and the distance the object has traveled up the "bell-ringer" column, can we still train the MLP?

Figure 3-26 System block diagram

We create the linguistic variables light, medium, and so on from the height and weight using fuzzifiers. The fuzzy module is built from sigmoid nonlinearities. The advantage of a fuzzy layer is that it includes in a very effective way the a priori knowledge that we may have about the task. Fuzzy sets are described by a membership function (MF), so the first problem is to devise a way to create membership functions in NeuroSolutions. This is not difficult because we can very easily create an MF by a cascade of one sigmoid layer with a linear layer with preset parameters (which may later be fine-tuned through adaptation). As an example see Figure 3-27.

Figure 3-27 Creation of a fuzzy layer with sigmoids

Pairs of sigmoids combined with +1 and -1 create a Gaussian like membership function (Fig. 3-27). The maximum is located at the average value of the bias. Several of these paired PEs with different biases span the input space with the linguistic variables. The knowledge from the tasks sets the bias and the slope of each sigmoid such that the required membership functions are constructed. Notice that there are no adaptable parameters in this layer. Open the Matrix Editors in the breadboard to see how we have created this layer. As usual you are free to modify any parameter.

The next submodule is the one-hidden-layer MLP which is followed by a second module that adjusts the height of the bell shaped curve. In this Example, module 2 implements the function

where z is the output of the MLP. This function is coded in the DLL at the output of the MLP. Notice that the Dual component has to implement the dual of the function. NeuroSolutions and its dataflow implementation of backpropagation is capable of training the MLP, with the activations and sensitivities being passed through the fuzzy layer and the output function. Run the system and verify that the MLP can still optimize the overall system.

NeuroSolutions Example

Feedforward neural networks are an example of ordered networks. But notice that the list of dependencies in Eq. 3.34 is static in the sense that it does not depend on time. When time dependencies are brought into the picture, as will be done in later chapters, the ordered list of dependencies must be revisited to apply backpropagation. Another requirement to apply this methodology is to choose smooth nonlinearities for the PEs, as was mentioned previously in several occasions. One aspect that should be stressed again is that backpropagation is computationally more efficient than direct differentiation to compute gradients in feedforward networks.

Go to next section


 3.8 MLPs as Optimal Classifiers

Before concluding this chapter, let us go back to the framework of statistical pattern recognition and ask the question: Can MLPs in fact implement optimal classifiers? An optimal classifier must have the potential to create arbitrary discriminant functions that separate data clusters according to the a posteriori probability. Since we know that the MLP has known universal mapping capabilities, we suspect that the MLP meets this requirement. An optimal classifier using the Bayes' framework should produce outputs that are the a posteriori probability of the class given the data.

Does the MLP produce outputs that can be interpreted as probabilities? The answer to this question is yes if the training is done under certain conditions (Bishop). Moreover, we can also show that the MLP is directly producing estimates of a posteriori probabilities, unlike any of the classical methods of pattern recognition. Remember that in statistical pattern recognition we needed to use Bayes' theorem to practically evaluate the a posteriori probabilities. With the MLP we obtain their estimates directly as outputs, provided that the training and the topology are appropriately specified. This is a departure from the well-established statistical reasoning that applies Bayes' rule to estimate a posteriori probabilities. Let us briefly cover the basic concepts of this theory. Assume the learning machine is being trained to minimize the MSE (Figure 3-28).

Figure 3-28 A learning machine whose outputs are estimates of a posteriori probabilities

We have to assume that the MLP has a sufficient number of PEs to produce the required map from input space to targets. We also have to assume that the training data is sufficient and that the training does indeed take the learning system to the global minimum. The final requirement is that the outputs are between 0 and 1 and that they all sum to 1 for every input pattern (so that each output can represent the probability that the input is in the specified class). To guarantee that the outputs sum to 1, we cannot utilize the logistic function PE. We must utilize a new output axon with the softmax activation function

(3.44)

where the denominator sums over all network outputs. The softmax function is similar to the tanh and logistic functions, except that the outputs are scaled by the total activation at the output layer (so that the sum of the outputs is 1). For the two-class case the single output PE can be a logistic function, since the probability requirements are still met The probability of class 1 is the value of the output PE, and the probability of class 2 is 1 minus the value of the output PE.

Notice that we did not specify that the learning machine must be an MLP. The MLP is just an example of a viable and efficient implementation that produces this result since it is a universal mapper. The important aspect for this behavior is the minimization of the MSE.

The output yk of such an MLP can be shown to be an estimator for the average conditional probability of the target data given the inputs, that is,

(3.45)

where w* is the optimal weight value and we are using t for the desired response (derivation of the conditional average). For a classification problem where the desired response is 1 and 0 and we assume c outputs (one output per class), it is easy to show that we have

(3.46)

What this equation says is that the output of the MLP is providing the a posteriori probability of the class given the data. Recall from the pattern-recognition chapter that the a posteriori probabilities minimize the classification error and are therefore the best one can hope for in building optimal classifiers. We can thus conclude that if the learning machine is powerful enough and has been trained appropriately, its outputs (with the restrictions previously mentioned) can be interpreted as a posteriori probabilities of the classes given the data. We now have a methodology to estimate a posteriori probabilities directly from the data, unlike when we use statistical pattern recognition, where Bayes' rule is conventionally applied.

This is a very important result because it allows us to work with the numerical outputs of the network as a posteriori probabilities for a variety of applications in both pattern recognition and signal processing. Here are some of these applications:

• Applying rejection thresholds for decision making

• Implementing minimum risk decisions in detection and diagnostic

• Estimating observation probabilities in a variety of applications (hidden Markov models, statistical signal processing)

We will address these aspects further in later chapters.

NEUROSOLUTIONS EXAMPLE 3.24

Softmax and A Posteriori Probabilities

This example uses the preceding probability considerations to show how an MLP can be used to estimate the a posteriori probabilities for the healthy-sick example discussed previously. We will train a one-hidden-layer MLP with two Softmax PEs. The Softmax PEs are similar to Sigmoid Axons except they are normalized so that the sum of the outputs is always one. The softmax function allows the outputs of the MLP to be considered as a posteriori estimates of the probability that the input exemplar belongs to each class. For example, if the output from PE 1 is 0.90 and the output from PE 2 is 0.10, then the probability that the subject is healthy is 90 percent and the probability that the subject is sick is 10 percent.

To check how well this really works, we need quantitative data regarding the class distributions, so we created a two class problem where each class is composed by two sets of Gaussian distributed data. The a posteriori probability is shown in the following figure.

Red (represented by the dashed line) is the a posteriori probability of class 1, while black is the a posteriori probability of class 2. Run the network and observe how similar the discriminant functions of the MLP are to this plot. This is a more specific example of the important optimality property of MLPs. Change the number of PEs to test what the net does when it does not have enough degrees of freedom. Let it train for a long time and compare the final separation surface with the preceding picture. Overtraining makes the separation surface go to ±1, and destroys the statistical interpretation.

NeuroSolutions Example

Go to next section


 3.9 Conclusions

In this chapter we covered one of the most important applications of neural networks: pattern recognition. ANNs are semiparametric classifiers since the discriminant functions are functions that belong to a given class. But we do not know a priori the actual discriminants that will be employed by the ANN.

In this chapter we studied multilayer perceptrons (MLPs), which are feedforward topologies. The topology is what defines the functions that can be used for discriminants. The perceptron can construct only linear discriminants, but the two-hidden-layer MLP is a universal approximator, that is, it can construct arbitrarily complex input-output mappings. MLPs are very efficient approximators in high-dimensional spaces.

MLPs are trained with a gradient-descent procedure called backpropagation. Backpropagation is a very powerful and computationally efficient algorithm. It is, in fact, a contribution of the field of neural networks to optimization theory. We have shown how the algorithm is derived using the chain rule, and we also covered the ordered derivative method, which is more rigorous. In fact, it tells us a great deal about the types of topologies that can be trained with backpropagation. It also led us to the interconnection implementation which is the best possible way to implement backpropagation with a computer algorithm. Its great advantage is that it can be applied to arbitrary topologies, as long as the computer can construct the dual (or transpose) network. This is trivial to do in ordered networks. Much of the power of NeuroSolutions is based on this innovation, which was first coded in 1991 by Lefebvre.

This chapter concentrated on MLP principles, but little was said about how to apply them to real-world data. This is the purpose of the next chapter.

Go to next section


 4.1 Introduction

In the previous chapter, we presented the multilayer perceptron (MLP) topology and its learning algorithm, but we used very simple data sets to illustrate its performance. In this chapter we will address the most important design choices required to build and train neural network classifiers. We focus on MLPs, but many of the topics transcend feedforward multilayer networks and apply to adaptive systems in general. The fundamental issues that we cover in this chapter are:

  • Efficiency and control of learning

  • Choice of error criterion

  • Choice of network topology

A well-designed network that is poorly trained produces inadequate results. Thus, controlling and speeding up the learning process is an extremely important part of any successful neural network application. Therefore, we cover in detail the issues regarding learning rates, more efficient search criteria, and how/when to stop the training for maximum performance.

The goal of presenting several error criteria is to provide added flexibility to the user. Although the MSE is the most widely used criterion, many other criteria exist that may be preferable in some circumstances. We cover the error norms and the information theoretic criterion of cross entropy.

The last issue we discuss is the problem of selecting an appropriate network topology to achieve good generalization. This is a crucial step in MLPs since MLPs do not control their generalization ability when trained with MSE. So we learn how to prune MLPs and improve generalization.

With these issues covered, the user is ready to solve meaningful classification problems with MLPs. This chapter includes a number of interesting real-life classification problems, including the rock crab data, the Iris data, and a computer aided diagnostic example for breast cancer. Although these data are provided in files, the student is encouraged to browse the internet and find new data.

Go to next section


 4.2 Controlling Learning in Practice

Learning (or adaptation) is a crucial step in neural network technology. Learning is the procedure that extracts the required information from the input data (with the help of the desired response in the supervised case). If learning is incomplete, the weight values will not be near their optimal values, and performance will suffer. As we have seen in Chapters 1 and 3, the good news is that there are systematic to search the performance surface. The bad news is that the search has to be controlled heuristically.

  • The user directly affects the search through

  • The selection of the initial weights

  • The learning rates

  • The search algorithms

  • The stop criterion

We have to understand the issues affecting each one of these topics to effectively train neural networks. We should also remember that the ultimate performance also depends on the amount and quality of the data set used to train the system.

A large portion of this chapter is devoted to extending the basic gradient-descent learning developed in Chapter 3, so we will concentrate on the aspects that can be improved. But it is good to remember up front that straight gradient-descent learning and its different implementations (LMS, delta rule, backpropagation) are among the most widely utilized methods to train adaptive systems because they are an excellent balance of simplicity, efficiency, and power. While the tone of the chapter may seem negative toward gradient descent-learning, this is just a consequence of the exposition goal. The reader should balance the impression with the amazing power of the technique already displayed in Chapters 1 and 3. algorithm locality and distributed systems

Before we develop a methodology to appropriately set the learning parameters, let's see how we can visualize what is happening inside the neural network during training and describe some of the features of learning.

4.2.1 Visualizing Learning in a Neural Network

An important point to remember is that learning is much richer than can be imaged from the learning curve (the thermometer of learning, as we call it in Chapter 1). All the internal parameters of the network are being changed simultaneously according to the activation flowing in the network, the errors flowing in the dual network, and the particular search algorithm used to update the weights.

Since the setup of the learning parameters is problem dependent, the user has to make decisions that are particular to the problem being solved. The only way to make appropriate judgments when a theory is lacking is to understand, through experimentation, the principles at work. Hence it is very instructive to visualize the behavior of the network parameters during learning, and we can do this effortlessly with NeuroSolutions.

NEUROSOLUTIONS EXAMPLE 4.1

Visualization of Learning

In this example we use the XOR network from Chapter 2 and place scopes on the weights and backpropagated errors. By viewing the errors, weights, decision surface, and learning curve, we will get a much better feel for what is going on in the network. Compare the evolution of the weights with the backpropagated errors. Also compare the location of the decision surface with the actual value of the weights. Finally, compare all this activity with the learning curve, the external variable that we normally observe. Do this several times. Try to understand the relationships among the different pieces. Ultimately, everything depends on the input data and the errors.

Notice that the time evolution of the weights differs every time we run the network, but the final MSE is almost the same from run to run. Every run also produces a different set of weights. Learning in a neural network is a very rich process, and the learning curve can only give a glimpse of these activities. Nonetheless it is a valuable tool for gauging the progress of learning.

NeuroSolutions Example

4.2.2 Network Weights and Minimum MSE

It is important to understand why the adaptation of the same topology with the same training data produces so many different sets of final weights. There are three basic reasons for this fact.

  • First, there are many symmetries in the input-output mapping created by the MLP. Thus, two networks that produce the exact same results may have different weights. For instance, as we discussed in Chapter 3, the position of the discriminant function is determined by the ratio of the weights, not their values. Also, changing the sign of the output weight of a PE compensates for input weights with a reversed sign.

  • Second, there is no guarantee in general that the problem has a single solution. In particular, when nonminimal topologies are utilized, the redundancies may create many possible solutions. Remember that the minimization of the output error is an external constraint. Nothing is said about the uniqueness of the weight values to provide a given output error. In fact, from the point of view of the problem formulation as long as the output error is minimized, any solution is as good as any other. system identification versus modeling

  • Third, the final weights are obtained in an iterated fashion from a random initial condition. Even when we stop the adaptation at a fixed iteration number in two different training runs over the same data, the random initial weights create different weight tracks during adaptation. Therefore, the final weight values will most likely be different.

The size of the topology often magnifies these differences and produces very different final weights from run to run. Additionally, if the topology is not minimal, there will be redundant discriminant functions, and thus there are many possible solutions for mapping the training data. Each one, however, may perform quite differently on data that the network has not seen yet (test set). This aspect will be addressed later.

This analysis points out one important methodological issue. Learning is a stochastic process that depends not only on the learning parameters, but also on the initial conditions. Thus, if we want to compare network convergence times (i.e., how much faster one update rule is with respect to another) or final values of the MSE after a number of iterations, it is pointless to run the network only once. We need to run each network several times with random initial conditions and pick the best (or use some other strategy such as committees).

When the goal is to compare different training algorithms, it is common practice to average the results, that is, to present the "mean" learning curve across the different trials. This means that learning curves should also be presented with error bars or at least with a percentage of the number of times the minimum was reached.

NEUROSOLUTIONS EXAMPLE 4.2

Learning as a Stochastic Process (XOR)

Remember that adaptation is a stochastic process: Depending on the initial conditions and other factors, the path that the network will take down the performance surface will be very different. There are many possible end points (local minimum, global minimum, saddle points, etc.) to the adaptation process and even more trajectories to get there. It is important to remember that if we are to compare one learning algorithm with another, we must average the comparison criteria over multiple runs. For example, the learning curve should always be presented as an average of the individual learning curves over many runs. In this example we show the many possible trajectories and end points for the XOR problem. We use a custom DLL to compute the average learning curve.

NeuroSolutions Example

4.2.3 Control of the Step Size During Learning

We have already encountered the problem of step size selection when we studied the linear regression and the MLP adaptation. In the linear case we can summarize the discussion by saying that the learning rate is a trade-off between speed of adaptation and accuracy in the final weight values.

It is important to realize that for quadratic performance surfaces there are ways of optimally selecting the step size at each iteration through a line search. Normally however, we use a trial-and-error approach due to the computational complexity of determining the best step size at each iteration (line search methods). For MLPs, the line search algorithm cannot be computed analytically, thus it is rarely used.

In a nonlinear network (e.g. MLP), the step size selection is even more important. The new situation is the existence of local minima and saddle points that may stall learning. We will discuss ways to manage this problem with more powerful search procedures later in the chapter.

A common technique for the step size selection is to use a large learning rate in the beginning of training to decrease the time spent in the search phase of learning, and then to decrease the learning rate to obtain good accuracy for the final weight values in the tuning phase. This is sometimes called learning rate scheduling or annealing. In addition to the step size scheduling from Chapter 1, we can use another method where the step size is controlled by

(4.1)

where [image] o is the initial step size, and n0 is an iteration count. Note that for n<<no, the step size is practically equal to [image] o, while when n>>no, it approaches zero geometrically.

The values of [image] o and no need to be found experimentally. If the initial value of [image] o is set too high, learning may diverge. The selection of no is tricky because it is highly dependant on the performance surface. If no is too small, the search phase may be too short, and learning can stall. If no is too large, then we spend too much time in the search phase, rattling around near the global minimum, before we fine-tune our solution with lower learning rates.

In nonconvex surfaces, the annealing schedule has the added advantage of enabling the search to escape from local minima when they are encountered early in the search. With a large learning rate, the search will bounce out of local minima, and when the learning rate decreases, the global minimum can be reached with accuracy. The problem is that we do not know a priori the best schedule rate, so the selection of the learning constants in Eq. 4.1 is problem dependent. The following example illustrates how the learning rate affects performance and how to schedule learning rates during adaptation.

NEUROSOLUTIONS EXAMPLE 4.3

Learning Rate Scheduling

In this example we show how to anneal the learning rate (change the step size over time during the simulation). We start with the XOR problem and add scheduling components to the gradient-descent layer. There are three available scheduling components in NeuroSolutions: the Linear, Exponential, and Logarithmic Schedulers. Each one varies the parameter over time in a slightly different manner.

Make sure you use the randomize button and play with the simulation. It is important that you change the learning parameters to make the network learn as fast as it can. You should also notice that from time to time the learning will get stuck at a MSE of 0.5 and will take a long time to get out of this mode. The weights remain practically constant, and the error does not decrease. This may be due to either a region of very low gradient (flat spot) or a local minimum.

NeuroSolutions Example

4.2.4 Setting the Learning Rates Across the Network PEs

The neurocomputing literature (Haykin) suggests that the goal for robust and fast convergence is to have all the adaptive network parameters learn at the same rate. This is possible in linear networks (with the information of the Hessian), but it is not so easy for MLPs, since in nonlinear systems the error is attenuated by the derivative of the PE nonlinearity evaluated at the operating point (see the error attenuation box of Chapter 3, Section 3.2.4). It is therefore essential to understand how the error flows inside the network to properly set the learning rates. The rule of thumb is to increase the learning rate from the output layer to the input layer by a factor of 2 to 5 from layer to layer. In the following example we observe the squelching effect of the nonlinearity from layer to layer.

NEUROSOLUTIONS EXAMPLE 4.4

Flow of Errors Across MLP Layers

In this example we again delve into the inner workings of the neural network. The problem to be solved is the star problem, a two-class problem with a set of 4 samples per class, which are placed on the vertices of two stars one smaller than the other and rotated by 45 degrees.

We will place Matrix Viewers at nearly every access point in the MLP and single-step through the training. By doing this we gain many insights. We can watch the data flow through the network and understand exactly how each component of NeuroSolutions fits into the big picture. It is OK to gloss over the details of exactly how the network operates for a while, but eventually it is important to understand the details. Notice that the data flows forward through the network, and then the error flows backward from the criterion back to the input layer. You should study these outputs in relation to the equations we derived for the MLP in Chapter 3. The following figure shows how the data flows through the arrangement of viewers in this Example:

An important insight is that the average magnitude of the errors flowing back through the network shrinks through each layer. This is due to the multiplication by the derivative of the saturating nonlinearity that is a bell-shaped curve around zero. This means that large-magnitude (negative or positive) errors are multiplied by values close to zero. If we set the learning rates constant across the net, the network learns faster in the layers closer to the output than in those deeper in the net (closer to the input). Hence, to equalize training in each layer, we should increase the learning rates of the components closer to the input. If we run the network with equal and with unequal learning rates (double the step size of the gradient search over the first synapse), there will be a marked difference.

NeuroSolutions Example

4.2.5 Nonlinear PEs as a Source of Internal Competition

The MLP's ability to learn to discriminate patterns in the input space is linked to the attenuation of the error through the nonlinearity coupled with the saturation of the PE. The PE nonlinearity works as an internal competition mechanism, which allows different PEs to specialize in different areas of the input space. In fact, recall that the weight update is a product of the local activity, the error, and the derivative of the nonlinearity f'(.), which is a bell-shaped curve for sigmoid nonlinearities (Figure 4-1). Thus, the question is this: given several PEs in the hidden layer with different operating points (different values of net), are they updated equally by the presentation of a given pattern?

Figure 4-1 The derivative of the nonlinearity and its effect on the stabilization of learning

For a particular input that the weights connected to PEs are operating just above or below net=0 will be adjusted the most (assuming that they receive a constant error from the top layer). Effectively, these are the operating points with the highest product of the activation and f 'max (net) (Figure 4-1). Thus, during learning, different PEs will train more effectively on different areas of the input space. A ballpark analysis shows that sigmoid PEs are most sensitive to samples that make net approximately ± 0.707. For normalized inputs these values correspond to approximately 45 degrees from their weight vectors. The overall effect is to stabilize the weight vector in the middle of the cluster.

If one of the PEs is saturated (net is very large in absolute value), the weights connected to it will be multiplied by a very small value of f '(net), so the weights will not change much. On the other hand, if the PE is operating near net = 0 its output will also be small, so the weights leaving the PE will likewise have a small increment. This diverse rate of updating the different weights is a source of internal competition that tends to assign some PEs to one pattern cluster and other PEs to a different pattern cluster. Since the PEs in an MLP create the discriminant functions, this is the source of the power of nonlinear systems for classification and what differentiates them from linear systems. If the PEs were linear (or nonlinear but nonsaturating), there would not be any internal competition, and the weights associated with each PE would tend to have the same value (remember what happens in the Adaline). PEs would never specialize, and the network would not be able to respond sharply with high values for some patterns and low values for other patterns.

4.2.6 Weight Initialization

The attenuation of the error across a layer also imposes some constraints on the learning process. Two such constraints involve the weight initialization and the choice of the number of hidden layers.

For the sake of training efficiency we should not create topologies with many layers, since the layers closer to the input will train very slowly, if at all. One should always start to solve a problem with shallow networks (i.e., a perceptron) for quicker training. If the perceptron does not provide a reasonable error, then try the one-hidden-layer MLP, and finally the two-hidden-layer MLPs should be tested. Since two-hidden-layer MLPs are universal mappers, more than two nonlinear hidden layers are rarely recommended.

Another variable that the user can set before starting a training run is the initial weight values. The initial weights affect the learning performance of the network, because an initialization far away from the final weights increases the training time, but also because we would like for all the PEs in the network to learn at the same speed. good initial weight values

To break possible symmetries that could stall learning (i.e., the degenerate solution where all weights are zero in the XOR problem) or saturate the PEs, it is common practice to start the network weights with random initial conditions. The random initial condition is implemented with a random-number generator that provides a random value.

As we discussed earlier, a PE that is in its linear region learns faster than one that is in the saturated region. For better training performance the goal is to have each PE learn at approximately the same rate. If we set the variance of the random initial conditions based on the fan-in of the PE (i.e., the number of inputs it receives), each PE will be near its linear region, so all will learn at approximately the same rate. For the tanh nonlinearity, a rule of thumb is to set the variance of the random initial weights at

(4.2)

where I is the fan-in of the PE.

NEUROSOLUTIONS EXAMPLE 4.5

Effect of Initial Conditions on Adaptation

In this example we will show how the initial conditions dramatically affect the adaptation of the weights in the XOR problem. When we set the initial conditions to the values previously discussed, we get significantly better performance and better learning curves.

NeuroSolutions Example

Go to next section


 4.3 Other Search Procedures

The popularity of gradient descent is based more on its simplicity (it can be computed locally with two multiplications and one addition per weight) than on its search power. There are many other search procedures more powerful than backpropagation. In Chapter 1 we discussed Newton's method, which is a second-order method because it uses the information on the curvature to adapt the weights. However Newton's method is computationally much more costly to implement and requires information not available at the PE, so it has been used little in neurocomputing. Although more powerful, Newton's method is still a local search method and so may be caught in local minima or diverge due to the difficult neural network performance landscapes. Other techniques such as simulated annealing and genetic algorithms (GA) are global search procedures, that is, they can avoid local minima. The issue is that they are more costly to implement in a distributed system like a neural network, either because they are inherently slow or because they require nonlocal quantities. Global Optimization is beyond the scope of this book, and the interested reader is directed to Horst, Pardalos and Thoai).

Here we cover improvements to the basic gradient-descent learning

which is generally called a first-order search method. Recall that the LMS algorithm, delta rule and backpropgation use weight updates that are all implementations of this basic concept. They use a sample (noisy) estimate of the gradient that essentially multiplies the local error by the local activation

(4.3)

We will improve Eq. 4.3 to cope with the added difficulty of searching nonconvex performance surfaces or surfaces that may have flat regions. It is obvious that a gradient-based algorithm can be trapped in local minima since the search is based on local gradient information only (see Chapter 3). In terms of local curvature, a local minimum and the global minimum are identical, so the gradient descent will be trapped in any local concavity of the performance surface. The gradient-descent method will move very slowly, if at all (called stalling), when the search traverses a flat region of the performance surface, because the weights are modified proportionally to the gradient. If the gradient is small, the weight updates will be small (for a constant step size), so many iterations are needed to move through the flat spot. This situation is easily confused with the end of adaptation, when the search reaches the global minimum of the performance surface.

These preceding arguments indicate a need for an appropriate control of the step sizes or learning rates and for improved search methodologies. In this section we introduce a few methods that help alleviate the problems of gradient-descent search of nonconvex performance surfaces (local minima and saddle points).

4.3.1 Momentum Learning

Momentum learning is an improvement to the straight gradient-descent search in the sense that a memory term (the past increment to the weight) is used to speed up and stabilize convergence. In momentum learning the equation to update the weights becomes

(4.4)

where [image] is the momentum constant. Normally [image] should be set between 0.5 and 0.9. This is called momentum learning due to the form of the last term, which resembles the momentum in mechanics. Note that the weights are changed proportionally to how much they were updated in the last iteration. Thus if the search is going down the hill and finds a flat region, the weights are still changed, not because of the gradient (which is practically zero in a flat spot), but because of the rate of change in the weights. Likewise, in a narrow valley, where the gradient tends to bounce back and forth between hillsides, the momentum stabilizes the search because it tends to make the weights follow a smoother path. Figure 4-2 summarizes the advantage of momentum learning. Imagine a ball (weight vector position) rolling down a hill (performance surface). If the ball reaches a small flat part of the hill, it will continue past this local minimum because of its momentum. A ball without momentum, however, will get stuck in this valley. Momentum learning is a robust method to speed up learning, and we recommend it as the default search rule for networks with nonlinearities.

Figure 4-2 Why momentum learning helps

NEUROSOLUTIONS EXAMPLE 4.6

Momentum Learning

This example compares the speed of adaptation of the straight gradient-search procedure with momentum learning. Since there are many local minima in this problem, momentum learning works much better since it helps us "roll" through the flat parts of the performance surface. The problem is our well known Star problem. Notice that with momentum learning the speed of adaptation is much higher for the same learning rate.

NeuroSolutions Example

4.3.2 Adaptive Step Sizes

Simplicity is the only reason to use the same step size for each weight during the entire training phase. In fact, we know from Chapter 1 that the step size should be determined according to the eigenvalue for that particular direction. The problem is that eigenvalues are not normally known, so it is impractical to set them by hand. However, we have the feeling that observing the behavior of the error, weight tracks or both, should allow better control of the training. In fact, when the learning curve is flat, the step size should be increased to speed up learning. On the other hand, when the learning curve oscillates up and down, the step size should be decreased. In the extreme the error can go steadily up, showing that the learning is unstable. At this point the network should be reset. We can automate this reasoning in an algorithm by adapting the step sizes independently for each weight through the training phase.

The idea is very simple: Instead of looking at the learning curve we use the weight tracks. When consecutive weight updates produce the same error sign, the learning rate is too slow. Conversely, when the error sign is toggling from iteration to iteration, the step size is too large. These simple principles can be put into a rule called the adaptive step size method that adapts each step size continuously during training. For the method to work correctly, you must use an independent step size for each weight, otherwise there will not be significant improvement in the performance. Let us denote the learning rate for the weight wij as [image] ij. The update to each step size is

(4.5)

where Sij measures the average of previous gradients, and Dij is the current gradient. When Sij and Dij has the same sign, their product will be greater than zero. The first case refers to slow convergence, so the step size is arithmetically increased at each iteration (by a constant, which is a slow process). The second case refers to the situation when the weight is oscillating so the algorithm decreases the step size proportionally to its current value, which is a geometric decrease (very fast). The slow increase and fast decrease of the step size is designed to avoid divergence at any cost if the algorithm diverges, it quickly loses all the information it has learned and must start over.

Now let us carefully analyze the conditions to increase or decrease the step size. Dij (n) is the partial derivative of the cost with respect to weight wij (i.e., the current gradient), and Sij (n) is a running average of the current and past partial derivatives given by

(4.6)

where [image] is a number between 0 and 1 (the exponential time constant). The product of Sij and Dij is checking whether the present gradient has the same sign as the previous average of the gradients (reflected in the value of Sij ). This algorithm is called the Delta-Bar-Delta algorithm.

NEUROSOLUTIONS EXAMPLE 4.7

Adaptive Step Sizes

Let us now solve the Star problem with the adaptive step size (delta-bar-delta) search to see the speedup achieved. The adaptive step size algorithm gives us the flexibility to have high learning rates when we are in flat parts of the performance surface and low learning rates when the adaptation begins to rattle or become unstable. The NeuroSolutions version of the delta-bar-delta algorithm includes momentum, so this algorithm should work better than any of the methods we have discussed so far.

NeuroSolutions Example

There are many other alternative algorithms to adapt the step size, such as Fahlman's quickprop, and Almeida's adaptive step size (see Almeida's adaptive step size). You should experiment with them in NeuroSolutions since they are implemented there. The problem is that all these algorithms increase the number of free parameters that the user has to set heuristically, so they require fine-tuning for each application.

4.3.3 Random Pertubation During Learning

Another alternative is to inject random noise at either the weights, or desired response. The motivation is to "shake" the weights during adaptation in order to minimize the probability of having the search caught in local minima. This idea is reminiscent of the operating principle of simulated annealing, which uses a scheduling of noise to reach the global minimum of the performance surface. The process is also very simple to implement when noise is added to the desired response. In fact, if we add zero-mean white Gaussian noise to the desired response, we obtain

which is then transmitted to the injected error e(n) and then to the weight update through backpropagation. The advantage of applying the noise to the desired response is that we have a single noise source, and since the dual network is linear, as we saw in Chapter 3, it is still an additive zero-mean perturbation to the weight update. But remember that the noise variance should be scheduled to zero so that the optimal solution is obtained. Unfortunately, there is no rigorous approach to set the noise variance nor to schedule it. The injection of noise at the input is not a linear contribution to the weight update, but it has been shown to produce better generalization (see Bishop) which is highly desirable.

NEUROSOLUTIONS EXAMPLE 4.8

Adaptation with Noise in the Desired Signal

This example adds the noise component to the desired signal to show how it can improve the performance. The noise helps the algorithm to bounce out of local minima. We also anneal the noise variance so that we have a large amount of noise at the beginning of training to help us move out of local minima quickly but little noise at the end of training to reduce the rattling and increase precision.

NeuroSolutions Example

Notice that until now all the modifications to the gradient-descent rule of Eq. 4.3 utilize the same basic information, namely, the activation of the PE and the error to its dual network which characterize first-order search methods. In NeuroSolutions the data-flow architecture for backpropagation and the layered nature of its implementation provide a straightforward selection of any of these methods. In fact, only the weight-update rule (the search component) needs to be modified, which can be accomplished by simply changing the search component icon on the breadboard. This further enhances the value of the data-flow concept for simulation of neural networks.

4.3.4 Advanced Search Methods

Numeric optimization techniques are a vast and mature field. We provide here only a synopsis of the classes of search methods that are important for neural network researchers. The interested reader is referred to Luenberger and Fletcher for a full coverage of the topic.

The problem of search with local information can be formulated as an approximation to the functional form of the cost function J (w) at the operating point w0. This immediately points to the Taylor series expansion of J around w0

where [image] J is our already familiar gradient, and H is the Hessian matrix, that is, the matrix of second derivatives with entries

evaluated at the operating point. We can immediately see that the Hessian can NOT be computed with the information available at a given PE, since it uses information from two different weights. If we differentiate J with respect to the weights, we get

(4.7)

so we can see that to compute the full gradient at w we need all the higher terms of the derivatives of J. This is impossible. Since the performance surface tends to be bowl shaped (quadratic) near the minimum, we normally are interested only in the first and second terms of the expansion.

If the expansion of Eq. 4.7 is restricted to the first term, we obtain the gradient-search methods (hence they are called first-order methods), where the gradient is estimated with its value at w0.

If we expand to use the second-order term, we obtain Newton's method (hence the name second-order method). It is interesting to note that if we equate the truncated Eq. 4.7 to 0 we immediately get

(4.8)

which is exactly the equation for the Newton's method presented in Chapter 1. Newton's method has the nice property of quadratic termination (it is guaranteed to find the exact minimum in a finite number of steps for quadratic performance surfaces). For most quadratic performance surfaces it can converge in one iteration.

The real difficulty is the memory and the computational cost (and precision) to estimate the Hessian. Neural networks can have thousands of weights, which means that the Hessian will have millions of entries. This is why methods of approximating the Hessian have been extensively researched. There are two basic classes of approximations:

  • Line search methods

  • Pseudo-Newton methods

The information in the first type is restricted to the gradient, together with line searches along certain directions, while the second seeks approximations to the Hessian matrix.

Line Search Methods

The basic idea of line search is to start with the gradient-descent direction and search for the minimum along the line, that is,

The problem with the gradient direction is that it is sensitive to the excentricity of the performance surface (caused by the eigenvalue spread), so following the gradient is not the quickest path to the minimum. We analyzed this aspect in Chapter 1. Alternatively, we can compute the optimal step size at each point, which corresponds to a line search as we saw earlier. We can prove that successive directions have to be perpendicular to each other (Figure 4-3), that is, the path to the minimum is intrinsically a zigzag path (Luenberger).

Figure 4-3 Path to the minimum with line search methods

We can improve this procedure if we weight the previous direction to the minimum with the new direction, that is, cut across the zigzag. The formulation becomes

where [image] is a dynamically computed parameter that compromises between the two directions. This is called the conjugate gradient method. For quadratic performance surfaces the conjugate gradient algorithm preserves quadratic termination and can reach the minimum in D steps, where D is the dimension of the weight space. As we will see later, the interesting thing is that we do not need to compute second derivatives (Hessian), and, in fact, the algorithm is compatible with backpropagation.

Notice that momentum learning is similar to the conjugate gradient method, since we can view the difference between the present and the previous weights as the estimate of the old direction. In momentum learning, however, [image] is fixed throughout instead of being estimated at each step, and there is no search for the best step size.

Pseudo-Newton Methods

In pseudo-Newton methods the idea is to come up with computationally simple and reasonable approximations to the Hessian. The simplest is just to forget about the cross terms in the Hessian matrix, and use only the diagonal terms. This is equivalent to performing Newton's algorithm separately for each weight, which transforms Eq. 4.8 into

Normally we replace this rule by

(4.9)

where [image] is a small constant that avoids the problem of negative curvature and a zero denominator. Notice that Eq. 4.9 is in fact very similar to the normalized LMS we presented in Chapter 1, since for the linear network we can estimate the diagonal entries of the Hessian by the power (or the trace) of the input.

This is a very crude approximation of the Hessian. More accurate approximations that are still less computationally expensive than the full procedure (which is O(N³), with N being the number of weights) are the Levenberg-Marquardt (LM), the Davidson-Fletcher-Powell (DFP), and the Broyden-Fletcher-Goldfarb-Shanno (BFGS). See Luenberger. LM is the most interesting for neural networks, since it is formulated as a sum of quadratic terms just like the cost functions in neural networks.

Go to next section


 4.4 Stop Criteria

We have addressed many aspects of gradient-descent learning, but we have not yet discussed how and when to stop the training of the neural network. Obviously, training should be stopped when the learning machine has learned the task. The problem, however, is that there are no direct indicators that measure this condition.

4.4.1 Stopping Based on Training-Set Error

One of the simplest ways to stop the training phase is to limit the number of iterations to a predetermined value, as we have done so far, but the only appeal of this criterion is simplicity. It does not use any information or feedback from the system before or during training. When the number of iterations is capped at a predefined value, there is no guarantee that the learning machine has found coefficients that are close to the optimal values.

This suggests an analysis of the output MSE to stop the training. We can choose an acceptable error level for the problem and apply a threshold to the MSE. Choosing the MSE threshold value, however, is tricky because the MSE is just an indirect variable in classification. Moreover, we have no guarantee that the learning system will achieve the preselected MSE value, so the training may never end.

Another alternative is to let the system train until the decrease of the MSE from one iteration to the next is below a given value (applying a threshold to the incremental change in MSE). The idea is to let the system train until the point of diminishing returns, that is, until it basically cannot extract more information from the training data. Unfortunately, this method has the problem of prematurely stopping the training in flat regions of the performance surface.

NEUROSOLUTIONS EXAMPLE 4.9

Stopping Based on MSE Value

In this example we use a new component, the Transmitter, to stop training. There are two types of Transmitters, the Threshold Transmitter and the Delta Transmitter. The threshold Transmitter transmits a message (in this case to the Controller to stop training) when the value of the error gets below a certain point. The Delta Transmitter stops training when the difference between two successive errors is below a certain value. When you look at the properties of the Delta Transmitter, you notice that there is a smoothing algorithm applied to the error – this can help reduce the probability of stopping because of outliers. Notice that both of these stop criteria have their problems. We discuss better methods in the next section.

NeuroSolutions Example

4.4.2 Stop Criterion Based on Generalization

The previous stop criteria did not address the problem of generalization, that is, how well the learning system performs with data that does not belong to the training set. Traditional knowledge from data modeling and recent developments in learning theory (Vapnik) clearly indicate that after a critical point an MLP trained with backpropagation will continue to do better in the training set, but the test set performance will begin to deteriorate. This phenomenon is called overtraining.

One method to solve this problem is stop the training at the point of maximum generalization (given the present data and topology). This method is called early stopping, or stopping with cross-validation. It has been experimentally verified that the training error always decreases when the number of iterations is increased (for a sufficiently large network). If we plot the error in a set of data with which the network was not trained (the validation set), we find that the error initially decreases with the number of iterations but eventually starts to increase again (Figure 4-4). Training therefore should be stopped at the point of the smallest error in the validation set.

Figure 4-4 Cross validation or early stopping

To implement this method, the training set should be divided into two sets: the training and the cross-validation sets. The cross-validation set is normally taken as 10 percent of the total training samples. Every so often (i.e., 5 to 10 iterations), the learning machine performance with the present weights is tested against the cross-validation set. Training should be stopped when the error in the cross-validation set starts to increase. This point is the point of maximum generalization.

The problem with this methodology is that cross-validation decreases the size of the training set. Since neurocomputing tends to suffer from a lack of data to begin with, cross-validation makes this situation even worse. In this case, however, the benefits (accurate stopping point) typically outweigh the costs (less data). This method is the recommended stop criterion for real world applications. NeuroSolutions implements this method, as we exemplify next.

NEUROSOLUTIONS EXAMPLE 4.10

Stopping with Cross-Validation

We again solve the star problem but this time using cross-validation as the stop criterion. We have created an additional set of points, which we use as our cross-validation (or test) set. The following figure shows the training and cross-validation sets.

To implement cross-validation in NeuroSolutions, we will use two input/desired file pairs. In the File component we have to specify that one pair will be the training/desired files, while the other will be the cross-validation files. Likewise in the Controller we have to specify that we are using cross-validation.

This example clearly shows that overtraining can greatly reduce the generalization ability of the network. You will also see how important it is to have a sufficient amount of training data to adequately cover the input space. Any points in the input space that are not in the training set are considered to be "don't cares" by the network. This can result in oddly shaped discriminant plots that have a very good mean square error in the training set but poor performance with new data.

NeuroSolutions Example

Go to next section


 4.5 How Good are MLPs as Learning Machines?

4.5.1 Training Set Size

The size of the training set directly influences the performance of any classifier trained nonparametrically (e.g., as neural networks). This class of learning machines requires a lot of data for appropriate training, because there are no a priori assumptions about the data. It is important to know how the requirement on the size of the training set size scales as a function of the size of the network for a given precision in the mapping.

The number of training patterns (N) required to classify test examples with an error of [image] is approximately given by [Haykin]

(4.10)

where W is the number of weights in the network. This equation shows that the number of required training patterns increases linearly with the number of free parameters of the MLP, which is excellent compared with other classification methods. A rule of thumb states that N ~ 10W, that is, the training set size should be 10 times larger than the number of network weights to accurately classify test data with 90 percent accuracy.

In this rule of thumb it is assumed that the training set data is representative of all the conditions encountered in the test set. The main focus when creating the training set should be to collect data that covers the full known operating conditions of the problem we want to model. If the training set does not contain data from some areas of pattern space, the machine classification in those areas will be based on extrapolation. This may or may not correspond to the true classification boundary (your desired output), so we should always choose samples for the training set that cover as much of the input space as possible.

We should remark that in Eq. 4.10 the practical limiting quantity is the number of training patterns, so most of the time we need to compromise the size of the network to achieve appropriate training for the learning machine. A reasonable approach to reduce the number of weights in the network is to sparsely connect the input layer to the first hidden layer (which normally contains the largest number of weights). This will help achieve the requirement of Eq. 4.10. Another possibility is to use feature extraction (i.e., a preprocessor) that decreases the input space dimensionality, thus reducing the number of weights in your network.

NEUROSOLUTIONS EXAMPLE 4.11

Sparse Connectivity in the Input Layer

Here we show that, with the Arbitrary Synapse, generalization can be improved with respect to the fully connected case. We will use the STAR data set that we have been training with 4 hidden PEs. But instead of 8 weights in the first Synapse, now we restrict the number to 4, randomly connected. Run the network and show that the solutions found are normally more reasonable than the ones using 8 weights. One way to show this is to use the cross-validation data as you did before.

NeuroSolutions Example

4.5.2 Scalability

Another important point for learning machines is to address how well their properties scale when the size of the problem increases. The literature is full of examples of systems that perform very well in small problems but are unable to extend the same performance to larger problems. One important proof advanced by Barron through the analysis of the MSE for several size problems, states that for large training sets the (one-hidden-layer) MLP error is independent of the size of the input space and scales as the inverse of the number of hidden PEs (O(1/N)). This is much better than polynomial approximators, where the error decreases geometrically with the dimension D of the space ( [image]). Hence, MLPs are particularly well suited to deal with large-input-dimensional problems. This may explain the good performance of MLPs in large classification problems.

4.5.3 Trainability

The training time of an MLP using backpropagation was experimentally determined to increase exponentially with the size of the problem, that is, although the required number of patterns increases only linearly with the number of weights, the training time of larger networks seems to scale exponentially to their size. This indicates that there are problems that cannot be solved practically with MLPs trained with backpropagation. However, to counteract this behavior, we can use modular network architectures or sparse connections, as well advanced training rules. In practice this exponential scaling of training times with network size provides another argument to start first with small networks and increase their size if the results are not satisfactory. Moreover, this property emphasizes the importance of training methods that extract information more efficiently from the data than gradient descent. Now you may understand better the importance of the discussion of the conjugate gradient and the quasi-Newton methods presented in Section 4.3.4.

4.5.4 Did the Network Learn the Task?

In the application of neural networks to real-world problems, it is very important to have a criterion for accepting the solution. Only then can we successfully act to overcome any potential difficulties.

The learning curve is a valuable indicator for observing the progression of learning, but the MSE in the training or test sets is only an indirect measure of classification performance. The MSE depends on the normalization and characteristics of the input data and desired response. We should normalize the total error by the variance of the desired response to have an idea of how much of the desired variance was captured by the neural model. This is reminiscent of the correlation coefficient for linear regression, but there is no precise relationship between classification accuracy and MSE.

The performance of a classifier is measured in terms of classification error as we saw in Chapter 2. The accuracy of the classifier is 1 minus the classification error. Therefore a much better approach is to construct the confusion matrix to count the number of misclassifications exactly. The confusion matrix is a table where the true classification is compared with the output of the classifier (see Table 4-1). Let us assume that the true classification is the row and the classifier output is the column. The classification of each sample (specified by a column) is added to the row of the true classification. A perfect classification provides a confusion matrix that has only the diagonal populated. All the other entries are zero. The classification error is the sum of off-diagonal entries divided by the total number of samples.

Table 4-1 Confusion Matrix

The confusion matrix also enables easy visualization of where the classifier has difficulties. In general, some of the classes will be more separable than others, so the confusion matrix immediately pinpoints which classes produce misclassifications (off diagonal entries that have large values). In summary, the confusion matrix is an excellent way of quantifying the accuracy of a classifier. As we discussed in Chapter 2, the test of the classifier should be performed in the test set, so the confusion matrix should be constructed in the test set.

NEUROSOLUTIONS EXAMPLE 4.12

Confusion Matrix for Classification Performance

In this example we show how the confusion matrix gives us a much better picture of the performance of the network for classification than the mean square error. The confusion matrix tells us exactly how well the network is classifying the data. The mean square error tells us only the average difference between the network output and the desired output. Since classification is an "all or nothing" type of problem, it doesn't always matter how close you are to the desired ± 1 as long as you are beyond the classification threshold.

NeuroSolutions Example

Once we find that learning is not successful (a large classification error assessed by the confusion matrix), the next step is to find out why the network did not learn correctly. Poor performance may have many different explanations:

  • The network may not have the discrimination power (number of layers) to correctly classify the data (remember the perceptron and the XOR).

  • The network may not have enough PEs (remember the case of the bump with two hidden PEs).

  • Learning may be stuck in a local minimum or flat spot.

  • We may have overtrained the network.

  • We may not have collect enough data to represent the problem well.

  • The problem may be intrinsically difficult with the features (measurements) that we are using, so we may need to transform or filter the inputs or add new inputs to simplify the classification problem.

Unfortunately there is no general rule to test any of these issues, so the designer of classifiers must use insight and knowledge of the problem to improve the machine classification. The designer should also manually supervise the learning instead of setting the learning parameters and dimming the monitor until the next morning. NeuroSolutions probes are particularly useful for monitoring the learning process.

Go to next section


 4.6 Error Criterion

4.6.1 Lp Norms

In supervised learning the difference between the desired response and the actual learning system output is used to modify the system so that the minimum error of the performance surface is achieved. The issue is how to define the performance, also called the error criterion or the cost. In normal operation the learning machine provides an output for each input pattern, so the total cost J is computed as a sum of individual costs, Jn,k, obtained from each input pattern presentation, i.e.

(4.11)

where k is an index over the system outputs and n is an index over the input patterns. Jnk is the individual cost defined as Jnk = f(dnk-ynk) = f( [image] nk). The only issue is then how to compute the individual cost as a function of [image] nk, which is called the instantaneous error [image] nk.

The mean square error (MSE) criterion defines the individual cost as the square of the instantaneous error between the desired response and the system output, that is, Jnk=(dnk [image] ynk)². The error power (MSE) has a meaning in itself and has three other major appeals:

• In linear feedforward networks, the MSE leads to a linear optimization problem, which has an analytical solution.

• It provides a probabilistic interpretation for the output of the learning machine, as we discussed in Chapter 2.

• The criterion is easy to implement, since it is the instantaneous error that is injected into the dual system (no additional computations are needed).

Is there a need for other error criteria? Let's look at Figure 4-5 and understand what the MSE criterion does.

Figure 4-5 Error norms

In Figure 4-5 we present several cost functions derived from different powers p of the instantaneous error. With the MSE the instantaneous cost is the square of the magnitude of the instantaneous error (p = 2). This means that when the learning machine minimizes the error power, it weights the large errors more (quadratically). If you recall the weight update performed by gradient descent, [image] wij = [image] [image] j xi, you will realize that the weight values are updated proportionally to the size of the error, so the weights are sensitive to the larger errors. This is reasonable if the data is clean without many large deviations, but in practice the data sets may have outliers. Outliers thus may have an inordinate effect on the optimal parameter values of the learning machine. Learning machines with saturating nonlinearities control this aspect better than linear PE machines, but they still are more sensitive to large errors than small errors. Since the values of the weights set the orientation and position of the discriminant function, we can deduce that outliers will "bias" the position of the discriminant function.

This argument shows that if we want to modify how the instantaneous error influences the weights, we can define the instantaneous cost more generally as

(4.12)

where p is an integer, which is normally called the p norm of the instantaneous error [image] nk. When p = 2 we obtain the L2 norm that leads to the MSE criterion. When p = 1 we obtain the L1 norm, which is also called the Manhattan metric. Notice that the L1 norm weights the differences proportionally to their magnitude, so it is far less sensitive to outliers than the L2 norm. For this reason it is called a more robust norm. In general the Lp norm for p > 2 weights large deviations even more. Different norms provide different solutions to a learning problem, because the weights are modified with information that depends on the choice of the norm, so the positioning of the discriminant functions is affected by the norm (for the same training data set).

NEUROSOLUTIONS EXAMPLE 4.13

Regression Performance as a Function of the Norm

In this example we use the linear regressor and the data from Chapter 1 to show how the choice of error criterion affects the performance of the linear regression. We have modified the data set so that one point is an outlier; it is much higher than the others. As you can see in the example, the norms that weight large errors more heavily skew their regression line much more closely to the outlier (giving a less accurate estimate of the rest of the data). Remember not to try to compare the errors as displayed in NeuroSolutions; the values of the error reported depend on the error criterion and thus can't be compared.

NeuroSolutions Example

For positive finite integers p, the derivative of the norm can be computed quite easily as

(4.13)

but Lp norms do not cover all the cases of interest. For errors larger than 1, the instantaneous cost Jnk for the Lp norms always increases at the same rate or faster than the instantaneous error which may not be our goal. There are cases of practical relevance that do not have an analytic solution such as the L [image] norm (all errors are zero except the largest). Another possible criterion is to simply use the sign of the deviation (p=0). Minskowski measures

NEUROSOLUTIONS EXAMPLE 4.14

Classification Performance as a Function of the Norm

In this example we show how the choice of error criterion affects the location of the discriminant function. We seek to classify male and female subjects based on their height and weight, but using different norms. You will see that the discriminant function ends up in different places, depending on the norms, since each norm weights the errors differently.

NeuroSolutions Example

4.6.2 Constructing the Error Directly

In backpropagation learning, the partial derivative of the cost with respect to the weight is given by

(4.14)

The first partial derivative is the instantaneous error that is injected in the dual network. Given a cost function one can differentiate it to compute the instantaneous error of Eq. 4.14 and then use backpropagation to compute the sensitivity with respect to the individual weights wij (the second term in Eq. 4.14).

An alternative practical approach is to separate the propagation of the instantaneous error (done with the backpropagation algorithm) from its generation. Instead of thinking of an error produced by the derivative of a cost function (which requires differentiable cost functions) a function of the error, g( [image] k,n), can be directly injected in the dual system, i.e.

(4.15)

The function g(.) is derived to meet our needs. For instance arctanh ( [image] k,n) implements a reasonable approximation of the L [image] norm, since it weights large errors exponentially. Another example is simply to take the sign of the instantaneous error and inject ± 1, but many more examples with unexplored properties exist.

4.6.3 Information Theoretical Error Measures

In information theory we can measure the divergence (relaxed definition of distance) between two probability mass functions q(x) and p(x) by the Kullback-Leibler (K-L) information criterion or cross entropy cross entropy criterion

(4.16)

Since the learning system output is approximating the desired response in a statistical sense, it is reasonable to utilize the K-L criterion as our cost. In this case p(x) becomes the target density constructed by +1 and -1, and q(x) the learning system output density. This is particularly appropriate for classification problems where the L2 assumption is weak because the distribution of the targets is far from Gaussian. For c classes the K-L information criterion becomes

(4.17)

where n is the index over the input patterns and k over the classes. Since this criterion works on probabilities, the output PEs should be softmax PEs. It can be shown (Hertz) that the instantaneous error backpropagated through the network (the partial derivative of J with respect to y) with the softmax is

(4.18)

This is an interesting result, since it says that the cross-entropy criterion can be implemented by the MSE criterion (which also specifies the injection of the error as Eq. 4.21). However, the network uses output PEs that implement the softmax function. This can be easily accommodated if we associate the softmax PE with a linear PE in the backplane as its dual (Figure 4-6). In the two-class problem (single output PE) the softmax becomes a logistic PE.

By not subjecting the error to the attenuation produced by the derivative of the output PE nonlinearity, the network converges faster. It is also possible to show that the cross entropy is similar to the L1 norm, which means that this criterion weights small errors more heavily than the L2 norm.

Figure 4-6 Implementation of the cross-entropy criterion

The other important aspect of training the MLP with cross-entropy is that the interpretation of the output as the a posteriori probability of the class given the data, also hold in this case (Bishop).

NEUROSOLUTIONS EXAMPLE 4.15

Cross-Entropy Training

In this example we show you how easy it is to implement the cross-entropy criterion for the two-class case in NeuroSolutions. All we need to do is change the Back PE of the logistic nonlinearity at the network output to a linear Back PE. This changes the backpropagated error so that the network will learn using entropy. Notice that the learning is faster, as we would expect, since the error is not attenuated by the derivative of the nonlinearity.

NeuroSolutions Example

Go to next section


 4.7 Network Size and Generalization

The coupling between the number of required discriminant functions to solve a problem and the number of PEs was heuristically established in Chapter 3, and the relation between the number of weights and training patterns was discussed in Section 4.5.1. From these facts we could think that the larger the learning machine, the better its performance (provided we have enough data to train it). The point about scalability shows, however, that larger machines may not learn well, but this is not the most pressing issue. All these arguments pertain to the training data. The fundamental question in any practical application is, how does the learning machine perform on the test-set data? This is the problem of generalization.

As we discussed earlier, MLPs trained with backpropagation do not control their generalization ability, which can be considered a shortcoming of the technology. Using a cross-validation set to stop the training allows us to maximize generalization for a given network size. However, it does not provide a mechanism for establishing the best network topology for generalization. The issue is the following: Do larger networks maintain the generalization of smaller ones?

If we reflect on how the network performs its function, we immediately see that the size of the machine (sometimes called the model complexity) is related to performance: Too few degrees of freedom (weights) affect the network's ability to achieve a good fit to the target function. If the network is too large, however, it will not generalize well, because the fit is too specific to the training-set data (memorization). An intermediate network size is our best choice. Therefore, for good performance, methods of controlling the network complexity become indispensable in the MLP design methodology.

The problem of network size can be stated in a simplified manner using Occam's razor argument as follows:

Any learning machine should be sufficiently large to solve the problem, but not larger.

The issue is to know what is large enough. Structural learning theory (V-C dimension ) gives a theoretical answer to generalization, but it is difficult to apply to the MLP. Alternative theories give partial answers that elucidate the principles at work and will be covered in Chapter 5. early stopping and model complexity

There are two basic approaches that deal with the learning machine size (Hertz, Krogh, Palmer). Either we start with a small machine and increase its size (growing method), or we start with a large machine and decrease its size by pruning unimportant components (pruning method). Pruning is the only method we will address here.

Pruning reduces the size of the learning machine by eliminating either weights or PEs. The basic issue of pruning is to find a good criterion to determine which parameters should be removed without significantly affecting the overall performance of the network. We describe two basic methods: eliminating weights based on their values and computing the importance of the weight in the mapping.

4.7.1 Weight Elimination

The idea in weight elimination is to create a driving force that will attempt to decrease all the weights to zero during adaptation. If the input-output map requires some large weights, learning will keep bumping up the important weights, but the ones that are not important will be driven to zero. This idea is called weight decay. Weight decay can be implemented very simply by adding an extra term into the weight adaptation, as shown here for the gradient descent rule

(4.19)

where [image] is the local error, x the local activation, [image] the learning rate, and [image] the weight-decay constant. Weights that are smaller than a certain value can be eliminated, reducing the overall number of degrees of freedom of the network. Weight decay should not be applied to the biases of the network, just to the weights.

Alternatively we can use only the sign of the weight to change its value:

(4.20)

where sgn(.) is the signum function. The problem with Eq. 4.20 is that it favors many small weights instead of a large one, producing normally model bias (Bishop). One way to counteract this problem is to create a weight-decay term that is smaller for larger weights:

(4.21)

The weight-elimination method is very easy to implement because the weights are being updated and decayed during adaptation. The issue is how to select a good weight-decay constant [image] such that convergence is achieved and unnecessary weights go to zero.

The weight-decay equations can also be applied in lieu of early stopping as a method to constrain the complexity of the model. It has been shown that for quadratic performance surfaces, weight decay is equivalent to early stopping. This will be further described in Chapter 5.

NEUROSOLUTIONS EXAMPLE 4.16

Weight-Decay

In this example we use the weight decay algorithm from Eq. 4.22. In general, we do not know how many PEs will be required to solve a given problem. However, based on generalization considerations, we know that we want as small a network as possible. If we have too many PEs, we can obtain low errors during training but the network generalization may be poor due to overtraining, that is, obtained a solution too specific to the training data. Weight decay subtracts a small portion from each weight during each update. Since the weights that are important to the solution will be constantly updated, while the others are not, the important weights will move to the correct value, and the other weights will approach zero. This effectively limits the number of PEs to only those which are required, which is the desired result.

NeuroSolutions Example

4.7.2 Optimal Brain Damage

The previous method is based solely on weight magnitude, which is a crude measure of the importance of the weight in the input-output map. A better measure to compute weight saliency is to find the effect on the cost of setting a weight to zero (LeCun). It has been shown that the Hessian H with elements

(4.22)

contains the required information. The problem with this computation is that it is nonlocal (requires information from pairs of weights). A local approximation (sometimes poor) to the Hessian can be computed by taking into consideration only the diagonal terms, which leads to the following calculation of the saliency si for each weight wi,

(4.23)

To apply this method, a large network should be trained in the normal way, and saliencies computed for each weight. Then the weights are ordered in terms of their saliency, and a percentage with the smaller saliences discarded. The network needs to be retrained using the previous values of the weights as the initial condition. The process can then be repeated.

4.7.3 Committee of Networks

We saw that learning was a stochastic process, which means we should run the same network several times to make sure we get good and stable training. We just saw that the topology of a neural network is a difficult decision because we have to take into consideration the generalization ability of the machine, which depends on its size.

A way of improving the performance of neural network classifiers is to use several networks of different sizes and characteristics to solve the same problem. Suppose we train C different networks with the same data. One temptation is to use the network that produced the best possible error in the training set. This strategy is not very good first, because it wastes the training of all the other runs and, second, because the best performer in the training set does not necessarily extrapolate to the test set. A much better strategy is to use ALL the trained networks, that is, make decisions with a committee of networks. Let us analyze what happens if we simply add their outputs:

(4.24)

If we assume that the errors from each network are zero mean and uncorrelated with each other, we can show (Perrone) that the error of the committee is

(4.25)

where [image] is the mean error of each network working individually. So the error of the committee is c times smaller, which is a large reduction. This is optimistic in general, since the errors among networks are not uncorrelated. The major advantage comes from a reduction of the variance of the error due to the averaging of each individual output. The size of a network in a committee should normally be larger than when used independently. The reason will be explained in Chapter 5 and has to do with the bias-variance dilemma. Practically, each network should have an accuracy above 50%.

Weight factors (terms proportional to the performance of each network) can be used instead of fixed weighting for even better results. This is easy to do using the backpropagation formalism. The goal is to train just the weights from each network to the output adder (or softmax). We now demonstrate committees in NeuroSolutions

NEUROSOLUTIONS EXAMPLE 4.17

Committees

The committee is an interesting way to avoid the approach of putting "all the eggs in one basket," that is, designing a single neural network that we tweak for optimal performance. We can in principle use very different topologies (the more distinct the better, since their performances will tend to be less correlated) in a more or less adhoc fashion to improve classification results. Of course, the price paid is a large increase in the computational cost of the simulation. But these days, with the increasing power of PCs and workstations, this is a small price to pay for increased performance.

When taken to the limit, this method of design avoids some of the difficulties we found in selecting the optimal topology and setting the number of PEs, the stopping criterion, and the training parameters. Committees rely on statistics to improve individual suboptimal solutions and achieve performance comparable to the best possible solution. This is possible only because the main difficulty in neural network performance is the variance of the estimation.

Here we use on-line training for each of the individual networks. But notice that each network has its own criterion to guarantee that each network is trained independently. We use two MLPs (one-hidden-layer and two-hidden-layer) and a RBF network (another class of networks -more on this in Chapter 5) to solve the male-female classification problem. If you recall, we had about 5 errors when we solved this problem with the MLP in Chapter 3, so this is the number to beat.

We start by fixing the vote of each network to 1/3. Notice that the performance is basically the same as a single network. Then we also train the vote of each network. The performance improves dramatically, since the number of mistakes decreases to 2 misclassifications. As you can see, this is a powerful way to "throw technology at the problem."

You should create a test set to validate the improvement of the committee over a single network. Notice also that this method lends itself very nicely to parallelization, since each network can be run in a separate machine and the results combined.

NeuroSolutions Example

Another option is to train the same networks concurrently with a single error criterion, as if they were a larger modular network made up of MLPs and the like. This is effectively similar to training a much larger network and setting some of its weights to zero, that is, adding some a priori structure. Normally, each will specialize in a given portion of the pattern space. The advantage comes when the networks have different discriminant functions. But notice that this training is collaborative, not competitive (as we will exemplify later in Chapter 7), so it is not a committee. Modular networks can be easily implemented in NeuroSolutions, so you can modify the previous breadboard to create a modular network and compare the results.

Go to next section


 4.8 Project: Application of the MLP to Real-World Data

In this section we are going to use MLPs to solve several real-world classification problems. There are several internet sites that store nice data for classification purposes. Please see the Web sites in Chapter 1, and also the following:

  • http://markov.stats.ox.ac.uk/pub/PRNN

  • http://128.2.209.79/afs/cs/project/ai-repository/ai/areas/neural/bench/0.html

  • http://www.scs.unr.edu/~cbmr/research/data.html

  • http://neural-server.aston.ac.uk/NN/databases.html

  4. 8.1 Classifications

We start with the crab data, taken from an article by Campbell and Mahon. The goal is to classify rock crabs from two species as male or female using anatomic measurements of front lip, rear width, length, width, and depth. The data set is composed of 200 specimens, 50 males and 50 females from the two species.

We then move to a more demanding data set -the iris data set (Fisher) to show the importance of the classifier topology in the performance. The goal is to classify three types of iris plants (Setosa, Versicolour, and Virginica) based on measurements of sepal length, sepal width, petal length, and petal width (all in centimeters). There are 50 samples of each class for a total of 150 samples. As you can see, this problem formulation is very similar to the previous one, however the distribution of the data clusters in the input space is more complex. The difficulty of the task is not known a priori, so a step-by-step approach to define the optimal classifier is normally required.

NEUROSOLUTIONS EXAMPLE 4.18

Crab Data Classification

The first thing to do is to examine the data sets. The data is composed of 5 parameters per crab and one tag for the species, for a total of 6 inputs. The desired result is a classification of male or female, so we can either use one or two outputs. In the first case we would code male as 1 and female as 0 (or viceversa). In the second case each gender will have its own output. The desired response file is composed of two columns with pairs of values (0, 1) depending upon the class membership.

The next step is to normalize and divide the data. Normalization of the data allows us to get experience with step sizes and use systematic weight initializations. Since the data is all positive, we will normalize it between [0, 1]. We divide the data in training, test, and validation sets. The training set is used to arrive at optimal weights, the test data is used to gauge the performance of the classifier, and the validation set is used to help us stop the training at the point of best generalization.

The next important definition is the topology. The idea is always to start small, so we should start with a perceptron to see whether the machine can separate the data. Since the data is normalized between [0, 1], we will use the logistic function as the nonlinearity.

The next important decision is how to stop the training. Here we can first use the number of iterations (I = 200) to get a feel for how difficult the problem is, and then switch to cross-validation when the topology is finally chosen.

We still have to decide on the search method and the criterion. The search, will be momentum and the criterion L2.

We are ready to run the network on the crab data. Do it several times and watch the learning curve. If the network converges to the same small error in basically the same number of epochs then the training is successful, and we can worry about how to stop the training with cross-validation. If the network does not converge, we should select a different learning rate or search procedure, modify the topology or both.

The error is around 0.1, which means that almost all the data was classified correctly in the training set. We also see that the training is repeatable, since the learning curves are very similar to each other. Let us go to the test set to see the performance. As we mentioned earlier, the best way to test the classifier is not by MSE but with the confusion matrix. We see that we get 3 errors in the test set, which is reasonable. But can we improve the performance?

To answer this question, we have to change the topology and try a one-hidden-layer network with 5 hidden PEs. Let us train the network and check the performance. As you can see, the error is much smaller. Let us test the system. The confusion matrix shows that the classification is perfect. Since the MLP performed moderately better than the perceptron, we could conculde that the data is not linearly separable, but we do not have enough data to be sure. The errors could be due to imprecise measurements (noise in the data).

We can now fine-tune the system. We can first train it with cross-validation and see how far we should train. The cross-validation for this problem and topology does not help, since there is no observable increase in the learning curve.

An alternative to cross-validation is to use weight decay on the weights. We start with the same network, but we use the idea of driving all the weights to zero. Training the network, we can see that the network size was slightly reduced but that our initial guess was pretty good.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 4.19

Iris Data Classification

To see how data-dependent the performance of a classifier is, let us change to the iris data. The inputs to the network are sepal length, sepal width, petal length, and petal width (i.e., 4 inputs) and the three classes of iris plants (which are used as the desired outputs) are Setosa, Versicolour, and Virginica. There are 50 samples of each class for a total of 150 samples. The 150 samples have been prerandomized. Furthermore, 100 samples will be used for training and 50 samples for testing.

Starting with the perceptron, we see that the result is not satisfactory. There is significant confusion between the classes of Virginica and Versicolour, so we need to change to a one-hidden-layer MLP. The issue is how to choose the number of hidden PEs.

Let us run the MLP first with one PE in the hidden layer for several runs and record the minimum MSE value, then increase this number to 2, 4, and 6 and run the network several times for each. Finally, we should plot the mean MSE as a function of the number of PEs. The MSE error curve stabilizes after 2 PEs, so we will put 2 PEs in the hidden layer.

The classification performance for this network is much improved (only 3 mistakes instead of 19). We can thus conclude that the data set is not linearly separable, since the perceptron can not solve it but the one-hidden-layer MLP succeeded.

NeuroSolutions Example

4.8.2 Neural Networks for Decision Making

The next problem we will discuss is very important, especially in decision making and medical diagnosics. Assume that we want to create a computer-aided diagnostic tool for breast cancer using a neural network. Ten features (radius, texture, perimeter, area, smoothness, compactness, concavity, concave points, symmetry, and fractal dimension) have been computed from a digitized image of a fine-needle aspirate of a breast mass. The inputs to the neural network model consist of the mean, standard error, and "worst" (mean of the three largest values) for each of these 10 features, resulting in 30 total inputs for each image. We have data from only 150 images. A two-hidden-layer MLP configured with 10 inputs and 2 outputs will be used as the neural network model.

The difficulty in this application is that we want to estimate a decision (sick or healthy) based on the outcome of the neural network, that is, we have to estimate the "probability" of a decision. How can a neural network do that? We saw that the MLP can be trained to give us the a posteriori probability of a class, given the input data example. The a posteriori probability interpretation means that a net output for class 1 of 0.9 can be interpreted as saying that the a posteriori probability of that example belonging to class 1 is 0.9. This is an important step, but from here to a diagnostic many other things must be taken into consideration before we make a diagnostic decision.

First, we used 50 percent of sick and 50 percent healthy cases to train the system, but the ratio of sick to healthy in the population is (fortunately) very different, so we have to compensate for the a priori probabilities when we interpret the results. Recall from Chapter 2 (Figure 2-2) that when the a priori probabilities for each class are the same, the decision that minimizes the probability of misclassification is given directly by the likelihoods. In general, when the a priori probabilities are different, the corresponding likelihoods should be multiplied by the a priori probabilities, so the decision boundary is moved.

The advantage of ANNs is that we do not need to retrain the network. We can still interpret the outputs as likelihoods and to obtain the a posteriori probability simply multiply the outputs by the priors (Bayes' theorem). This assumes that the training set is constructed with an equal number of exemplars from each class. For example, suppose that the net output for the class sick was 0.9, we know that the probability of being sick in the population is 0.2, and equal number of cases were used in the training set. The probability that the subject has the disease is therefore 0.18. If the training set does not contain an equal number of the two classes, we should also divide the outputs by the relative frequency of training exemplars in the respective class.

The other difficulty is that there is a risk in making a decision, and the two types of errors (the subject is healthy and the network says sick, called a false positive, or the subject is sick and the network says healthy, called a false negative) have different costs. It is preferable to initially tell a healthy patient that she needs to have further exams than to simply state that she is healthy when in fact she has breast cancer, since the implications are vastly different.

This means that minimizing the probability of misclassification may not be the best strategy. We should weight the a posteriori probabilities by the risk of making the decisions. To do this, a matrix of penalties Lij of making the wrong decisions must first be constructed. Normally in medicine this is rather subjective, but the idea is as follows. We would like to penalize the false negatives (calling a sick subject healthy) much more than the false positives (calling a healthy patient sick). The penalties are numbers between 0 and 1, where the penalty for being correct is 0, while the penalty for being wrong is 1. Let us call sick hypothesis 1, and healthy hypothesis 2. Here let us say that the penalty of the false negative is L12 = 0.3 and the penalty of the false positive is L21 = 0.5. The matrix of penalties is therefore

We compute the average penalty by

where c is the number of classes and x is the random variable. We call risk the expected value of the penalties, i.e.

where P(Ci) are the a priori probabilities. Thus the best decision should be done by minimizing the risk (instead of minimizing the probability of misclassification, as we did when we applied Bayes' rule), as

Notice that when the network provides p(x|Ci) it is trivial to make the decision based on risks.

NEUROSOLUTIONS EXAMPLE 4.20

Risk Decision in the Cancer Data

To solve this problem we start with a MLP and make sure that the network is well configured (try several numbers of hidden PEs, as we did earlier) and well trained. We should use cross-validation to stop at the point of maximum generalization. We should also try different initial conditions to make sure that we are really converging to the absolute minimum.

Then the next step is to create a DLL that will compute the risk based on the preceding formulas. The computer-aided diagnostic is then made based on the class that produces the smallest risk.

NeuroSolutions Example

Go to next section


4.9 Conclusion

 4.9 Conclusions

In this chapter we covered the most fundamental practical aspects of applying MLPs to real-world problems. We learned about new search procedures, how to control the initialization, the stopping criterion, and how to decide effectively whether a reasonable solution was obtained (the confusion matrix).

We also discussed ways to build more flexibility into the solution in the form of different norms. One of the fundamental problems of MLPs (and other learning machines) is that they cannot control their generalization ability. This is crucial for good results, so we presented a method (weight decay) to prune unnecessary weights to provide better generalization. Cross-validation was also presented as the method of choice to stop network training, because it ensures the best possible generalization for the chosen topology. We also introduced the idea of committees of networks as a way to decrease the variance of performance in the test set. We ended the chapter with the application of MLPs to several interesting real-world problems.

We now present a summary of heuristics that will help decrease the training times and produce better performance in general. For more information see Orr and Muller.

  • Normalize your data to the range of the network activations.

  • Use the tanh nonlinearity instead of the logistic function.

  • Use a softmax PE at the output layer.

  • Normalize the desired signal to be just below the output nonlinearity "rail" values (i.e., if you use the tanh, use desired signals of ± 0.9 instead of ± 1).

  • Add a constant value of 0.05 in the derivative of the nonlinearity (errors will always flow through the dual network).

  • Set the step size higher in the layers closer to the input.

  • Shuffle the training set from epoch to epoch in on-line learning.

  • Initialize the network weights in the linear region of the nonlinearity (choose the standard deviation of the random noise source as in Eq. 4.2).

  • Use more sophisticated learning methods (delta bar delta, added noise, conjugate gradients).

  • Always have more training patterns than weights. You can expect the test-set performance of your MLP to be limited by the relation N > W/ [image], where N is the number of training patterns, W the number of weights, and [image] the performance error. You should train until the mean square error is less than [image] /2.

  • Use cross-validation to stop training.

  • Always run the network several times to gauge performance.

  • Use a committee of networks to improve classification.

Go to next section


 5.1 Introduction

In Chapters 1 and 3 we presented two of the most common applications of adaptive systems, which are respectively linear regression using a linear adaptive system (the Adaline), and classification using the multilayer perceptron (MLP). We saw that the nature of the applications is different, since in regression the problem is one of representing the relationship between the input and the output data, while in classification the input data is assumed to be multi-class, and the purpose is to separate it into classes as accurately as possible. We also verified that the machinery developed for regression, namely gradient descent on a cost function, could be applied to classification. When properly extended the gradient-descent procedure gave rise to the backpropagation algorithm developed to train the MLP.

The purpose of this chapter is to more formally unify the two applications of regression and classification. We will be demonstrating that both problems are in fact aspects of the more general problem of function approximation. Linear regression is function approximation with linear topologies, and classification is function approximation with special functions called indicator functions. What we gain is a very broad and unifying perspective of adaptive systems that can be applied to many practical applications beyond classification. These applications range from system identification to data modeling and motivate the study of the MLP as a nonlinear regressor. This new perspective also raises the question of alternative topologies such as the radial basis function (RBF) network, which is another universal approximator that can also be used for classification. Later in the chapter we present different methodologies to study the problem of generalization, we conclude with a brief introduction to support vector machines. To achieve this unifying view, we will present the basic concepts of function approximation.

5.1.1 The Discovery of the Input-Output Map as Function Approximation

We demonstrated in Chapters 1 and 3 that a neural network combines a set of inputs to obtain an output that mimics the desired response. Given a set of input vectors x and a set of desired responses d, the learning system must find the parameters that meet these specifications. This problem can be framed as function approximation, if we assume that the desired response d is an unknown but fixed function of the input d = f(x) (Figure 5-1).

Figure 5-1 Supervised training as function approximation

The goal of the learning system is to discover the function f(.) given a finite number (hopefully small) of input-output pairs (x,d). The learning machine output [image] depends on a set of parameters w, which can be modified to minimize the discrepancy between the system output y and the desired response d. When the network approximates d with y, it is effectively approximating the unknown function f(x) with its input-output map [image].

The nature of f(.) and the error criterion define the learning problem. As studied in Chapter 1, linear regression is obtained when the error criterion is the mean square error (MSE) and f(.) is linear. Classification, studied in Chapter 2, specifies functions f(.) that produce 1, -1 (or 0) which are called indicator functions.

The problem of generalization, briefly discussed in Chapter 4, can also be treated mathematically with this view of function approximation. This means that the ideas embodied in Figure 5-1 are rather relevant for the design of learning machines, specifically neural networks. Neural networks are in fact nonlinear parametric function approximators, so we should not think of them simply as classifiers.

ANNs are useful for function approximation because

  • They are universal approximators

  • They are efficient approximators

  • They can be implemented as learning machines

We already alluded to the universal approximation property of the MLP in Chapter 3. It basically says that any continuous function can be approximated by the MLP topology, provided that enough PEs are available in the hidden layer. Here we will present these concepts more precisely.

With neural networks the coefficients of the function decomposition are automatically obtained from the input-output data pairs and the specified topology using systematic procedures called the learning rules, so there is no need for tedious calculations to obtain the parameters of the approximation analytically. Once trained, the neural network becomes not only a parametric description of the function but also its implementation. Neural networks can be implemented in computers or analog hardware and trained on-line. This means that engineers and scientists now have a means to solve function approximation problems involving real-world data. The impact of this advance is to take function approximation out of the mathematician's notebook and bring it to industrial applications.

We would like to argue that neural networks and learning are bringing focus to a very important problem in the scientific method called induction. Induction along with deduction are the only known systematic procedures to build scientific knowledge. Deduction applies general principles to specific situations. Deduction is fairly well understood and has had enormous impact on the entire fabric of mathematics, engineering, computer science, and science in general. For instance, deductive reasoning is the core of artificial intelligence. On the other hand, induction is poorly understood and less frequently applied. Induction is the principle of abstracting general rules from specific cases. As we all know from real life, this principle is much harder to apply with validity than deduction. Sometimes true statements in a small set of cases do not generalize. Mathematically, induction is also much less formalized.

It turns out that a neural network uses an inductive principle when it learns from examples. Examples are specific instances of a general rule (the function that created the examples), and the goal of neural network learning is to seek the general principle that created the examples. Theoretically, these issues are studied in learning theory. The difficulties we face in appropriately training a neural network are related to the difficulties of inducing general principles from examples. In practice, the ANN is not always able to capture the rule, and the prerequisites (neural network architecture, training data, stopping criterion) to extrapolate from examples need to be carefully checked as we saw in Chapter 4.

Go to next section


 5.2 Function Approximation

Function approximation seeks to describe the behavior of very complicated functions by ensembles of simpler functions. Very important results have been established in this branch of mathematics. Here we discuss only a few that bear a direct relation on our goal of understanding neural networks better. Legendre (and Gauss) used polynomials to approximate functions. Chebyshev developed the concept of best uniform approximation. Weierstrass proved that polynomials can approximate any continuous real function in an interval arbitrarily well. Series expansions (i.e., Taylor series) have been utilized for many years to approximately compute the value of a function in a neighborhood of the operating point. The core advantage is that only multiplications and additions are necessary to implement a series approximation. Trigonometric polynomials are also widely used as function approximators, but their computation is a bit more involved. Before we apply any of this theory, we will formalize the concept of function approximation.

Let f(x) be a real function of a real-valued vector x = [x1 x2 … xd] [image] that is square integrable (over the real numbers). Most real-world data can be modeled by such conditions. We are also going to restrict this study to the linear projection theorem. The goal of function approximation using the projection theorem is to describe the behavior of f(x), in a compact area S of the input space, by a combination of simpler functions [image] i(x):

(5.1)

where wi are real-valued entries of the coefficient vector w = [w1 w2 … wn] such that

(5.2)

and where [image] can be made arbitrarily small. The function [image] is called an approximant to f(x). The block diagram of Figure 5-2 describes this formulation.

Let us examine Eq. 5.1 and Eq. 5.2. A real function is a map from the input domain to the real numbers. This expression states that one can obtain the value of the function when x is in S by using an intermediate set of simpler functions, { [image] i(x)}, called the elementary functions, and then linearly combining them (Figure 5-2).

Figure 5-2 Implementation of the projection theorem

When one can find coefficients wi that make [image] arbitrarily small for any function f(.) over the domain of interest, we say that the elementary function set { [image] i(.)} has the property of universal approximation over the class of functions f(.), or that the set of elementary functions [image] i(x) is complete. From Eq.5.1 we see that there are 3 basic decisions in function approximation:

  • The choice of elementary functions [image] i(.)

  • How to compute the weights wi

  • How to select the number of elementary functions N

The first problem is very rich because there are many possible elementary functions that can be used. We will illustrate this later, and we will show that the hidden PEs of a single hidden-layer MLPs implement one possible choice for the elementary functions [image] i(.). The second problem is how to compute the coefficients wi, which depends on how the difference or discrepancy between f(x) and [image] is measured. In Chapter 1 we have already presented one possible methodology to solve this problem for the case of the minimization of the power of the error between [image] and f(x). Least squares can also be used to analytically compute the values for wi. If the number of input vectors xi is made equal to the number of elementary functions [image] i(.), the normal equations can be written as

(5.3)

and the solution becomes

(5.4)

where w is a vector with the coefficients, f is a vector composed of the values of the function at the N points, and [image] a matrix with entries given by the values of the elementary functions at each of the N points in the domain. An important condition that must be placed on the elementary functions is that the inverse of [image] must exist such as computational efficiency.

In general, there are many sets { [image] i(.)} with the property of universal approximation for a class of functions. We would prefer a set { [image] i(.)} over another { [image] i(.)} if { [image] i(.)} provides a smaller error [image] for a preset value of N . This means that the speed of convergence of the approximation (i.e., how fast the approximation error [image] decreases with N) is also an important factor in the selection of the basis. Other considerations may be imposed by the computer implementation.

5.2.1 Geometric Interpretation of the Projection Theorem

Let us provide a geometric interpretation for this decomposition, because it exemplifies what is going on and what we try to accomplish. As long as the function f(.) is square integrable and D is finite, this geometric representation is accurate. Consider x as a given point in a D-dimensional space. Its transformation by f(.) is also assumed to be another point in the same D-dimensional space. We can alternatively think of x and f(x) as vectors, with end points 0 and x. For illustration purposes let us make D = 3 and assume that we only have two elementary functions.

Eq.5.1 and Eq.5.2 describe the projection of the vector f(x) into a set of basis functions [image] i(x). These basis functions can also be considered vectors, and they define a manifold (i.e., a projection space) in M ( [image]) dimensions, which is linear in our formulation. is the image or projection of f(x) in this manifold. In this example the projection manifold is a plane (M = 2) depicted as the horizontal plane, and [image] is a vector that exists in the horizontal plane. We can interpret wi as the magnitude of (or proportional to) [image] along each one of the axes of the manifold.

If f(x) belongs to the manifold, then there is always a set of constants wi that will make [image] exactly equal to f(x). Figure 5-3 represents this in case A. If f(x) does not belong to the manifold created by the basis { [image] i(x)}, then there will always be an error between [image] and f(x) (case B). The best solution (least possible error) found in Eq.5.4 is the orthogonal projection of f(x) onto the manifold. As we saw in Chapter 1 this is exactly the solution that the least squares provide, since the error becomes orthogonal to all the bases { [image] i(x)}.

Figure 5-3 Approximation as a projection. A) Vector is in the manifold. B) Vector is outside the manifold.

When f(x) is external to the projection manifold, decreasing the error means making [image] closer to f(x). This can be accomplished by increasing the number of elementary functions (i.e., the dimension M of the manifold) because the manifold will fill more and more of the available signal space. This view is correct, provided that the basis set is complete, that is, that in the limit of large M the projection manifold will fill all the available signal space.

Let us now study in more detail each one of the steps in function approximation. We will see throughout this study that we obtain a very different view of what the MLP is, and we will tie this topology with other very well known basis functions.

Go to next section


 5.3 Choices for the Elementary Functions

One decisive step in function approximation is the choice of the elementary functions [image] i(.) because they affect how close [image] can be made to f(x) for a given N. If the choice is not appropriate, there will be a non vanishing error between [image] and f(x), no matter how big N is. The search for sets of elementary functions { [image] i(.)} that are universal approximators of a class of functions f(.) is therefore very important. Moreover, we would like the elementary functions [image] i(.) to have nice mathematical properties and to be easy to work with.

One requirement for the usefulness of elementary functions is that [image] must exist (Eq.5.4). This condition is met if the elementary functions constitute a basis, that is, if they are linearly independent or

(5.5)

A simplifying assumption that is often imposed on the elementary functions is that the basis be orthonormal, that is,

(5.6)

where [image] (x) is the Dirac delta function. This means that in orthogonal decompositions the projection of a basis in any other basis is always zero. An orthonormal basis is very appealing because we can evaluate the projection on each basis independently of the projection on the other bases, and they provide a unique set of wi for the projection of f(x). Many elementary functions obey the orthogonality conditions, and so different sets may provide different properties.

With complete orthonormal basis functions the weights of the decomposition become very simple to compute. It can be shown that (calculation of the weights in orthonormal bases)

(5.7)

where <.> is the inner product of f(x) with the bases, given by

(5.8)

and D is the domain where f(x) is defined.

5.3.1 Examples of Elementary Functions

Many important function approximation theories and decompositions are commonly applied in engineering. One example is digital signal processing that relies heavily on sampling theorem. The sampling theorem shows that one can approximate any real smooth signal (i.e. a function with finite slope) in an interval (infinitely many points) by knowing the functional values only at a finite set of equally spaced points in the interval (called the samples). The value of the signal at any other point in the interval can be exactly reconstructed by using sums of sinc functions. In this case the bases are the sinc functions, and the weights are the values of the signal at the sampling points, as shown in Figure 5-4.

Figure 5-4 Decomposition by sinc functions

This result opened up the use of sampled representations to reproduce sound (digital music on CDs) and to reproduce images (the forthcoming digital TV), and is the basis for the very important field of digital signal processing. sinc decomposition

NEUROSOLUTIONS EXAMPLE 5.1

Sinc Interpolation

Here we use NeuroSolutions to interpolate an input waveform to a higher frequency using the sinc function. This example is not as dramatic as the one that produces a continuous representation from a digital sequence, but it is based on the same principles. We start with a digital waveform representing a ramp, and introduce two zero samples between each two consecutive points as shown in the input Megascope. As we can expect, the ramp becomes distorted. The idea is to recreate the ramp by filling in the missing values. We do this by designing an interpolator that implements a close approximation of the sinc function. We use a new component, the Delay Line, and enter in the Synapse the values that correspond to a sampling of the sinc function.

NeuroSolutions Example

Another example of the power of function approximation is the Fourier series. Fourier series are an example of expansions with trigonometric polynomials. Everybody in engineering has heard of frequency representations (also called the spectrum) because of the following amazing property. Any periodic function (even with discontinuities) can be approximated by sums of sinusoids, as shown in Figure 5-5. Moreover, there are simple formulas that allow us to compute the components in the frequency domain from any time signal. Fourier formulas

Figure 5-5 Decomposition by sine waves

NEUROSOLUTIONS EXAMPLE 5.2

Fourier Decomposition

This example is a demonstration of how an addition of sinusoids does in fact produce a waveform that resembles a square wave. To compute the coefficients, we have to perform a Fourier series decomposition of the square wave, which is not difficult but is cumbersome and requires an infinite number of sinusoids. By including more and more terms of the Fourier series, we make the composite waveform closer and closer to the square.

NeuroSolutions Example

Wavelets are yet another example of elementary functions used for function approximation. One of the problems with the Fourier decomposition is that the sine waves have infinite extent in time, that is, they exist for all time. In many practical problems we would like to decompose signals that have a finite extent (transients), in which case the Fourier analysis is not very efficient. Wavelets provide such a decomposition for transients. The idea is to choose a wave shape that is appropriate to represent the signal of interest (the mother wavelet), and create many translated and scaled versions of the wavelet so that one can reconstruct the desired signal.

The wavelet expansion uses a two-parameter decomposition:

(5.9)

where the [image] are the wavelet bases. The bases are obtained from a single function (the mother wavelet [image] (x)) by the operations of scaling and translation,

(5.10)

hence the two indices. Figure 5-6 shows the scaling and translation operations.

Figure 5-6 Translation and scaling for a wavelet

All the preceding methods construct arbitrary functions by weighting the contributions of predetermined elementary functions (sine waves of different frequencies, translation of sincs, or the dilation-translation of the mother wavelet). When decomposing the same function using different bases, only the weights of the decomposition vary. Normally, there are closed-form solutions to compute the weighting from the signal. In neurocomputing the problem is more complicated for two reasons: First, we want to find the coefficients through adaptation instead of through analytic formulas as in the Fourier case; second, the bases themselves are dependent on the data and the coefficients (adaptive bases).

In some situations the basis can be chosen naturally from the type of problem being investigated, as in linear systems theory.

5.3.2 Bases for Linear Function Approximation

When the unknown function f(x) in Figure 5-1 is linear, the linear regression concepts explained in Chapter 1 can be applied to construct an approximation.

NEUROSOLUTIONS EXAMPLE 5.3

Linear Regression

Here we show that through adaptation we can find the coefficients of a very simple linear transformation between x and d of Figure 5-1. The transformation is simply

d=2x+3

For this case a linear system constructed from a Synapse and a Bias Axon can solve the problem very easily. This is simply linear regression which we studied in Chapter 1. We create the transformation by applying one of the function generators to the input of the system and using another function generator at the output, producing the same wave shape but with twice the amplitude and with a bias of 3. Then we let the system adapt using the LMS rule. We can see that the system very quickly finds the relationship, the synaptic weight becomes 2 and the bias becomes 3.

NeuroSolutions Example

Linear functions are much more complicated than the previous example. For linear systems with constant coefficients there is actually a preferred choice for the elementary functions. Linear system theory shows that the natural bases are the complex exponentials e [image] because they are complete for square integrable functions and they are the eigenfunctions of linear shift-invariant operators. eigendecomposition

The implication of this fact is thoroughly explored in linear systems, which are networks that implement signal decomposition using complex exponentials. We use eigendecompositions in Chapter 9 when we study adaptive filters. Here we just remark that eigendecompositions are the most efficient, since we are constructing a function from its "elementary pieces", so the reconstruction error can be made equal to zero with a small number of bases. Sometimes other considerations, such as ease of implementation, may overshadow the use of complex exponentials.

5.3.3 Bases for Nonlinear System Approximation: The MLP Network

When the function f(x) in Figure 5-1 is nonlinear, there is in general no natural choice of basis functions. Many have been attempted, such as the Volterra expansions, the splines, and the polynomials [Haykin]. Weierstrass proved that polynomials are universal approximators. (see Weierstrass Theorem) The problem is that either many terms are necessary or the approximations are not very well behaved. One of our requirements is that the bases have to be powerful and easy to work with.

In neurocomputing there are two basic choices for the elementary functions that build the approximant [image]: local and global elementary functions. An elementary function is global when it responds to the full input space, while local elementary functions respond primarily to a limited area of the input space. It is easy to link the operation of function approximation (Figure 5-2) to a neural topology such as a one-hidden-layer perceptron with a linear output, where [image] i(x) is

(5.11)

and [image] is one of the sigmoid nonlinearities (logistic or tanh). The system output is given by y = [image] wi [image] i . Note that the first-layer weights are denoted by aik and they change the position and size of [image] i(x). The one-hidden-layer MLP with a linear output PE can thus be thought of as an implementation of a system for function approximation ( Eq.5.1), where the bases are the outputs of the hidden PEs. Note that the sigmoid PE responds to the full input space x with a nonzero value (1, -1 (or 0), or intermediate values), so the MLP implements an approximation with global elementary functions.

The interpretation is that the MLP is performing function approximation with a set of adaptive bases that are determined from the input-output data. This means that the bases are not predefined as in the sinc, wavelet, or Fourier analysis, but depend on the first-layer weights and the input. In this respect the MLP is much closer to the function approximation implemented by adaptive linear systems. Function approximation with adaptive bases is thus different than the picture we previous presented in Figure 5-3. First, the bases change with the data, which means that the projection manifold is data dependent, and second, the weights in the network layers perform different functions. The input-layer weights change the bases by orienting the manifold, while the output-layer weights find the best projection within the manifold. Training will find the set of weights (aij) that best orient the manifold (first layer weights) and that determine the best projection (wij). Therefore the training is more difficult because not only the projection but also the basis is changing. However, we can obtain much more compact representations.

This view should be compared with the description of MLPs for classification in Chapter 3. Each PE in the hidden layer creates a discriminant function with a shape defined by the PE nonlinearity and with an orientation and position defined by the first-layer weights. Therefore the views agree, but in function approximation the PEs are less prone to saturate. Due to the highly connected topology and the global nature of the elementary functions, good fitting is obtained with reasonably few bases (i.e., few PEs). However, the training is difficult because the basis functions are far from orthogonal. multiple-hidden-layer MLPs

In terms of function approximation the one-hidden-layer MLP decides the orientation, placement, and relative amplitude of a set of multidimensional sigmoid functions (one per PE). This function decomposition is shown in Figure 5-7 and resembles the approximation obtained with step functions well known in linear systems.

Figure 5-7 Function approximation with logistic functions

NEUROSOLUTIONS EXAMPLE 5.4

Function Approximation with the MLP

From this point on,, many of the examples study the function approximation abilities of various networks. We have chosen a fourth order polynomial as our target function. The polynomial was chosen to give an interesting shape over the input range of 0..1 and has the equation 27x4 - 60x³ + 39x² - 6x. The graph of the polynomial from 0.1 is shown in the figure.

We will use an MLP with a linear output to approximate the preceding function. In this case the MLP approximates the function with tanh bases (the hidden-layer PEs are tanh). These elementary functions are stretched and moved over the range and then summed together to approximate the polynomial.

NeuroSolutions Example

MLPs are universal approximators as we stated in Chapter 3. The proof is based on an extension of the Weierstrass theorem outline of proof. But as we stated earlier, another important characteristic to study is how the error decreases with the order or the dimension of the problem, or both. The importance of MLPs for function approximation was recently reinforced by the work of Barron. He showed that the asymptotic accuracy of the approximation with MLPs is approximately independent on the dimension of the input space. This is unlike approximation with polynomials, where the error convergence rate is exponentially related to the number of dimensions of the input (the error decreases exponentially with the dimension of the input space). This means that MLPs become much more efficient than polynomials for approximating functions in high-dimensional spaces. The better approximation properties of MLPs explain why MLPs are more efficient than other methodologies for classification and why they are key tools in identification of nonlinear systems, as we will see in Chapter 10 and 11.

What are the differences between using a MLP for function approximation and for classification? The obvious answer is to say that for function approximation the output is linear while for classification the output PE must be nonlinear. In fact, we can also use a nonlinear PE for function approximation if we carefully set the dynamic range of the output, so the difference is not solely in the output PE, but also in the nature of the problem. In function approximation the operating point of the hidden PEs is normally far away from saturation, since the mappings tend to be smooth. In classification, where the outputs are 1 and 0, the operating point of the hidden PEs is normally driven to saturation. This is easily observed when we use a square wave as the desired signal, because this choice implements an indicator function.

NEUROSOLUTIONS EXAMPLE 5.5

MLP to Approximate a Square Wave (Classification)

In this example we use an MLP with a tanh output to approximate a square wave. Notice that, since a square wave is either on or off, this function approximation problem is identical to a classification problem. Thus classification is a subset of function approximation with the desired signal having on/off characteristics. The important point to show here is that when doing classification, the PEs become saturated and the weights increase greatly. This allows the tanh or logistic function to approximate the on/off characteristics of the desired signal. Thus, for classification, the MLP tends to operate in the saturated regions of the hidden PEs (on/off), while for general function approximation the hidden PEs tend to operate closer to the linear region.

NeuroSolutions Example

5.3.4 Alternative Basis for Nonlinear Systems: The RBF Network

In neurocomputing, the other popular choice for elementary functions is the radial basis functions (RBFs), where [image] i(x) becomes

(5.12)

where [image] (.) is normally a Gaussian function

one dimensional

multidimensional (5.13)

with variance [image] ² or covariance [image]. Notice that the Gaussian is centered at xi with variance [image] ², so its maximum response is concentrated in the neighborhood of the input xi, falling off exponentially with the square of the distance. The Gaussians are thus an example of local elementary functions. If we plug Eq.5.13 in Eq.5.1 we obtain the following implementation for the approximant to the function f(x):

(5.14)

which implements the input-output map of the RBF network.

Let us think of an arbitrary function and a set of localized bell shaped functions (of the Gaussian shape). Function approximation in a limited area of the input space (see Figure 5-8) requires:

  • the placement of the localized Gaussians to cover the space

  • the control of the width of each Gaussian

  • the setting of the amplidude of each Gaussian.

If we can accomplish these three tasks we can approximate arbitrary continuous functions with an RBF network. approximation properties of RBF

Figure 5-8 Approximation by RBFs in one-dimension

The RBF and the MLP achieve the property of universal approximation with different characteristics, since the basis functions are very different. In the RBF the bases are local, so each can be changed without disturbing the approximation of the network in other areas of the space. But we need exponentially more RBFs to cover high-dimensional spaces (the curse of dimensionality). In the MLP this is not the case, as we mentioned in the result by Barron. As we saw in Chapter 3, changing one MLP weight has the potential to produce drastic changes in the overall input-output map. This has advantages in some aspects, such as more efficient use of PEs, but also disadvantages, since the training becomes slower and the adaptation can be caught in local minima.

RBFs train very efficiently once the centers are determined, since the error is linear in the weights. This fact also guarantees the convergence for the global minimum (if the centers are optimally set). This makes RBFs very useful for system identification.

From the theoretical point of view of function approximation, RBFs possess the property of best approximation as defined by Chebyshev, unlike the MLP (i.e., there is always an RBF that provides the minimum error for a given function to be approximated).

NEUROSOLUTIONS EXAMPLE 5.6

Function Approximation with RBFs

Now we solve the same polynomial approximation problem with a radial basis function network. We can vary the number of RBFs and see how this affects the power of the network to approximate the given polynomial.

NeuroSolutions Example

Go to next section


 5.4 Probabilistic Interpretation of the Mappings: Nonlinear Regression

So far we have assumed a deterministic framework to study the input-output mapping. It enhances our understanding to look now at the mappings discovered by MLPs and RBFs from a statistical perspective. The result presented next is valid as long as the mean square error (MSE) criterion is utilized in the training.

We assume that the input data is a random variable x, and the desired response t is also a random variable, not necessarily Gaussian distributed. The topology is an MLP or an RBF with a linear output PE, as we have been discussing. The important result is the following: A network with weights obtained by minimizing the MSE has an output that approximates the conditional average of the desired response data tk, that is, the regression of t conditioned on x:

(5.15)

where w* means the optimal weights, and <<.>> refers to the conditional average defined by

(5.16)

This result was demonstrated in Section 3.8 (derivation of the conditional average) The MLP and RBF networks are thus nonlinear regressors, extending the Adaline for cases where the input-output map is nonlinear. They will be able to "discover " any deterministic input-output relationship corrupted by additive zero-mean noise, since the network output will approximate the average of the desired response. The only requirements are that the network has converged to the global minimum, that the number of degrees of freedom in the network topology is large enough, and that there is enough data to train the system. These are nontrivial issues, but we have learned ways to cope with them in Chapters 3 and 4.

NEUROSOLUTIONS EXAMPLE 5.7

Nonlinear Regressors

We illustrate this important point by creating a nonlinear mapping problem corrupted by additive noise. We again use the polynomial approximation case and add noise to the desired signal. Since the network output can be thought of as the average of d with respect to the distribution p(d|xi) at a given point xi of the domain, the network should clean the noise and produce the polynomial. This clearly shows that the MLP is doing regression but now with nonlinear mappings. You can also use the RBF to produce the same result, since this behavior is derived from the use of the MSE criterion and is independent of the topology.

NeuroSolutions Example

Go to next section


 5.5 Training Neural Networks for Function Approximation

5.5.1 Training MLPs for Function Approximation

An important problem that needs to be solved in applying neural networks for function approximation is a procedure to automatically find the coefficients from the data. Notice that the backpropagation algorithm studied in Chapter 3 solves this problem. In fact, straight backpropagation minimizes the error power between the desired response and the system output (the L2 norm). The backpropagation algorithm is independent of the fact that in function approximation the network has a linear output and the formulation uses the absolute value of the error instead of the error power. In fact, we saw how to integrate backpropagation with arbitrary norms in Chapter 4, so we can also solve Eq. 5.2. L1 versus L2

NEUROSOLUTIONS EXAMPLE 5.8

MLPs for Function Approximation with L1 Norm

We again show the MLP network approximating the function of Example 5.4 except that this time the L1 criterion is used. In theory, this should produce a better fit (uniform approximation) to the data but may train more slowly.

NeuroSolutions Example

5.5.2 Adapting the Centers and Variances of Gaussians in RBFs

Backpropagation can be applied to arbitrary topologies made up of smooth nonlinearities, so it can also train the newly introduced RBFs. However, there are other procedures to adapt RBF networks that are worth describing. One simple (but sometimes wasteful) approach to assigning the Gaussians is simply to uniformly distribute their centers in the input space. This was the method used in Example 5.3. Although this may be a reasonable idea for approximation of complicated functions that cover the full input space, it is not recommended in cases where the data is clustered in certain areas of the input space. There are basically two ways to select the positioning and width of the Gaussians in RBFs: the supervised method and self-organization.

The supervised method is a simple extension of the backpropagation idea for the RBF network. In fact. the Gaussian is a differentiable function, so errors can be backpropagated through it to adapt [image] and [image] in the same way as done for tanh or sigmoid nonlinearities. The backpropagation algorithm can theoretically be used to simultaneously adapt the centers, the variance, and the weights of RBF networks. The problem is that the method may provide suboptimal solutions due to local minima (the optimization is nonlinear for the centers and variances).local minima for Gaussian adaptation

The self-organizing idea is very different. It divides the training phase into the independent adaptation of the first layer weights (i.e., the location and width of the Gaussians), followed by a second step that only adapts the output weights in a supervised mode, keeping the first layer frozen. The idea is appealing because it treats the adaptation of the centers and variances as a resource-allocation step that does not require external labels. This means that only the input data is required in this step. Since training the hidden layer with gradient methods is time-consuming, the self-organizing method is more efficient.

The clusters of the data samples in the input space should work as attractors for the Gaussian centers. If there is data in an area of the space, the system needs to allocate resources to represent the data cluster. The variances can also be estimated to cover the input data distribution, given the number of Gaussians available. This reasoning means that there is no need for supervised learning at this stage. The shortcoming is that a good coverage of the input data distribution does not necessarily imply good results.

Once the centers and variances are determined, the simple LMS algorithm presented in Chapter 1 (or the analytic method of the least squares) can be used to adapt the output weights, since the adaptation problem is linear in the weights. Let us describe the algorithms that adapt the centers of the Gaussians and their variances.

Gaussian Centers

The goal is to center the Gaussians on data clusters. There are many well-known algorithms to accomplish this task (see Haykin). Here we will address only the K-means and its on-line implementation, the competitive learning algorithm.

In K-means the goal is to divide the N input samples into K clusters, minimizing the final variance. The clusters are defined by their centers ci. First a random data assignment is made, and then the goal is to partition the data in sets Si to minimize the Euclidean distance between the data partition Ni and the cluster centers ci:

(5.17)

where the data centers ci are defined by

(5.18)

K-means clustering requires a batch operation where the samples are moved from cluster to cluster to minimize Eq. 5.17. An on-line version of this algorithm starts by asking which center is closest to the current pattern xn. The center that is closest, denoted by c*j, wins the competition, and it is simply moved incrementally toward the present pattern xn:

(5.19)

where [image] is a step size parameter. We recommend that an annealing schedule be incorporated in the step size. The c* are the weights of the layer preceding the RBFs. This method is fully described in Chapter 7.

Variance Computation

To set the variance, the distance to the neighboring centers have to be estimated. The idea is to set the variance of the Gaussian to be a fraction (for example, approximately ¼) of the distance among clusters. This ensures adequate coverage of the input space between the clusters. The simplest procedure is to estimate the distance to the closest cluster,

(5.20)

where wkj represent the weights of the kth PE that is closest to the ith PE. In general, the distances to more neighbors (P) provides a more reliable estimate in high-dimensional spaces so the expression becomes

(5.21)

where the P nearest neighbors to the [image] PE are chosen.

NEUROSOLUTIONS EXAMPLE 5.9

Training RBFs for Classification

In this example we train an RBF network using the competitive learning approach. We use a new Synapse called the Competitive Synapse, which clusters the centers of the RBFs where most of the data resides. Notice that the Gaussian Axon will appear "cracked," meaning that the data flow is interrupted. This is implemented in the simulator because there is no point in adapting the top-layer weights until the centers are properly placed over the input data.

NeuroSolutions Example

Go to next section


 5.6 How to Select the Number of Bases

The selection of the number of bases is rather important. If too few bases are used, the approximation suffers throughout the domain. At first we might think that for better approximation more and more bases are needed, but in fact this is not so. In particular, if the bases are orthogonal, more bases mean that the network has the potential to represent a larger and larger space. If the data does not fill the input space but is corrupted by white noise (white noise always fills the available space), the network starts to represent the noise also, which is wasteful and provides suboptimal results. Let us illustrate this with NeuroSolutions.

NEUROSOLUTIONS EXAMPLE 5.10

Overfitting

This example demonstrates that when the data is noisy, too many bases distort the underlying noiseless input-output relationship. We use an RBF to approximate the polynomial, but instead of doing it without noise as before, we are going to add random noise to the desired signal. We then change the number of bases and the width of the Gaussians. We see that for larger networks the noise begins to contaminate the discovered function. We also observe that if the network doesn't have enough degrees of freedom, the approximation is inadequate and oversimplified.

NeuroSolutions Example

Experience shows that the problem is not just one of the pure size of the network, but the values of the coefficients are also very important. Learning therefore complicates the matter of selecting the number of bases. Effectively, this is the same problem that was encountered in selecting the size of the MLP for classification. Here we revisit the problem, presenting a statistical view and then offering two approaches to deal with it: penalizing the training error and using regularization. Although this problem was briefly treated in Chapter 4, here we provide a more precise view of the problem and relate the findings to the previous techniques.

5.6.1 The Bias-Variance Dilemma

The optimal size of a learning machine can be framed as a compromise between the bias and variance of a model. We will address this view fully in the next section, so here we will motivate the arguments with a simple analogy. Let us use polynomial curve fitting to exemplify the problem faced by the learning machine. A polynomial of order N can exactly pass through N + 1 points, so when a polynomial fits a set of points (fiducial points) two things can happen. If the polynomial degree is smaller than the number of points, the fitting may have errors (model bias) because there are not enough degrees of freedom to pass the polynomial through every point (left panel of Figure 5-9). For example, the linear regressor, which is a first-order polynomial, produces errors at nearly every point of a quadratic curve (second-order polynomial). At the other extreme, if the order of the polynomial is higher than the number of fiducial points, the polynomial can pass exactly through every point. The problem is that for the other points in the domain the polynomial was not constrained, and thus its values can oscillate widely between the fiducial points (model variance), as illustrated in the right panel of Figure 5-9. The best solution is to find an intermediate polynomial order that will provide low bias and low variance across the domain.

Figure 5-9 Under and overfitting polynomials to a set of points

This simple example provides a conceptual framework for the problem of generalization encountered in learning machines.

  • The fiducial points are the training samples.

  • The full domain represents all the possible test data that the learning machine will encounter.

  • The polynomial becomes the input-output functional map created by the learning machine.

  • The learning machine weights are equivalent to the coefficients of the polynomial.

  • The size of the polynomial is the number of weights.

Therefore we can see that for a good fit over the entire domain, both the size of the network and the amount of training data are relevant.

The model bias is the error across the full data set, which can be approximated by the error in the training set. Given a set of training samples, the learning machine will try to approximate them to minimize the training-set classification error. If the complexity of the machine is low (few parameters) the error in the training set is high, and performance in the test set will also suffer, meaning that the separation surfaces have not been properly placed (Figure 5-9 left). If the machine complexity is increased, the training-set error will decrease, showing a smaller model bias.

Too large a model produces an exact fit to the training set samples (memorization of the training samples) but may also produce large errors in the test set. The source of this test-set error for larger machines (Figure5.9 right) differs from the small-machine case. It is produced by the model variance, that is, using parameters fine-tuned for a specific subset of samples (training samples) that do not generalize to a different set of samples. This is the reason that the committees, which basically reduce the variance through weighted averaging, improve the test-set performance.

The difference in performance between the training and the test set is a practical measure of the model variance. We can always expect that the error in the test set will be larger than in the training set. However, a large performance difference between the training and test sets should be a red flag indicating that learning or model selection, or both, was not successful.

This argument means that the goal of learning should not be a zero error in the training set. It also clearly indicates that information from both the training and test sets must be used to set a compromise between model bias and variance appropriately. This is why we presented cross-validation in Chapter 4 as the best way to stop the training of a learning machine, since cross-validation brings the information from the unseen samples to stop training at the point where the best generalization occurs.

5.6.2 The Bias-Variance Dilemma Treated Mathematically

The problem of generalization can be studied using a statistical framework by interpreting the network as a regressor and decomposing the output error into its bias and variance.

A measure of how close the output is to the desired response is given by

(5.22)

But note that this error depends on the data set utilized. To remove this dependence, we average over the training set to yield

(5.23)

Where ETS indicates the expected value (mean) over the training set. Now rewrite the expression inside the square brackets of Eq. 5.23 as

(5.24)

When we compute the expected value we obtain

(5.25)

The first term is the square of the bias of the model because it measures how much, on average, the output differs from the desired response. The second term is the variance because it measures how much each particular output y differs from its mean across the training sets.

Now let us assume that we add noise to the desired response, that is,

(5.26)

where f is the true input-output map and [image] is the noise. One extreme is the case where the model is so small that the output is not dependent at all on the variability of the data (no free parameters, just an a priori chosen function g). The model bias may therefore be large (if the function g we picked is not the true function f), but the model variance is zero, since y is the same across all training sets.

The other extreme is a model with so many parameters that it passes by every training data point exactly. In this case, the first term of Eq. 5.25, which measures the model bias, is zero, but the second term, which measures the model variance, is the power of the noise [image]. A good-sized model is one where both the model bias and variance are small. This view, however, does not tell us how to select the size of the model, but illustrates the issues.

5.6.3 Penalizing the Training Error

The problem is to find a general criterion to determine the model order for the problem under consideration. Generalization can also be formulated in this way. Many theories have addressed this issue. One that we would like to mention is Rissanen's minimum description length (MDL) criterion, because it is central to extracting models from data (or composing complex functions from simpler ones).

The previous explanation shows that we cannot look only at the fitting error as the criterion of optimality. We have to counterbalance it with the number of degrees of freedom of the model. Rissanen presented this idea very intuitively in terms of code lengths.

Our data can be thought as having an intrinsic code length in the following way: We may try to describe the data using a code we define. The data set thus requires a certain number of bits to be described (Figure 5-10). If the data is random noise, every sample needs to be used to describe the data, and the code length is the same as the data length. But the data may have been created by a linear system, for which two numbers (slope and bias) are sufficient to describe all the samples.

Figure 5-10 Code lengths of data and several models

When we model the data, we are effectively describing it in a different way: by the topology and parameters of our model and also by the fitting error. If we add the error length to the model code length, we again describe the original data exactly. Consider that we have to assign bits to codify the error, C(Ei), and also to represent the parameters of our model, C(Mi). Thus the description of the data using a particular model i is

The most efficient description of the data set is the one that minimizes the overall code length, that is, the total number of bits to represent both the error and the model parameters (Figure 5-10),

Notice that this is a very interesting idea because it couples the complexity (size) of the machine with the size of the fitting error. If we use a small number of parameters, the error will be larger, and we use many bits to represent the error. On the other hand, if we use too large a machine, we use too many bits to describe the machine, although only a few are needed to represent the error. The best compromise in terms of code length lies in the middle of smaller machines and manageable errors. MDL and Bayesian theory

Possibly the simplest implementation of this idea is to penalize the mean square error obtained in the training by including a term that increases with the size of the model, as was first proposed by Akaike. Akaike's information criterion (AIC) reads

(5.27)

where J(k) is the MSE in the training set, k is the number of free parameters of the model, and N is the number of data samples. AIC has been used extensively in model-based spectral analysis ( Kay). This expression shows that even if the error decreases with the size of the model k, there is a linear penalty with k, so the minimum value is obtained at some intermediate value of model order k (Figure 5-11).

Figure 5-11 Best model order according to Akaike's criterion

Notice that in Akaike's criterion the penalty is independent of the number of samples of the training data. According to Rissanen MDL criterion, a larger penalty for larger data sets can be obtained by substituting 2k with k/2ln(N), yielding

(5.28)

The appeal of these equations is that they allow us to use all the available data for training (unlike cross-validation) and can be easily computed for practical applications since they require only the mean square error in the training set for a collection of models of different sizes k. Akaike's method works well for one-layer systems (particularly linear ones). However, it becomes less acccurate for multilayer networks, since the size of the model is not only related to the number of weights.

It is also important to relate this method to the early stopping criterion that we established in Chapter 4 using cross-validation. Remember that we stopped training based on the performance in the validation set. The early stopping criterion directly measures some type of distance between the model and the data. We can choose the best model by utilizing different model sizes k and picking the one that provides the minimum error in the cross-validation set, that is,

(5.29)

It has been shown that this use of cross-validation is asymptotically equivalent to Akaike's criterion. In neural networks these equations have to be interpreted in an approximate sense, particularly for multilayer architectures. In fact, the role of the PEs and their weights is very different, so it is not enough to naively count the number of free parameters. The principle of structural risk minimization and the VC dimension is a more rigorous way to select the best-sized model. We will address it shortly.

NEUROSOLUTIONS EXAMPLE 5.11

Akaike's Criterion for RBFs

This example demonstrates Akaike's criterion for selecting the number of PEs in the RBF network. We have added a few DLLs to the breadboard, one of which computes Akaike's criterion. The others change the widths and centers of the RBFs to automatically span the input range [0, 1] based on the number of hidden PEs. Thus you can change the number of RBFs and run the network to see what the final Akaike's criterion value will be. Akaike's criterion normally provides a good estimate of the "best" number of centers in RBF networks.

NeuroSolutions Example

5.6.4 Regularization

Regularization theory was proposed by Tikhonov to deal with ill-posed problems. As an example, the equation xA = y is said to be ill-conditioned when a slight modification [image] y due to noise in the dependent variable y produces an enormous change in the solution for x. One way to solve this type of problem is to minimize the residue:

(5.30)

Tikhonov proposed to stabilize the solutions to such problems by adding a regularizing function [image] (x) to the solution

(5.31)

and was able to show that when [image] y approaches zero, the solution approaches the true value A [image] y. is a small constant called the regularization constant, and the regularizing function is a nonnegative function that includes a priori information to help the solution. Normally these regularizers impose smooth constraints, that is, they impose limits on the variability of the solution.

When we deal with the determination of the complexity of a learning machine with information restricted to the training set, the problem is ill-posed because we do not have access to the performance in the test set. The basic idea of regularization theory is to add an extra term to the cost function so that the optimization problem becomes more constrained, that is,

(5.32)

where Jc is the cost function, Jr is the regularizer, and [image] is a parameter that weights the influence of the regularizer versus the cost. Tikhonov regularizers penalize the curvature of the original solution, that is, they seek smoother solutions to the optimization problem. If we recall the training algorithms, we should choose regularizers for which derivatives with respect to the weights are efficiently computed. One such regularizer is

(5.33)

which penalizes large values of the second derivative of the input-output mapping. There is evidence that even a first-order penalty works in practice. The value of [image] must be experimentally selected.

Regularization is closely related to optimal brain damage (which uses the Hessian to compute saliencies) and to the weight-decay ideas to eliminate weights. In fact, weight decay (Eq. 4.19) is equivalent to a regularization term that is a function of the L2 norm of the weights (Gaussian prior), that is,

(5.34)

The square in Eq. 5.34 can be substituted by the absolute value to obtain an L1 norm of the weight yielding Eq. 4.20 (Laplacian prior).

It is interesting to compare Eq. 5.34 with Eq. 5.27. Both are effectively creating a new cost function that penalizes large models. However, the principles used to derive both expressions are very different. This analogy suggests that the determination of the regularization constant [image] is critical to find the best possible model order. Too large a value for [image] will choose networks that are smaller than the optimum, while too small a [image] will yield networks that are too large. Moreover, we can relate these choices to the bias and variance of the model. We can expect that a large [image] will produce smooth models (too large a bias), while a small a [image] will produce models with large variance. The best value of the regularization constant can be computed from statistical arguments Wahba. Let us experimentally verify these statements.

NEUROSOLUTIONS EXAMPLE 5.12

Weight-Decay to Prune RBFs

The weight-decay DLL that we introduced in Chapter 4 can be used to implement the regularization discussed above. We use the same RBF breadboard and set the number of hidden PEs to 20. Then, using weight decay on the output synapse, we can dynamically "turn off" unnecessary PEs by driving their output weights to zero. By adjusting the decay parameter of the weight-decay algorithm, we can produce smoother or more exact outputs from the network.

NeuroSolutions Example

Go to next section


 5.7 Applications of Radial Basis Functions

5.7.1 Radial Basis Functions for Classification

Let us interpret Eq.5.1 and Eq.5.2 for a classification problem. In classification f(x) becomes the indicator function {-1 (or 0),1}. What these equations say is that one can construct an arbitrary discriminant function in the input space by constructing linear discriminant functions in an auxiliary space (the space of the elementary functions) of large dimension that is nonlinearly related to the input space x (by [image] (x)). This is a counterintuitive result that was first proved by Cover (Cover Theorem ) and is associated with the fact that in sufficiently high-dimensional spaces data is always sparse. If the data clusters are always far apart, it is always possible to use hyperplanes to separate them. The problem is that we need to determine many parameters.

Radial basis function networks directly implement this idea by using Gaussian functions to project the input space to an intermediary space where the classification is done with a hyperplane implemented as the weighted sum of the Gaussian PE outputs. This result can be understood if we focus on the output of each Gaussian. No matter how intertwined the classes are, if the centers of the Gaussians and their radius (the variance) are properly controlled, each Gaussian can always be made to respond to a single class. The obvious limit is to assign a Gaussian to each sample, but generally this is not necessary.

We can assign a Gaussian to each subcluster of the classes. The classification is then made by linearly combining the responses of each of the Gaussians so that it is +1 for one class and -1 (or 0) for the other. From this discussion, we see that there are two fundamental steps in designing RBFs for classification: The first is the placement and the radius selection for each Gaussian, and the second is the weighting of the individual Gaussian responses to obtain the desired classification. It would also be convenient to have Gaussians with a different radius in each direction. This extends the contours of the Gaussians from circles to ellipses.

NEUROSOLUTIONS EXAMPLE 5.13

MLPs and RBFs for Classification

This example uses RBFs to do classification. We repeat the problem from Chapter 3 of classifying various people whether they are male or female based on the height and weight data. Remember that we cannot achieve perfect classification. For reference, we have included a link to the MLP example that solves this problem.

NeuroSolutions Example (MLP)

Run the RBF network and see that the classifier performs at the same level but the discriminant functions are different (here they are more curved). You should change the number of RBF PEs and see how the discriminant plot and confusion matrix are affected.

NeuroSolutions Example (RBF)

5.7.2 Radial Basis Functions as Regularizers

Radial basis functions can be derived mathematically from the theory of Tikhonov regularizers Poggio. Interestingly, when the solution of Eq. 5.32 is carried out using the calculus of variations, radial basis functions appear as the natural solution for regularization (for rotationally and translation-invariant kernels). This means that instead of using the data collected from the outside world directly, there are advantages in first fitting one radial basis function to each data point and working with their outputs. In a sense, the RBFs are interpolating the collected data, but the method is too cumbersome for large data sets (and the variances must be experimentally determined).

5.7.3 Radial Basis Functions as Regressors: The Probabilistic Neural Network

We can also utilize radial basis functions to estimate a regression function from noisy data following the ideas of kernel regression. In kernel regression we seek to estimate the probability density function p(x, d) of the input-desired pairs (xi, di) using the Parzen window method (which is a nonparametric method). It can be shown that the regression of the target data yields RBF as kernel regression

(5.35)

where [image] is the width of the Gaussian and has to be experimentally determined. Basically, the method places a Gaussian at each sample multiplied by the desired response di and normalized by the response in the input space. This network is called the probabilistic neural network and can be easily implemented using RBF networks.

NEUROSOLUTIONS EXAMPLE 5.14

Density Estimation with RBFs

In this example we train a normalized radial-basis-function network according to Eq. 5.35 to show how the network handles probability density function approximation. We have a few samples in the input space that belong to two classes, and we train a probabilistic neural network to solve the problem. We select the number of RBFs to equal the number of samples. During training, notice that the RBF centers converge to the input data, and that the output of the net provides the conditional average of the target data conditioned on each input. Change the variance of the RBFs to see how they affect the estimates for the targets as given by the output weights.

NeuroSolutions Example

Go to next section


 5.8 Support Vector Machines

Support vector machines (SVMs) are a radically different type of classifier that have attracted a great deal of attention lately due to the novelty of the concepts that they bring to pattern recognition, their strong mathematical foundation, and their excellent results in practical problems. In Chapter 2 and Chapter 3 we covered two of the motivating concepts behind SVMs, namely, the idea that transforming the data into a high- dimensional space makes linear discriminant functions practical and the idea of large margin classifiers to train the perceptron. Here we will couple these two concepts and create the support vector machine. We refer to Vapnik's books for a full treatment.

Let us go back to the concept of kernel machines. We saw in Chapter 2 that the advantage of a kernel machine is that its capacity (number of degrees of freedom) is decoupled from the size of the input space. By mapping the input to a sufficiently large feature space, patterns become linearly separable, so a simple perceptron in feature space can do the classification. In this chapter we have discussed the RBF network, which can be considered a kernel classifier. In fact, the RBF places Gaussian kernels over the data and linearly weights their outputs to create the system output. It conforms exactly with the notion of the kernel machine presented in Chapter 2, Figure 2-12. When used as an SVM, the RBF network places a Gaussian in each data sample such that the feature space becomes as large as the number of samples.

But an SVM is much more than an RBF. To train an RBF network as an SVM, we will use the idea of large margin classifiers discussed in Chapter 3. There we presented the Adatron algorithm, which works only with perceptrons. Training an RBF for large margins will decouple the capacity of the classifier from the input space and at the same time provides good generalization. We cannot get better than this on our road to powerful classifiers. We extend the Adatron algorithm here in two ways: We apply it to kernel-based classifiers such as RBFs, and we modify the training for nonlinearly separable patterns.

5.8.1 Extension of the Adatron to Kernel Machines

Recall that the Adatron algorithm was able to adapt the perceptron to maximize its margin. The idea was to work with data-dependent representations, which lead to a very simple on-line algorithm to adapt the multipliers.

We will write the discriminant function of the RBF in terms of the data-dependent representation:

(5.36)

where G(x, [image] ²) represents a Gaussian function, L is the number of PEs in the RBF, wk are the weights, N is the number of samples, [image] i are a set of multipliers (one for each sample), and we consider the input space augmented by one dimension with a constant value of 1 to provide the bias. Notice that for the special case of a Gaussian the innerproduct of Gaussians is still a Gaussian. The kernel function (the Gaussian) first projects the inputs (x, xi) onto a high-dimensional space and then computes an inner product there. The amazing thing is that the Gaussian kernel avoids the explicit computation of the pattern projections into the high-dimensional space, as shown in Eq. 5.36 (the inner product of Gaussians is still a Gaussian). Any other symmetric function that obeys the Mercer condition has the same properties. This topology is depicted in Figure 5-12, where we can easily see that it is an RBF, but where each Gaussian is centered at each sample and the weights are the multipliers [image] i.

Figure 5-12 Topology of the SVM machine with RBF kernels

The Adatron algorithm can be easily extended to the RBF network by substituting the inner product of patterns in the input space by the kernel function, leading to the following quadratic optimization problem:

(5.37)

Following the same procedure as in Chapter 3 we define

and choose a common starting multiplier (e.g. i=0.1), learning rate [image], and a small threshold (e.g., t = 0.01).

Then, while M>t, we choose a pattern xi and calculate an update [image] [image] i = [image] (1 [image] g(xi)) and perform the update

After adaptation only some of the [image] i are different from zero (called the support vectors). They correspond to the samples that are closest to the boundary between classes. This algorithm is called the kernel Adatron and can adapt an RBF to have an optimal margin. This algorithm can be considered the "on-line" version of the quadratic optimization approach utilized for SVMs, and it can find the same solutions as Vapnik's original algorithm for SVMs, Freiss. Notice that it is easy to implement the kernel Adatron algorithm since g(xi) can be computed locally to each multiplier, provided that the desired response is available in the input file. In fact, the expression for g(xi) resembles the multiplication of an error with an activation, so it can be included in the framework of neural network learning. The Adatron algorithm essentially prunes the RBF network of Figure 5-12 so that its output for testing is given by

NEUROSOLUTIONS EXAMPLE 5.15

The Kernel Adatron and the Spiral Problem

In this example we compare the performance of the kernel adatron to the MLP on the spiral problem. The goal of the spiral problem is to separate the 2-dimensional input into two classes, when the inputs correspond to points in the space that spiral inward toward the center as shown below.

Since the two classes are intertwined, this is a very difficult problem. Imagine trying to correctly place the discriminant lines of an MLP to create the correct decision surface. The MLP does very poorly on this task, although there are special training rules in the literature that perform much better than gradient descent on this particular problem. The kernel adatron, however, can easily solve this problem. The only difference between the adatron from Chapter 3 and the kernel adatron is that the dot product is replaced with RBF PEs (Gaussians), thus providing a kernel projection instead of a linear projection.

NeuroSolutions Example

5.8.2 The Adatron with a Soft Margin

What happens if the patterns in feature space are not exactly linearly separable? The idea is to introduce a soft margin using a slack variable [image] i = 0 and a function [image], which will penalize the cost function. We still minimize the function F, but now subject to the constraints di(w.xi + b) = 1 [image] [image] i i = 1,…N and w.w =cn. The new cost function becomes

Normally, instead of computing the optimal C, we choose a value a priori. C can be regarded as a regularizer. This means that the matrix of kernel inner products is augmented in the diagonal by the factor 1/C, that is,

The only difference in the algorithm for this case is the calculation of g(xi) which becomes

As we can see, these calculations can be easily implemented as an iterative algorithm, but notice that large data sets produce very large RBF networks (one Gaussian per data sample). Since the input layer has no free parameters, the mapping can effectively be computed once and saved in memory.

SVMs have been applied to numerous problems with excellent results. Their performance is consistently on par with the best reported results, which have taken many years of fine-tuning. One of the weaknesses of the method is that it does not directly specify the number of support vectors to solve the problem. In principle, SVMs should be sensitive to outliers, even when using a soft margin.

5.8.3 Summary of the SVM Theory

We would like to present in a more rigorous manner the beautiful theory that gave rise to the SVMs and show its equivalence to the preceding algorithms. However, this theory is beyond the scope of an introductory textbook such as this one, so we will only highlight the most important concepts that gave rise to this design methodology. We will see how the adhoc observations made in Chapter 2 and 3 have been formulated mathematically by Vapnik.

Learning theory can be framed as a function approximation problem in spaces with a probability measure. The goal is to approximate a function d = f(x) where f(.) is a fixed but unknown conditional distribution function F(d|x). The approximant is a learning machine that implements a set of functions [image] where the parameters w are to be determined through learning. The inputs x are random vectors with a fixed but unknown probability distribution function F(x). The selection of the parameters w is done through a finite number M of input-output observations (xi,di), which are independent and identically distributed (i.i.d.).

You should be able to see how similar this is to the block diagram of Figure 5-1. We are saying that what links the desired response to the input is a conditional distribution function, which is unknown but fixed. The machine should discover this function by the repeated presentation of a finite set of exemplars that are assumed i.i.d. In learning theory the best parameters w* are chosen to minimize the risk functional

(5.38)

where L(d,f(x, w)), the loss function, measures the discrepancy between the desired response d and the learning machine output. However, we cannot compute this integral because we do not know the joint distribution F(x, d). Since we have the finite number of observation pairs (xi,di), so we will substitute Eq. 5.38 by

(5.39)

which is called the empirical risk, and we will minimize this quantity instead. This method of solving the risk problem is called the empirical risk minimization (ERM) principle. Notice that ERM is a principle based on induction. We may think that this substitution of the risk functional by the empiric risk would constrain the possible cases that could be solved. It turns out that Glivenko and Cantelli proved that the empiric distribution function (Eq. 5.39) converges to the actual distribution (Eq. 5.38). Kolmogorov even proved that the empirical distribution function has an asymptotic exponential rate of convergence. These are the basis for statistical inference.

The two problems treated so far in Chapter 1 and 3, respectively, the regression problem and the classification problem, are special cases of this formulation. In fact it is enough to define the loss function as

(5.40)

to obtain the formulation of the regression, provided that the output y is a real value and we assume that the class of functions [image] includes the regression function that we are seeking.

If the output y takes the integer values d = {0,1}, if the function [image] is the set of indicator functions (i.e., functions that take only two values zero and one), and if the loss function is defined as

(5.41)

then the risk functional computes the probability of an error in the classification.

We could even show that this same formalism can provide density estimation over the class of functions p(x,w) as a special case if

(5.42)

Learning theory provides the most general way to think about training adaptive systems. The theory mathematically addresses the problem of generalization that is vital to neurocomputing. Vapnik establishes four fundamental questions for learning machines:

  • What are the necessary and sufficient conditions for consistency of a learning process?

  • How fast is the rate of convergence to the solution?

  • How can we control the generalization ability of the learning machine?

  • How can we construct algorithms that implement these prerequisites?

We will restrict ourselves to the special case of pattern recognition (where the function is an indicator function). To study SVMs, we need to address the last two questions, but first we provide the definition of VC (Vapnik-Chervonenkis) dimension. One of the fundamental problems in pattern recognition has always been the estimation of the Bayes' error. There is no known procedure to directly minimize the Bayes' error, because it involves integration over the tails of the pdfs, which are unknown (and the multidimensional integral is not trivial either). Our procedure of designing classifiers by minimizing the training error (which in this theory corresponds to the empiric risk) is not appropriate, as we discussed in Chapter 4 and also in this Chapter. All the methods we discussed to control the generalization error are, in fact, indirect (and sometimes ad hoc), so researchers have tried to find methods that minimize an upper bound of the Bayes' error. It is in this framework that Vapnik's contributions should be placed. Vapnik argues that the necessary and sufficient conditions of consistency (generalization) of the ERM principle depend on the capacity of the set of functions implemented by the learning machine. He has shown that the VC dimension provides a way to estimate an upper bound for the Bayes' error.

The VC dimension h of a set of functions is defined as the maximum number of vectors that can be separated into two classes in all 2 [image] possible ways, using functions of the set. For the case of linear functions in n-dimensional space, the VC dimension is h = n + 1. So the VC dimension is a more rigorous way to measure the capacity of a learning machine, which we discussed in Chapter 2. For general topologies the VC dimension is not easy to determine, but the trend is that larger topologies correspond to larger VC dimension.

The VC dimension of a learning machine appears as a fundamental parameter to determine its generalization ability. In fact, Vapnik proved that the generalization ability (the risk R) of a learning machine Q(x, [image]) of size k parameterized by [image] is bounded by

(5.43)

where Remp( [image]) is the empirical risk (the error measured in the training set) and the second term is a confidence interval. The generalization ability depends on the training error, the number of observations and the VC dimension of the learning machine. There are basically two ways to handle the design.

The first is to design a learning machine with a given topology, which will have a given VC dimension (which needs to be estimated). This is the conventional neural network design. Once this is done, Eq.5.43 tells us everything. We train the ANN, and this gives us an estimate of the empirical risk, but also a confidence interval. Equation 5.43 describes the bias variance dilemma very precisely. To decrease the training-set error, we may have to go to large ANNs, which will provide a large confidence interval, that is, the test error may be much larger than the training error. We say that the machine memorized the training data, so the problem becomes one of trading off training-set error and small VC dimension, which is handled heuristically by the size of the learning machine. This compromise was thought intrinsic in inductive inference, encapsulated by the famous Occam's razor principle (the simplest explanation is the best).

The second approach is called the structural risk minimization (SRM) principle and gives rise to the SVMs. The principle specifies keeping the empirical risk fixed (at zero, if possible) and minimizing the confidence interval. Since the confidence interval depends inversely on the VC dimension, this principle is equivalent to searching for the machine that has the smallest VC dimension. Notice that there is no compromise in the SRM principle. It states that the best strategy is to use a machine with the smallest VC dimension. Another point to make is that the VC dimension and number of free parameters appear as two distinct quantities, unlike the indications from Akaike and Rissanen work. We now know that we can apply very large machines to small data sets and still be able to generalize due to capacity control, so this SRM approach has profound implications in the design and use of classifiers. Let us now see how we can implement SRM in practice.

Here the concepts of hyperplanes and margin become critical. Although the VC dimension of the set of hyperplanes in n dimensions is n+1, it can be less for a subset. In fact, Vapnik proved that the optimal hyperplane (the smallest norm) provides the smallest confidence interval, so the problem in SRM is one of designing a large margin classifier. Let us briefly describe Vapnik's formulation here to allow us a comparison with our previous approaches.

Assume we have a set of data samples

What we want is to find the hyperplane y = w.x+b with the smallest norm of coefficients ?w?² (the largest margin). To find this hyperplane, we can solve the following quadratic programming problem: minimize the functional

under the constraint of inequality

where the operation is an inner product. The solution to this optimization is given by the saddle points of the Lagrangian:

(5.44)

By using the dual formulation, we can rewrite Eq. 5.44 as

under the constraint [image]. The solution is a set of [image] *. We can show that only some of the samples will correspond to Lagrangian multipliers different from zero and will be called the support vectors. They are the ones that control the positioning of the optimal hyperplane. The large margin classifier thus is specified by

(5.45)

One of the characteristics of the SVM is that the user has no control over the number of support vectors (i.e. the size of the final machine). During training, all the RBFs are used, but once the SVM is trained, the RBF should be trimmed, discarding the RBFs that are not support vectors. The number of support vectors depends on the data, which makes sense but is not always useful since we never know the size of the model. The expressions we arrived at are exactly the same as the one for the Adatron algorithm discussed in Chapter 5, except that Vapnik suggests a quadratic programming solution, while the Adatron is an "on-line" solution, easily implemented in neural network software. Like any on-line algorithm, the Adatron requires control of learning rate and suffers from the problem of misadjustment and stopping criterion. We can expect that training SVMs with large data sets demands a lot from computer resources (memory or computation).

Now we have a better understanding of why optimal margins are good for classification. SVMs can also be used for regression and density estimation.

Go to next section


 5.9 Project: Applications of Neural Networks as Function Approximators

In Chapter 4 we saw how neural networks can be applied to classification. Here we would like to show how the same topologies can be applied as function approximators (nonlinear regressors) in a wealth of practical applications. We selected one application in the financial industry and another in real estate. The goal is to discover nonlinear mappings between input variables and important outcomes. In the financial arena the outcome is to predict the value of the S&P 500 using several financial indicators, while in the real estate application the problem is to estimate the price of a house based on several indicators. We will see that neural networks provide a very powerful analysis tool.

5.9.1 Prediction of S&P 500

This example develops a very simple model for predicting the S&P 500 one week in advance. You can use this demostration as a starting point for developing your own more complex financial models. The inputs to the model consist of the one year treasury bill yield, the earnings per share and dividend per share for the S&P 500, and the current week's S&P 500. The desired output is the next week's S&P 500. There are 507 weeks worth of data, which is approximately 10 years.

The data has been presampled so that the first 254 exemplars contain the data for weeks 1, 3, 5,…, 505, 507 and the last 253 exemplars contain the data for weeks 2, 4, 6,..., 504, 506. The first 254 exemplars are used for training, and the last 253 exemplars are used for evaluating the trained networks performance.

NEUROSOLUTIONS EXAMPLE 5.16

Prediction of S&P 500

We use a simple one-hidden-layer MLP to model this data. The network has 4 inputs, and the desired response is the next week value of the S&P 500. The topology has to be carefully developed, as we exemplified in Chapter 4. We recommend that weight decay be used to avoid overfitting. Alternatively, Akaike's criterion could be used to find the best model order, as we did in Example 5.11. Let us train the network until the error stabilizes.

The next step is to verify the performance of the network on the unseen data. In the figure we show the performance of the network that we trained. The solid line is the actual value, while the network output is dashed. As we can see, the network fits the actual value of the S&P 500 rather well with the 4 inputs selected. We can compute the correlation coefficient between the actual and predicted curves to have a normalized (but linear) measure of performance.

We suggest that you try RBFs for the same problem and compare performances. This can be the embryo of a financial model, but remember that predicting the value of the stock market is just one of many factors needed to come up with an investment strategy.

NeuroSolutions Example

5.9.2 Estimating the Price of a House

The final example that we address in this chapter is how to decide which inputs are more significant in our application. This is an important issue because in many practical problems we have many different indicators or sensors, which may require very large input layers (hence many network weights), and very few exemplars to train the networks. One possibility is to prune the number of inputs without affecting performance.

We have to understand that this is a compromise. The more variables we have for a problem, the better is the theoretical performance, assuming that we have infinite, noise-free data. You might think that each variable is a dimension to represent our problem, so the higher the number of dimensions, the better the representation. But notice that for each extra input the representation problem is posed in a larger-dimensionality space, so training the regressor (or the classifier) appropriately requires many more data samples. This is where the compromise comes in. Since we always have finite, noisy data, the fundamental issue is to find the best "projection" to represent our data well.

One approach is to use all the available data to train a neural network and then ask which are the most important inputs for our model. It is obvious that this requires the calculation of the relative importance of each input for the overall result, that is, the sensitivity of the outcome with respect to each input.

In this example we develop a model for real estate appraisal in the Boston area. We use 13 indicators as inputs to this model. These indicators are per capita crime rate by town (CRIM), proportion of residential land zoned for lots over 25,000 square feet (ZN), proportion of nonretail business acres per town (INDUS), bounds Charles River (CHAS), nitric oxides concentration (NOX), average number of rooms per dwelling (RM), proportion of owner-occupied units built prior to 1940 (AGE), weighted distances to five Boston employment centers (DIS), index of accessibility to radial highways (RAD), full-value property tax rate per $10,000 (TAX), pupil-teacher ratio by town (PTRATIO), 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town (B), percent of the population lowerstatus (LSTAT).

The desired output for this model is the median value of owner-occupied homes (in $1000s). Hence this is a mapping problem, which we solve with an MLP (nonlinear regression). There are 400 total samples. Three hundred of them are used as training data and the other 100 as testing data.

The way we can perform input-sensitivity analysis is to train the network as we normally do and then fix the weights. The next step is to randomly perturb, one at a time, each channel of the input vector around its mean value, while keeping the other inputs at their mean values, and then measure the change in the output. The change in the input is normally done by adding a random value of a known variance to each sample and computing the output. The sensitivity for input k is expressed as

where [image] is the ith output obtained with the fixed weights for the pth pattern, o is the number of network outputs, P is the number of patterns, and [image] is the variance of the input perturbation. This is very easy to compute in the trained network and effectively measures how much a change in a given input affects the output across the training data set. Inputs that have large sensitivities have more importance in the mapping and therefore are the ones we should keep. The inputs with small sensitivities can be discarded. This helps the training (because it decreases the size of the network), decreases the cost of data collection, and when done right has negligible impact on performance.

NEUROSOLUTIONS EXAMPLE 5.17

Estimating Prices in the Boston Housing Data

Let us initially train a one-hidden-layer MLP with 14 inputs and one output. The choice of the number of hidden PEs should be done as before, that is, by starting small and plotting the output MSE for several different runs as a function of the number of PEs. Train the network and run it on the test set to observe the performance in the test set. The network produces a very good fit in most cases, indicating a successful model.

The next step is to run the sensitivity analysis, which estimates the importance of each input to the overall performance. NeuroSolutions has a built in feature to do this. Let us work with the trained network and turn learning off (fix the weights). We have to specify the amount of random noise that we want to add to each individual input (which is done automatically, one input at a time), while keeping the other inputs at their mean values. NeuroSolutions computes the sensitivity at the output. Let us place a Matrix Viewer at the L2 Criterion in the sensitivity access point and write down the values. We should use different values for dithering to obtain a reasonable linear approximation to the operating point of the regressor. We can then plot the different values of the sensitivity for each input variable, as shown in the following Figure.

From the figure we see that there are 5 inputs that display a very low sensitivity, so they can be omitted without appreciably affecting the quality of the mapping. Hence a reduced network with the inputs INDUS, CHAS, NOX, RM, DIS, RAD, PTRATIO, LSTAT will be trained again. As you can see in the following figure, the performance is basically the same, but now we have a smaller network that will generalize better, and we can reduce the cost of collecting data for this problem.

NeuroSolutions Example

Go to next section


 6.1 Introduction

In the 1940s the neurophysiologist Donald Hebb enunciated a principle that became very influential in neurocomputing. By studying the communication between neurons, Hebb verified that once a neuron repeatedly excited another neuron, the threshold of excitation of the latter decreased, that is, the communication between them was facilitated by repeated excitation. This means that repeated excitation lowered the threshold or, equivalently, that the excitation effect of the first neuron was amplified (Figure 6-1).

Figure 6-1 Biological and modeled artificial system

We can extend this idea to artificial systems very easily. In artificial neural systems, neurons are equivalent to PEs, and PEs are connected through weights. Hence Hebb's principle will increase the common weight wij when there is activity flowing from the jth PE to the ith PE. If we denote the output of the ith PE by yi and the activation of the jth PE by xj, then

(6.1)

where [image] is our already known step size which controls the percentage of the product that is used to change the weight. There are many more ways to translate Hebb's principle in equations, but Eq. 6.1 is the most commonly used and is called Hebb's rule.

Unlike all the learning rules studied so far (LMS and backpropagation), there is no desired signal required in Hebbian learning. To apply Hebb's rule, only the input signal needs to flow through the neural network. Learning rules that use only information from the input to update the weights are called unsupervised. Note that in unsupervised learning the learning machine changes the weights according to some internal rule specified a priori (here the Hebb rule). Note also that the Hebb rule is local to the weight.

Go to next section


 6.2 Effect of the Hebbian Update

Let us see the net effect of updating a single weight w in a linear PE with the Hebbian rule. Hebbian learning updates the weights according to

(6.2)

where n is the iteration number and [image] a step size. For a linear PE, y = wx, so

(6.3)

If the initial value of the weight is a small positive constant (w(0) ~ 0), the update will always be positive. Hence the weight value will increase with the number of iterations without bounds, irrespective of the value of [image]. This is unlike the behavior we observed for the LMS or backpropragation, where the weights would stabilize for a range of step sizes. Hence Hebbian learning is intrinsically unstable, producing very large positive or negative weights. In biology this is not a problem because there are natural nonlinearities that limit the to synaptic efficacy (chemical depletion, dynamic range, etc).

NEUROSOLUTIONS EXAMPLE 6.1

Training with the Hebbian Rule

In this example, we introduce the Hebbian Synapse. The Hebbian Synapse implements the weight update of Eq.6.2. The Hebbian network is built from an input Axon, the Hebbian Synapse and an Axon, so it is a linear network. Since the Hebbian Synapse, and all the other Unsupervised Synapses (which we will introduce soon), use an unsupervised weight update (no desired signal), they do not require a backpropagation layer. The weights are updated on a sample-by-sample basis.

This example shows the behavior of the Hebbian weight update. The weights with the Hebbian update will always increase, no matter how small the step size is. We have placed a scope at the output of the net and also opened a MatrixViewer to observe the weights during learning. The only thing that the step size does is to control the rate of increase of the weights.

Notice also that if the initial weight is positive, the weights become increasingly more positive, while if the initial weight is negative, the weights become increasingly more negative.

NeuroSolutions Example

6.2.1 The Multiple Input PE

Hebbian learning is normally applied to single-layer linear networks. Figure 6-2 shows a single linear PE with D inputs, which is called the Hebbian PE. The output is

Figure 6-2 A D-input linear PE

(6.4)

According to Hebb's rule, the weight vector is adapted as

(6.5)

It is important to get a solid understanding of the role of Hebbian learning, and we will start with a geometric interpretation. Eq. 6.4 in vector notation is simply

(6.6)

that is, the transpose of the weight vector is multiplied with the input (which is called the inner product) to produce the scalar output y. We know that the inner product is computed as the product of the length of the vectors and the cosine of their angle [image],

(6.7)

Thus, assuming normalized inputs and weights, a large y means that the input x is "close" to the direction of the weight vector (Figure 6-3), that is, that x is in the neighborhood of w.

Figure 6-3 The output of the linear PE in vector space

A small y means that the input is almost perpendicular to w (the cosine of 90 degrees is 0), that is, x and w are far apart. The magnitude of y therefore measures similarity between the input x and the weight w using the inner product as the similarity measure.

This is a very powerful interpretation. During learning the weights are exposed to the data and condense all this information in their value. This is the reason that the weights should be considered as the long-term memory of the network. long and short term memory

The Hebbian PE is a very simple system that creates a similarity measure (the inner product, Eq. 6.7) in its input space according to the information contained in the weights. During operation, once the weights are fixed, a large output y signifies that the present input is "similar" to the inputs x that created the weights during training. We can say that the output of the PE responds high or low according to the similarity of the present input with what the PE "remembers" from training. The Hebbian PE thus implements a type of memory that is called an associative memory.

NEUROSOLUTIONS EXAMPLE 6.2

Directions of the Hebbian Update

This example shows how the Hebbian network projects the input onto the vector defined by its weights. We use an input composed of samples that fall on an ellipse in 2 dimensions and allow you to select the weights. When you run the network, a custom DLL displays both the input (blue) and the projection of the input onto the weight vector (black) The default is to set the weights to [1, 0], which defines a vector along the x axis. Thus you would be projecting the two-dimensional input onto the x axis. Changing the value of the weights rotates the vector. Notice that in any direction the output tracks the input along that direction, that is, the output is the projection of the input along that specified direction.

Notice also the Megascope display. When the input data circles the origin, the output produces a sinusoidal component in time, since the projection increases and decreases periodically with the rotation. The amplitude of the sinusoid is maximal when the weight vector is [1, 0], since this is the direction that produces a larger projection for this data set.

If we release the weights, that is, if they are trained with Hebbian learning, the weights seek the direction [1, 0]. It is very interesting to note the path of the evolution of the weights (it oscillates around this direction). Note also that the weights are becoming progressively larger.

NeuroSolutions Example

6.2.2 The Hamming Network as a Primitive Associative Memory

The idea that a simple linear network embeds a similarity metric can be explored in many practical applications. Here we exemplify its use in signal detection, where noise normally corrupts messages. We assume that the messages are strings of bipolar binary values (-1, 1), and that we know the strings of the alphabet (for instance the ASCII code of the letters). A practical problem is to find the string sent from a given string of five bits received. We can think of an n-bit string as a vector in n-dimensional space. The ASCII code for each letter can also be thought of as a vector. The question of finding the value of the received string is the same as asking which ASCII vector is closest to the received string (Figure 4.4). Using the preceding argument, we should find the ASCII vector on which the bit string produces the largest projection.

Figure 6-4 The problem of finding the best match to the received character in vector spaces

A linear network can be constructed with as many inputs as bits in an ASCII code (here we use only 5 bits, although the ASCII code is 8 bits long) and a number of outputs equal to the size of the alphabet (here 26 letters). The weights of the network are hard coded as the bit patterns of all ASCII letters. More formally, the inputs are vectors [image], the output is a scalar, and the weight matrix S is built from rows that are our ASCII codes, represented by [image], with i=1,..,26. The output of the network is y = Sx.

The remaining question is how to measure the distance between the received vector and each of the ASCII characters. Since the patterns are binary, one possibility is to ask how many bit flips are present between the received string and all the ASCII characters. We should assign the received string to the ASCII character that has the fewest bit flips. This distance is called the Hamming distance, HD, also known as the Manhattan norm or L1 norm.

When a character is received, each output i of the network is the dot product of the input with the corresponding row vector si. Since the inputs and weights are bipolar binary values, this dot product can be expressed as the total number of positions in which the vectors agree minus the number of positions in which they differ, which is quantified as their HD. Since the number of positions they agree is 5-HD, we have

This equation states that if we add a bias equal to 5 to each of the outputs of our net, we can directly interpret the network output as a Hamming distance (to be exact, the weights should be multiplied by 0.5, and the bias should be 0.5 to obtain the HD). A perfect match will provide an output of 5. We therefore just need to look for the highest output to know what was the character that was sent.

NEUROSOLUTIONS EXAMPLE 6.3

Signal Detection with Hamming Networks

In this example we create the equivalent of a Hamming net which will recognize the binary ASCII of 5 letters (A, B, L, P, Z). The input to the network is the last 5 bits of each letter. For instance, A is (-1,-1,-1,-1,1), B is (-1,-1,-1,1,-1), and so on.

Because we know ahead of time what the letters will be, we set the weights to the expected ASCII code of each letter. Here we are not going to use the Hamming distance but the dot-product distance of Hebbian learning. According to the associative memory concept, when the input and the weight vector are the same, the output of the net will be the largest possible. For instance, if (-1, -1, -1, -1, 1) is input to the network, the first PE will respond with the highest possible output. Single-step through the data to see that, in fact. the net gives the correct response. Notice also that the other outputs are not zero, since they depend on the Hamming distance between the input and each set of weights.

When noise corrupts the input, this network can be used to determine which letter the input was most likely to be. Noise will affect each component of the vector, but the net will assign the highest output to the weight vector that lies closest to the noisy input. When the noise is small, the Hamming network still provides a good assignment. Increase the noise power to see when the system breaks down. It is amazing that such a simple device still provides the correct output most of the time when the variance of the noise is as high as 2.

NeuroSolutions Example

Note that in the example we utilized the inner-product metric intrinsic to the Hebbian network instead of the Hamming distance, but the result is very similar. In this example the weight matrix was constructed by hand, due to the knowledge we have about the problem. In general the weight matrix has to be adapted, and this is where the Hebbian learning is important. Nevertheless, this example shows the power of association for signal detection.

6.2.3 Hebbian Rule as Correlation Learning

There is a strong reason to translate Hebb's principle as in Eq. 6.1. In fact, Eq. 6.1 prescribes a weight correction according to the product between the jth and the ith PE activations. Let us substitute Eq. 6.6 in Eq. 6.5 to obtain the vector equivalent

(6.8)

In on-line learning the weight vector is repeatedly changed according to this equation, using a different input sample for each n. However, in batch learning, after iterating over the input data of N patterns, the cumulative weight update is the sum of the products of the input with its transpose multiplied by the original weight vector w(0):

(6.9)

The factor that multiplies the initial weight in Equation 6.9 can be thought of as a sample approximation to the autocorrelation of the input data (apart from the normalizing factor 1/N), which is defined as Rx = E{xx [image]] where E[.] is the expectation operator (see the Appendix). The Hebbian algorithm is effectively updating the weights with a sample estimate of the autocorrelation function

(6.10)

Correlation is a well-known operation in signal processing and in statistics, and it measures the second order statistics of the random variable under study. First-and second-order statistics are sufficient to describe signals modeled as Gaussian distributions, as we saw in Chapter 2 (that is, first and second statistics are what is needed to completely describe the Gaussian data cluster). Second order moments also describe many properties of linear systems, such as the adaptation of the linear regressor studied in Chapter 1.

6.2.4 Power, Quadratic Forms, and Hebbian Learning

As we saw, the output of a linear network is given by Eq. 6.6. We define the power (energy, power and variance) at the output given the data set {x(1),x(2),….,x(N)} as

6.11

V in Eq. 6.11 is a quadratic form, and it can be interpreted as a field in the space of the weights. Since R is positive definite, we can further say that this field is a paraboloid facing upward passing through the origin of the weight space (Figure 6-5).

Figure 6-5 The power field as a performance surface

Let us take the gradient of V with respect to the weights:

We can immediately recognize that this equation provides the basic form for the Hebbian update of Eq. 6.10. If we recall the performance surface concept of Chapter 1, we conclude that the power field is the performance surface for Hebbian learning. Therefore when we train a network with the Hebbian rule, the algorithm is doing gradient ascent (seeking the maximum) in the power field of the input data. The sample by sample adaptation rule of Eq. 6.8 is merely a stochastic version and follows the same behavior. Since the power field is unbounded upward, we can also expect that Hebbian learning will diverge, unless some type of normalization is applied to the update rule. (Hebbian as Gradient Search.). This is a shortcoming for our computer implementations, because the limited dynamic range causes overflow errors. But there are many ways to normalize the Hebbian update.

NEUROSOLUTIONS EXAMPLE 6.4

Instability of the Hebbian Rule

This example shows that the Hebbian update rule is unstable since the weights grow without bound. We use a simple 2-D input example to show that the weight vector grows. We have opened a Matrix Viewer to see the weights, and we also plot the tip of the weight vector in the Scatter Plot as a blue dot (think of the weight vector as going from the origin to the blue dot). Notice, however, that the weight vector always diverges along the same direction. This is not by chance. Although unstable, the Hebbian network is finding the direction where the output is the largest. The more you train the network, the larger the weights get. Rerun the simulation several times to observe the behavior we describe.

NeuroSolutions Example

6.2.5 Data Representations in Multidimensional Spaces

What does the direction of the gradient ascent represent? To understand the answer to this question, we have to talk about data representations in multidimensional spaces. We normally collect information about real-world events with sensors. Most of the time the data to model a real-world phenomenon is multidimensional: that is, we need several sensors (such as temperature, pressure, flow, etc.). This immediately says that the state of the real-world system is multidimensional. Each set of inputs is a point in a space where the axes are exactly our measurement variables. In Figure 6-6 we show a two-dimensional example. The system operation create a cloud of points somewhere in this measurement space.

An alternative to describe the cloud of points is to define a new set of axes that are "attached" to the cloud of data instead of the measurement variables. This new coordinate system is called a data-dependent coordinate system. From the Figure 6-6 we see that the data dependent-representation moves the origin to the center (the mean) of our cloud of samples. But we can do more. We can also try to align one of the axes with the direction where the data has the largest projection. This is called the principal coordinate system for the data. For simplicity we would also like the principal coordinate system to be orthogonal (more on this later). Notice that the original (measurement) coordinate system and the principal coordinate system are related by a translation and a rotation, which is called an affine transform in algebra. If we know the parameters of this transformation, we have captured a great deal of information about the structure of our cloud of data.

Figure 6-6 The principal coordinate system

The principal coordinate system gives us not only knowledge about the structure of the data, but also versatility. We may want to represent our data in a smaller dimensional space to simplify the analysis, or to be able to visualize the data. Often times we are interested in preserving the variance of the cloud of points, since variance is loosely associated with information (see information and variance). To make the point clear, let us try to find only a single direction (i.e. a one-dimensional space) to represent most of the variance on our data. What direction should we use? If you think a bit, the principal coordinate system is the one that makes more sense, because we aligned one of the axes with the direction where the data has the largest variance. In this coordinate system we should choose the axis where the data has the largest projected variance.

Now let us go back to the Hebbian network. The weights of the network that was trained with the Hebbian learning rule find the direction of the input power gradient. The output of the Hebbian network (the projection of the input onto the weight vector) will then be the largest variance projection. In other words, the Hebbian network finds the axis of the principal coordinate system where the projected output has the highest variance. What is amazing is that the simple Hebbian rule automatically finds this direction for us with a local learning rule!

Even though Hebbian learning was biologically motivated, it is a way of creating network weights that are tuned to the second-order statistics of the input data. Moreover, the network does this with a rule that is local to the weights. We can further say that Hebbian learning extracts the maximum amount of information about a Gaussian input, since from all possible linear projections it find the one that maximizes the variance at the output (which is a synonym for information for Gaussian-distributed variables).

Go to next section


 6.3 Oja's Rule

To make Hebbian learning useful, we must create a stable version by normalizing the weights. Perhaps the simplest normalization to Hebb's rule was proposed by Oja. Let us divide the new value of the weight in Eq. 6.2 by the norm of the new weight vector connected to the PE, i.e.

(6.12)

We see that this expression effectively normalizes the size of the weight vector to 1. If a given weight component increases, the others have to decrease to keep the weight vector at the same length, so weight normalization is in fact a constraint. Assuming a small step size, Oja approximated the update of Eq. 6.12 by

(6.13)

producing Oja's rule (Oja's rule and the Raleigh quotient). Note that this rule can be considered the Hebbian update with a normalized activity xi(n)=xi(n)-y(n)wi(n). The normalization is basically a "forgetting factor" proportional to the square of the output (see Eq. 6.13).

Eq. 6.13 addresses the fundamental problem of Hebbian learning. To avoid unlimited growth in the weights, we applied a forgetting term. This solves the problem of weight growth but creates another problem. If the pattern is not presented frequently, it will be forgotten since the network forgets old associations.

NEUROSOLUTIONS EXAMPLE 6.5

Oja's Rule

This example introduces the Oja's Synapse (look at the Synapse with the label Oja). The network is still a linear network, but the Oja's synapse implements Oja's weight update described in Eq. 6.13. The overall network function is similar to the Hebbian network, except that now the weights stabilize, producing a vector in the direction of maximum change. The input data is the same as in the previous case. Notice that now the weights of the single-output network produce a vector oriented along the largest axis of the cloud of input samples (45 degrees). This is the direction that produces the largest possible output. Randomize the weights several times during learning to see that the network quickly finds this direction. Depending on the sign of the initial weights, the final weights will be both positive or both negative, but the direction does not change.

The step size now controls the speed of convergence. If the step size is too large, the learning will become unstable, as in the gradient-descent learning case. Large step sizes also produce rattling of the final weights (note that the weights form a linear segment) which should be avoided. If the step size is too small, the process will converge slowly. The best procedure is to start with the adaptation large and anneal its value to a small constant to fine-tune the final position of the weights. This can be accomplished with the scheduler.

NeuroSolutions Example

6.3.1 Oja's Rule Implements the Maximum Eigenfilter

What is the meaning of the weight vector of a neural network trained with Oja's rule? To answer this question, let us study a single linear PE network with multiple inputs (Figure 6-2), using the ideas of vector spaces. The goal is to study the projection defined by the weights created with Oja's rule. We already saw that Hebbian learning finds the direction where the input data has the largest projection, but the weight vector grows without limit. With Oja's rule we found a way to normalize the weight vector to 1. We should expect that this normalization will not change the geometric picture we developed for the Hebbian network. In fact, it is possible to show that Oja's rule finds a weight vector w = e1 which satisfies the relation proof of eigenequation

(6.14)

where R is the autocorrelation function of the input data and [image] is a real-value scalar. This equation was already encountered in Chapter 1 and tells us that e1 is an eigenvector of the autocorrelation function, since rotating e1 by R (the left side) produces a vector colinear with itself. We can further show that in fact [image] 1 is the largest eigenvalue of R, so e1 is the eigenvector that corresponds to the largest eigenvalue. We conclude that training the linear PE with Oja's algorithm produces a weight vector that is the eigenvector of the input auto correlation matrix, and produces at its ouput the largest eigenvalue. This linear network is called the maximum eigenfilter.

Figure 6-7 shows a simple case in two-dimensions. It shows a data cluster (black dots) spread along the 45 degree line. The principal axis of the data is the direction in 2-D space where the data has its largest power (projection variance). Imagine a line passing through the center of the cluster and rotate it so that the data cluster produces the largest spread in the line. For this case the direction will be close to 45 degrees. The weight vector of the network of Figure 6-2 trained with Oja's rule coincides exactly with the principal axis, also called the principal component. The projection on the direction perpendicular to it (the minor axis) produces a much smaller spread. For zero-mean data, the direction of maximum spread coincides with the direction where most of the information about the data resides. The analysis is the same when the data exists in a larger-dimensionality space D, but we cannot visualize it as well.

Figure 6-7 Projection of a data cluster onto the principal components

If you relate this figure with the NeuroSolutions example, the Oja's weight vector found the direction where the data produced the largest projection. This is a very important property, because the simple one-PE network trained with Oja's rule extracts the most information that it can from the input. In engineering applications where the input data is normally corrupted by noise, this system will provide a solution that maximizes the ratio signal power (normally the sinusoidal component) to noise power (definition of eigenfilter).

Go to next section


 6.4 Principal Component Analysis

We saw that Oja's rule found a normalized weight vector that is colinear with the principal component of the input data. But how can we find other directions where the data cluster has still appreciable variance? We would like to create more axes of the principal coordinate system mentioned in Section 6.2.5. For simplicity we would like to create a data dependent orthogonal coordinate system (i.e., all the vectors are orthogonal to each other) with unit-length vectors (orthonormal coordinate system). How can we do this? Principal component analysis answers this question.

From linear algebra (Golub and Van Loan) we know that if a D-dimensional vector x has zero mean and covariance matrix C, then there exists an orthogonal transform y = Q [image] x such that the covariance matrix of y is [image] =E{yy [image]}and

,

where [image] 1 [image] [image] 2 [image] … D are the eigenvalues of C and the columns of Q are the corresponding eigenvectors. Q is called the eigenvector matrix. The ith component of y i [image] has the maximum variance among all normalized linear combinations uncorrelated with the other components. This vector y defines the vector of principal components of x, and the procedure is called principal component analysis (PCA). For zero mean variables, the covariance matrix C defaults to the autocorrelation matrix R that we have been using in previous sections.

PCA is a very well known statistical procedure that has important properties. Suppose that we have input data of very large dimensionality (D dimensions). We would like to project this data onto a smaller-dimensionality space M (M < D), a step that is commonly called feature extraction. Projection always distort our data somewhat (think of a 3-D object and its 2-D shadow). Obviously, we would like to do this projection to M-dimensional space while maximally preserving the dispersion (variance from a representation point of view) about the input data. The linear projection that accomplishes this goal is the PCA.

PCA is therefore the best linear feature extractor for signal reconstruction. The error e in the approximation when we utilize M features is given by

(6.15)

Eq. 6.15 tells that the error power is exactly equal to the sum of the eigenvalues that were discarded. For the case of Figure 6-7 the minimum error in representing the 2-D data set in a 1-D space is obtained when the principal direction is chosen as the projection axis. The error power is exactly given by the projection on the minor direction. If we decided to keep the projection in the minor direction, the error incurred would have been much higher. This method of representing signals in successively larger spaces whose axes are determined from the data is called subspace decomposition, and it is widely applied in signal processing and statistics to find the best subspace of a given dimension that maximally preserves data information. There are well-known algorithms that analytically compute PCA, but they have to solve matrix equations (singular value decomposition).

Can we build a neural network that implements PCA on-line with local learning rules? The answer is affirmative. We have to use a linear network with multiple outputs (equal to the dimension M of the projection space), as in Figure 6-8.

Figure 6-8 A PCA network to project the data from D to M dimensions

The idea is very simple. First, we compute the largest eigenvector, as done earlier with Oja's rule. Then we project the data onto a space perpendicular to the largest eigenvector, and we apply the algorithm again to find the second largest principal component, and so on, until order [image]. The projection onto the orthogonal space is easily accomplished by subtracting the output of all previous output components (after convergence) from the input. This method is called the deflation method and mimics the Gram-Schmidt orthogonalization procedure. Gram-Schmidt orthgonalization

By slightly modifying Oja's learning rule we can implement the deflation method, as was first done by Sanger. We are assuming that the network has M outputs, each given by

(6.16)

and D inputs ( [image]). To apply Sanger's rule the weights are updated according to

(6.17)

This rule resembles the Oja's update, but now the input to each PE is modified by subtracting the product of the outputs from the preceding PEs and the respective weights. This implements the deflation method, after the system converges. The weight update of Eq. 6.17 is not local, since we need all the previous network outputs to compute the weight update to weight wij. However, there are other rules that use local updates (such as the APEX algorithm Diamantaras).

As we can expect from Eq. 6.17 and the explanation, the adaptation is coupled; that is, only after convergence of the first PE weights will the second PE weights converge completely to the eigenvector that corresponds to the second largest eigenvalue. A two-output PCA network will have weight vectors that correspond to the principal and minor components of Figure 6-5. The power of the two outputs correspond to the largest and smallest eigenvalues, respectively. The interesting thing about subspace projections is that in many problems the data is already restricted to an (unknown) subspace, so PCA can effectively perform data compression while preserving the major features of the data.

NEUROSOLUTIONS EXAMPLE 6.6

Sanger's rule and PCA

This example introduces Sanger's rule (look at the synapse with the label "Sang" in the breadboard ). Sanger's rule implements PCA. The dimension M of the output determines the size of the output space, that is, the number of eigenvectors, and also the number of features used to represent the input data. PCA finds the M weight vectors that capture the most variance about the input data. For instance, a 3 output Sanger's network will find 3 orthogonal vectors; the principal axis, which captures more variance than any other vector in the input space, and the two vectors that capture the second and third most variance. In this example we take a high-dimensional input, 8 x 8 images of the 10 digits and project them onto their M principal components. M is a variable that you can control by setting the number of outputs of the Sanger's network. The outputs of the PCA network are the features obtained by the projection.

We then use a custom DLL to recreate the digits using only the M features. This DLL takes the output of the Sanger's network and multiplies it by the transpose of W, so it recreates a 64-output image. This image shows us how much of the original information in the input we have captured in the M-dimensional subspace. When the two images are identical, we have preserved in the features the information contained in the input data.

The display of the eigenvectors (the PCA weights) is not easy, since they are vectors in a 64-dimensional space. After convergence they are orthogonal. We can use the Hinton probe to visualize their values, but it is difficult to find patterns. (In fact, the signs should alternate more frequently toward the higher order meaning that finer details are being encoded.) Try different values for the subspace dimension M, and verify that PCA is very robust, that is, even with just a few dimensions the reconstructed digits can be recognized.

A word of caution is needed at this point. The PCA finds the subspace that best represents the ensemble of digits, so the best discrimination among the digits in the subspace is not guaranteed with PCA. If the goal is discrimination among the digits, a classifier should be designed for that purpose. PCA is a linear representation mechanism, and only guarantees that the features contain the most information for reconstruction.

NeuroSolutions Example

PCA decomposition is a very important operation in data processing because it provides knowledge about the hidden structure (latent variables) of the data. Thus, there are many other possible formulations for the problem. PCA, SVD, and KL transforms

6.4.1 PCA for Data Compression

PCA is the optimal linear feature extractor. This means that there is no other linear system that is able to provide better features for reconstruction. One of the obvious PCA applications is therefore data compression. In data compression the goal is to be able to transmit as few bits per second as possible while preserving as much of the source information as possible. This means that we must "squeeze" into each bit as much information as possible from the source. We can model data compression as a projection operation where the goal is to find a set of bases that produce a large concentration of signal power in only a few components.

In PCA compression the receiver must know the weight matrix containing the eigenvectors since the estimation of the input from the eigenvalues is done by

(6.18)

The weight matrix is obtained after training with exemplars from the data to be transmitted. It has been shown that for special applications this step can be completed efficiently and is done only once, so the receiver can be constructed beforehand. The reconstruction step requires M x D operations, where D is the input vector dimension and M is the size of the subspace (number of features).

6.4.2 PCA Features and Classification

We may think that a system able to optimally preserve signal energy in a subspace should also be the optimal projector for classification. Unfortunately, this is not the case. The reason can be seen in Figure 6-9, where we have represented two classes. When the PCA is computed, no distinction is made between the samples of each class, so the optimal 1-D projection for reconstruction (the principal direction) is along the x1 axis. However, it is easy to see that the best discrimination between these two clusters is along the x2 axis, which from the point of view of reconstruction is the minor direction.

PCA chooses the projections that best reconstruct the data in the chosen subspace. This may or may not coincide with the projection for best discrimination. A similar situation occurred when we addressed regression and classification (first example of Chapter 2). A linear regressor can be used as a classifier, but there is no guarantee that it produces the optimal classifier (which by definition minimizes the classification error).

Figure 6-9 The relationship between eigendirections and classification

However, PCA is appealing for classification, since it is a simple procedure and experience has shown that it normally provides good features for classification. But this depends on the problem and there is no guarantee that classifiers based on PCA features work well.

NEUROSOLUTIONS EXAMPLE 6.7

PCA for Preprocessing

In this example we use PCA to find the best possible linear projection for reconstruction, and then we use an MLP to classify the data into one of 10 classes (the digits). Notice that this problem was already solved in Chapter 3 with the perceptron, and we obtained perfect classification using the input data directly.

The only way we can do a fair comparison is to limit the number of weights in the two systems to the same value and compare performances.

NeuroSolutions Example

Go to next section


 6.5 Anti-Hebbian Learning

We have seen that Hebbian learning discovers the directions in space where the input data has the largest variance. Let us make a very simple modification to the algorithm and include a minus sign in the weight update rule of Eq. 6.1:

(6.19)

This rule is called the anti-Hebbian rule. Let us assume that we train the system of Figure 6-2 with this rule. What do you expect this rule will do?

The easiest reasoning is to recall that the Hebbian network maximizes the output variance by doing gradient ascent in the power field. Now, with the negative sign in the weight update equation, the adaptation will seek the minimum of the performance surface, that is, the output variance will be minimized. Hence the output of the linear network trained with anti-Hebbian learning will always produce zero output, because the weights will seek the directions in the input space where the data clusters have a point projection. This is called the null (or orthogonal) space of the data. The network finds this direction by doing gradient descent in the power field.

If the data fills the input space, the only way to minimize the network output is to drive the weights to zero. On the other hand, if the data exists in a subspace, the weights will find the directions where the data projects to a point. For Figure 6-7 anti-Hebbian learning will provide zero weights. However, if the data was one-dimensional, that is, along the 45-degree line, then the weights would be placed along the 135-degree line.

NEUROSOLUTIONS EXAMPLE 6.8

Anti-Hebbian Learning

In this example we use the Hebbian synapse with a negative step size to implement an anti-Hebbian network. The anti-Hebbian rule minimizes the output variance, thus it will try to find a weight vector that is orthogonal to the input (the null space of the input) such that the projection of the data onto the weight vector is always zero.

There are two cases of importance. Either the data lies in a subspace of the input space, in which case the zero output can be achieved by adapting the weight vector perpendicular to the subspace where the input lies, or the input samples cover the full input space, so the only way to get a zero output is to drive the weights to zero.

Notice how fast the anti-Hebbian network trains. If the data moves in the input space, notice that the weights are always finding the direction orthogonal to the data cluster.

NeuroSolutions Example

This behavior of anti-Hebbian learning can be considered as decorrelation, that is, a linear PE trained with anti-Hebbian learning decorrelates the output from its input. We must realize that Hebbian and anti-Hebbian learning have complementary roles in projecting the input data, which are very important for signal processing. For instance the new high-resolution spectral analysis techniques (such as MUSIC and ESPRIT Kay) are based on ways of finding the null space of the data, so they can be implemented on-line using anti-Hebbian learning. We provide an example in Chapter 9.

6.5.1 Convergence of Anti-Hebbian Rule

Another interesting thing is that the convergence of the anti-Hebbian rule can be controlled by the step size, as in LMS or backpropagation. This means that if the step size is too large, the weights will get progressively larger (diverge), but if the step size is below a given value the adaptation will converge. From the known paraboloidal shape of the power field in weight space, we know it has a single minimum. Hence the situation is like gradient descent which we studied in Chapter 1. What is the value under which the weights converge to finite values?

The anti-Hebbian update for one weight is

(6.20)

If we take expectations and use induction, as we did in Section 1.6.3 to compute the largest step size for the LMS, we can conclude that

(6.21)

which is stable if

(6.22)

where [image] is the eigenvalue of the autocorrelation function of the input. We can immediately see the similarity to the convergence of the LMS rule. For a system with multiple inputs the requirement for convergence has to be modified to

(6.23)

where [image] max is the largest eigenvalue of the input autocorrelation function as for the LMS case.

NEUROSOLUTIONS EXAMPLE 6.9

Stability of Anti-Hebbian Rule

This example shows that the anti-Hebbian rule is stable for the range of values given by Eq. 6.23 when random data is used. Change the step size to see what is the compromise between rattling and speed of convergence achieved with the anti-Hebbian rule. Since the weight update is sample by sample, when the data has deterministic structure divergence may occur at step sizes smaller than the ones predicted by Eq. 6.23. The same behavior was encountered in the LMS.

NeuroSolutions Example

Go to next section


 6.6 Estimating Cross-correlation with Hebbian Networks

Suppose that we have two data sets formed by N exemplars of D-dimensional data x1,....xD and d1,... dD, and the goal is to estimate the cross-correlation between them. Cross-correlation is a measure of similarity between the two sets of data which extends the idea of correlation coefficient (see Appendix A and Chapter 1)

In practice we are often faced with the question of how similar are the {x} and {d} data sets are. Cross-correlation answers this question. Let us assume that the data samples are ordered by their indices. The cross-correlation for index i, j is

(6.24)

where N is the number of patterns and D is the size of the input and desired response vectors. The fundamental operation of correlation is to cross multiply the data samples and add the contributions. We can define the average operator

(6.25)

The cross-correlation can then be defined as

(6.26)

where the vector

is built from the ith sample of all the patterns in the input set (likewise for d). The cross-correlation matrix Rxy is built from all possible shifts i, j,

(6.27)

The cross-correlation vector used in regression (Chapter 1) is just the first column of this matrix. Now let us relate this formalism to the calculations of a linear network trained with Hebbian learning. Assume that we have a linear network with D inputs x and D outputs y (Figure 6-10)

Figure 6-10 A multiple-input multiple-output Hebbian network

To compute the cross-correlation between x and the data set d, we substitute the network output y in the Hebbian rule by the data set d, i.e.

(6.28)

which implements what we call forced Hebbian learning. We can write the output yj as Eq. 6.4, but now with two indices i and j:

(6.29)

The weight wi,j when adapted with forced Hebbian learning takes the form

(6.30)

If wij(0)=0 after N iterations we get

(6.31)

So by comparing Eq. 6.24 with Eq. 6.31 we conclude that the weight wij trained with forced Hebbian learning is proportional after N iterations to the cross-correlation element rij. If [image] =1/N and the initial conditions are zero this is exactly rij. Notice also that the elements of the cross-correlation matrix are precisely the weights of the linear network (Eq. 6.26). For this reason the linear network trained with forced Hebbian learning is called a correlator or a hetero-associator. Hence forced Hebbian learning is an alternate, on-line way of computing the cross-correlation function between two data sets.

NEUROSOLUTIONS EXAMPLE 6.10

Forced Hebbian Computes Cross-Correlation

In this example we show how forced Hebbian learning computes the cross-correlation of the input and desired output. We have a 3-input network that we would like to train with a desired response of 2 outputs. We have created a data set with 4 patterns. The cross-correlation computed according to Eq. 6.24 is

r(0, 0) = 0.5 ; r(0, 1) = r(1, 0) = 0; r(1, 1) = 0.25; r(0, 2) = 0.5 ;r(1, 2) = 0.25

Let us use the Hebbian network and take a look at the final weights. Notice that we started with the weights at zero value and stopped the network after 10 iterations of each batch (4 patterns) with a step size of 0.025 (1/40).

NeuroSolutions Example

There are two important applications of this concept that we address in this chapter. One uses cross-correlation with anti-Hebbian learning to find what is different between two data sets and can be considered a novelty filter. The other, possibly even more important, is a memory device called an associative memory.

Go to next section


 6.7 Novelty Filters and Lateral Inhibition

Let us assume that we have two data sets x and d. Taking x as the input to our system, we want to create an output y as dissimilar as possible to the data set d (Figure 6-11). This function is very important in signal processing (decorrelation) and in information processing (uncorrelated features), and it seems to be at the core of biological information processing. We humans filter out with extreme ease what we know already from the sensor input (either visual or acoustic). This avoids information overload. It seems that what we do first is to equalize the incoming information with what is expected so that unexpected things stand out.

Figure 6-11 The function of a decorrelation PE

We can consider that the incoming data is represented by x and what the system already knows is represented by d, so novelty is the part of x that is not represented in d. From the point of view of vector operations, this is equivalent to finding a rotation of x such that y is orthogonal to d (i.e. in the null space of d). The system of Figure 6-11 with the learning rule of Eq. 6.19 where d substitutes for y (i.e. wij= - [image] xjdi) does exactly this job.

NEUROSOLUTIONS EXAMPLE 6.11

Novelty Filter with Anti-Hebbian Learning

In this example we show a novelty filter. We have created a three-dimensional input signal that represents the output of a system under normal operating conditions. This system could be a car (outputs = velocity, acceleration, and turning angle), a power plant, or any other system. We train the novelty filter on this data, and the anti-Hebbian learning will learn its null space – the vector where the input projection is always very close to zero. The weights are fixed at this point.

When the system changes slightly (abnormal system operation) and its output is fed to the trained novelty filter, the filter output is no longer close to zero because the new signal is no longer in the null space of the filter weights. This indicates that the system is no longer operating normally. We change the parameters of the system midway in the experiment. From the filter output you should be able to pinpoint where the change occurred. Notice that the network inputs (normal and abnormal operating conditions) look very similar throughout the segment, so it would be difficult to find the change in the system by hand.

NeuroSolutions Example

6.7.1 Lateral Inhibition

Another very useful strategy to decorrelate signals is to create lateral connections between PEs adapted with anti-Hebbian learning (see Foldiak). Let's analyze the topology depicted in Figure 6-12. In the Figure, c is the lateral inhibition connection from yi to yj. We use the + superscript to mean the preactivity input of the PEs. Note that

Figure 6-12 Lateral inhibition connections

The cross-correlation between yi and yj is

If the power of yi is greater than zero, there is always a value

which will decorrelate yi and yj, that is, will make R(yi,yj) = 0. Notice that this value is the negative of the cross-correlation between the ith and jth PE activations, so if we use the anti-Hebbian learning with a small step size, the outputs will be decorrelated. Notice that one of the characteristics of PCA is that the outputs are orthogonal, that is, the outputs are uncorrelated. Lateral inhibition basically achieves the same effect; however, the variance of the outputs is not constrained, nor are the weight vectors.

The interesting thing about lateral inhibition is that it can provide an alternative method to construct networks that find the principal component space with a local learning rule, or even provide whitening transforms (i.e., a transform that not only orthgonalize the input data but also normalize the eigenvalues).

6.7.2 APEX Model for PCA

Diamantaras has shown that the network of Figure 6-13 implements PCA when the weights are adapted according to

Note that the weights are adapted using Oja's rule, while a form of anti-Hebbian learning is used to adapt the lateral connections. Note that all the quantities are local to the weights, so the rule is actually local. The system learns the principal components in parallel, unlike Sanger's rule that is based on deflation.

Figure 6-13 The APEX topology for PCA

6.7.3 Whitening Transform

A whitening transform is a very important linear transformation in adaptive systems, because it transforms any data described by an autocorrelation R with an arbitrary eigenvalue spread into an orthonormal matrix, (i.e. a matrix with all the eigenvalues equal to a constant). For whitened data the LMS algorithm is as fast as Newton's method since the eigenvalue spread is 1. Whitening the input data drastically improves the speed of linear learning systems using first-order methods. We now present a topology and learning rule that produces a whitening transform (see Silva ).

The network that implements the whitening transform is like the one in Figure 6-13. The idea of the algorithm is very similar to the Gram-Schmidt procedure, but it adapts all the vectors at the same time, yielding a symmetric adaptation structure. The adaptation rule reads

Notice that this formula specifies a weight update that is not local to the weights. However, with lateral inhibition we can easily implement it in a single layer network (Figure 6-13). Notice that the sum over k can be implemented by bringing a lateral inhibition connection from the kth PE to the ith PE with a weight copied from the forward connections and connecting the jth input with the kth PE. Silva discusses another implementation and also proves the convergence of the algorithm. The interesting thing about this transformation is that it creates an orthonormal space at the output by equalizing the eigenvalues instead of by rotating the axis, as done in PCA (Figure 6-14). This is reportedly much faster than PCA for a variety of problems.

Figure 6-14 Whitening transform and the corresponding weight update

Go to next section


 6.8 Linear Associative Memories (LAMs)

Information processing requires memorization of information. In digital computers one memory location stores one bit of information, so the information is stored individually. An interesting task is to seek ways to store information in a more global way, that is, by having several PEs store many data patterns, and then to ask the question, What system is more efficient and robust to noise?

The linear associator, also called a linear associative memory (LAM), provides an alternative computer memory paradigm. The research has strong ties to psychology since today it is farily well accepted that the brain does not store each bit of information separately. Many neurons (cell assembly) store many patterns.

The system of Figure 6-10 trained with the forced Hebbian rule (Eq. 6.28) can be used as an associative memory, that is, a device that can be trained to associate an input x to a response d. Then in the absence of d, x can produce an output y that resembles d. The question is, How can information be stored globally and how can we retrieve it?

We use the matrix notation for convenience. Let the input set of N elements be denoted as a vector x. Likewise y is the corresponding N-component output vector. The output, being linear can be obtained as y = Wx, where W is the weight matrix. Forced Hebbian learning constructs each weight according to Eq. 6.28, which can be written in matrix notation as the outer product, i.e. W = dx [image]. When the input x is entered in the linear associator, the output created by the system is thus

(6.32)

which is proportional to the original output utilized in the training (remember that x [image] x is a constant equal to the length of the vector x).

The interesting question is, What happens when more than one input vector is stored in the memory? Can we still recover each one of the inputs, or is the output contaminated by the other inputs?

NEUROSOLUTIONS EXAMPLE 6.12

LAM Application

In this example we use a linear associative memory (LAM) to associate area codes (3 digits) with prices for long-distance phone calls (2 digits). When we input an area code, we would like the network to output the correct price for the corresponding long distance phone call rate. During training we use heteroassociation to train the LAM. We have encoded the area codes and rates as binary digits (12 and 8 bits, respectively). Hence this LAM will have 12 inputs and 8 outputs.

We have created input files that contain a set of 3 binary-encoded area codes and prices. We have also added a custom DLL that allows us to display a sequence of binary digits as the equivalent number. Once the network is trained, we can present the area code, and the system will produce the corresponding long-distance call rate at the output.

It is interesting to ask where the information is stored. The answer is in the weights throughout the network. This is rather different from the storage we use in digital computers, where the memory is addressed. If we lose the address, the item stored can never be recovered. Here we recall the output by providing the input (i.e., the content of the memory, so these memories are called content addressable memories (CAMs)).

CAMs are very robust. Use a Matrix Editor and zero one (or several) weights, and observe that the output barely changes. (Notice that the numbers displayed are subject to an encoding, so they change only when there are drastic modifications in one of the bits.) If one bit was lost in the address or content of a computer memory, the original content would be impossible to retrieve (unless coding - which is redundancy- was used).

Another interesting thing is that these memories cover the input space with a similarity measure (the inner-product metric, as we have seen previously). For numeric imputs this is not often important, since numeric information is normally precise. But for names, words, concepts, and so on, similarity is very useful. (Is his name Gary, Cary, Gerry, or Larry?). To see this property of LAMs let us change just one of the input digits and verify that the output is basically unchanged. These are nice properties of LAMs that make them very good models for human memory in cognitive science.

NeuroSolutions Example

6.8.1 Crosstalk in LAMs

Let us assume that we have K input-output vector pairs xk [image] dk. The associative memory is trained by repeated presentation of each input, so using the principle of superposition, the final weight matrix is the sum of the individual weight matrices:

(6.33)

where each [image]. When an input vector xl is presented to the network its output is

(6.34)

The associative memory output is built up from two terms. The first, which is the true output for the input xl is added with a term that is called the crosstalk because it measures how much the other outputs interfere with the true one. But if the crosstalk term is small, Eq. 6.34 tells us that in fact the associative memory is able to retrieve the pattern that corresponds to x during training (the association).

The crosstalk is a function of how similar the input xl is to the other inputs xk. This can be better understood in a geometric setting. Assume that the input patterns are vectors in a D-dimensional vector space. The output of the linear associator, being a product of a matrix by a vector (Eq. 6.34 ), rotates the input xl to obtain y. The goal is to obtain a rotation that produced the expected association to dl. What Eq. 6.34 is saying is that the actual output y is constructed by two terms. The first is the desired output d scaled by the length of xl, and the second is a sum of contributions that depend on the inner product of all the other input patterns xk with xl. Figure 6-15 shows the construction for two vectors only.

Figure 6-15 Output with crosstalk

If the inputs are all orthogonal, the inner product of xl and xk is zero (zero crosstalk), and the linear associator produces perfect recall. However, if the input patterns are not orthogonal, each one of the yk in the sum is multiplied by the projection of the input vector k on l, which can add up to a large number, rotating and changing the length of the true output d. If the crosstalk term is comparable to the first term, the linear associator produces an output that has nothing to do with the expected response d.

This analysis brings us immediately to the concept of storage capacity, which is defined as the maximum number of patterns that can be stored and recalled without degradation. Associative memories, unlike computer memories, have finite storage capacity. We know that in a space of dimension D, there are only D possible orthogonal directions, so perfect recall is limited to a number of patterns equal to the size of the input space (length of the input vector). In practical conditions the inputs may not be orthogonal to each other, so if orthogonality is not enforced, the crosstalk term may be large even for a number of patterns less than D. But it is always possible (although computationally expensive) to project a set of D vectors onto a D-dimensional orthogonal basis (as we saw with PCA). In fact, we do not need to perform PCA, we just need to find a spanning orthogonal set of vectors, which is possible with simpler algorithms. Using such a preprocessor, we can thus say that perfect recall can be achieved for a number of patterns equal to the size of the input layer. Therefore the storage capacity of the linear associator equals D. When the number of patterns is larger than the space dimensionality, a severe degradation of performance can be expected.

NEUROSOLUTIONS EXAMPLE 6.13

LAM and Crosstalk

This example is exactly the same as before, but now we have added more patterns that happen to be correlated (nonorthogonal). This will produce crosstalk. Run the network and observe that now the output values do not correspond to the desired response for two reasons. First, the outputs that were zero now have nonzero values (watch the size of the bars), and second, for some patterns the output does not match the desired values. The errors get worse when the number of 1 in the patterns increases, and also when more patterns are included. This is the problem of crosstalk.

NeuroSolutions Example

This analysis describes the theoretical basis for associative memories. When we train such a system with a set of input-output vectors using Hebbian learning, the network produces an output similar to the individual output, provided that the number of patterns is less than the input space dimensionality. Orthogonalization of the patterns may have to be performed to achieve perfect recall.

LAMs are very different from computer memories. They are content addressable and global, while computer memories are location addressable and local. Hence they have very different properties: computer memory is precise (no crosstalk), has no limitation on size (just increase the width of the address bus), but is brittle. Once a bit is in error, the full memory system breaks down (which requires error correction). On the other hand, LAMs are very robust to errors in the weights, but they suffer from limited storage and crosstalk. They also have the wonderful property of association, that is, the pattern that is closest to the input is recalled. How often have you wished to have the property of association when retrieving information from a computer database.

In Chapter 11 we will see another type of associative memory with recurrent connections that is able to clean crosstalk to a certain extent, clean the noise from the input, or even complete partially occluded patterns by feeding back the output to the input several times. In each iteration a better approximation of the stored pattern is obtained, so the system can self-correct errors. The most famous of these recurrent memories is the Hopfield network.

Go to next section


 6.9 LMS Learning as a Combination of Hebbian Rules

The LMS rule studied in Chapter 1 can be created by a combination of Hebbian-type rules between the desired response and the learning system input. In fact, if we recall the LMS rule

(6.35)

and note that the error [image] can be expressed by

(6.36)

we get

(6.37)

that is, the LMS rule is a combination of a Hebbian term between the desired response and the input and an anti-Hebbian term between the PE output and its input. The first term substitute the desired response for the system output, so it is the forced Hebbian term. LMS is therefore a combination of a forced Hebbian and anti-Hebbian rules.

We can interpret the LMS adaptation as a composition of two different Hebbian forces: the forced Hebbian term that makes the output similar to the desired response and the anti-Hebbian term that decorrelates the input and the system output. The forced Hebbian term does gradient ascent on the performance surface and is unstable as we saw earlier. The anti-Hebbian term decorrelates the input from the output and drives the output to zero, allowing a range of step sizes to produce convergence to the minimum of the performance surface. The anti-Hebbian term controls the convergence of the LMS algorithm, since the product of desired and input responses is independent of the weights. It is understandable that the range of step sizes for convergence of the anti-Hebbian and LMS rules are the same.

An important conclusion is that the Hebbian principle of correlation is also present in supervised learning. This simple derivation calls our attention to the fact that the learning principles studied so far in neurocomputing (Hebbian, LMS, and backpropagation) are based on correlation learning (or compositions of correlation learning).

We can alternatively consider the LMS algorithm as a "smart" forced Hebbian learning rule because it correlates the system output with the desired response as Hebbian learning does, but does so without being unstable (for a range of step sizes) due to the anti-Hebbian component. Hence we can expect that the LMS will improve forced Hebbian learning in the same way as Oja improved the Hebbian learning.

6.9.1 Improving the Performance of Linear Associative Memories (OLAMs)

An alternative to orthogonalization of the input patterns is to use different learning rules during training. We can interpret the individual output pattern as the desired response for the linear associator, and then train it with the error

(6.38)

This equation should remind us of the supervised learning procedure used in regression (Chapter 1), which led to the design of the LMS algorithm. Supervised learning can thus be applied to train the linear associator for heteroassociation. Note that with LMS training the weights are being modified at each iteration by

(6.39)

instead of directly with d (Eq. 6.28). The first term is the desired forced Hebbian update, which is combined with a term that decorrelates the present output y from the input (the anti-Hebbian term). If we compare Eq. 6.39 with Eq.6.28 we can conclude that the anti-Hebbian term reduces the crosstalk term at each iteration, so training the associative memory with LMS is an efficient way to improve its performance in terms of reduced crosstalk for correlated input patterns. A LAM trained with LMS is called an optimal linear associative memory (OLAM).

NEUROSOLUTIONS EXAMPLE 6.14

Optimal LAMs

This example uses the same basic network and files as the previous example, but now we train the LAM using LMS. Notice the difference in the breadboard (the backpropagation plane). Observe the network during training, and watch the response approximate the ideal response obtained with the orthogonal patterns. If we train enough and the number of patterns is less than the size of the space, the ideal response will be obtained.

NeuroSolutions Example

One issue that is worth raising is why are we interested in using forced Hebbian learning to train associative memories when LMS works better. From an engineering point of view optimal LAMs should be used. It turns out that the Hebbian paradigm has been utilized by cognitive scientists to study models of human memory. The mistakes associative memories make have the same general character as human memory deficiencies.

Associative memories trained with forced Hebbian learning become rather bad when the density of 1 in the patterns is high, that is, they work reasonably well only for sparse patterns. We can understand this, since when the patterns have sparse nonzero values (e.g., only 5 ones in 50-bit-long patterns), they are approximately orthogonal, so there is little crosstalk. It turns out that the human brain has so many neurons that the encoding in human memory is very probably also sparse, so Hebbian learning makes sense. Moreover, there is physiological evidence for Hebbian learning while the biological implementation of LMS is unclear at this stage.

6.9.2 LAMs and Linear Regression

You may have noticed that the topologies are similar for the LAM of Figure 6-10 and the linear regression problem we studied in Chapter 1. The marked difference here is that we are interested in multiple-input, multiple-output linear topologies, while in Chapter 1 we studied only the multiple-input, single-output case. But the desired response in multiple regression can also be a vector, in which case the topology for regression becomes exactly that of Figure 6-10. The desired response is effectively the forced response in LAMs. So the difference has to be found in other aspects.

You may recall that in Chapter 1 we used the LMS algorithm to find the optimal regression coefficients while in LAMs we utilized Hebbian learning. But now that we also propose to use the LMS to find the optimal LAM weights, even this difference is watered down. What is the difference if any, between an LAM and a linear regressor, which provide least squares solutions?

The difference is very subtle. In linear regression we want to pass a single, optimal hyperplane by ALL the desired samples, while in the LAM we want to output a response that is as close as possible to EACH of the true forced responses. The answer lies in the number of exemplars. In LAMs we just saw that the number of exemplars must be less than the size of the input layer to guarantee small crosstalk. In linear regression the opposite is true. We normally want (and have) more patterns than the size of the input layer of the regressor, so the real difference between a LAM and a regressor is the amount of data, which provides two distinct solutions to the least squares problem. The solution to the LAM is the unconstrained case (more equations than data), while the regression solution is overconstrained (more data than equations). The solution obtained by the LMS for associative memories is in fact is one of infinitely many solutions (R is not full rank because we have less data than dimensions). It is interesting that the storage capacity quantifies the dividing line between the unconstrained and the constrained least squares. optimal LAMs

This also gives a new insight into the terminology of memorization in our discussion of generalization in Chapters 4 and 5. The linear regressor can either memorize (working as a LAM) or generalize the statistical properties of the input-desired response pairs (working as a regressor). We see that the distinctive factor is the relation between the number of input samples versus the number of network weights. Remember that, if we want to create a regressor and the data is less than the number of input dimensions, LMS will provide an associative memory, not a regressor! This clearly shows the risk and the weakness of MSE learning and emphasizes the importance of capacity control (optimal hyperplanes) discussed in the support vector machine theory, Section 5.8. We conclude that the existence of crosstalk is critical for generalize.

A similar phenomena occurs with a nonlinear system. If we train it for function approximation with a small number of samples (either for nonlinear regression or classification), we may end up with an associative memory, and it will never generalize well.

Go to next section


 7.1 Introduction

Competition for resources is a way of diversifying and optimizing the function of the elements of a distributed system. We see it everywhere, from the way our society is organized to biology. The important aspect of competition is that it leads to optimization at a local level without the need for global control to assign system resources. This is of paramount importance in distributed systems, since for them global control can not be implemented without an enormous overhead.

The PEs of a competitive network receive identical information from the input, but they compete (either by lateral connections in the topology or by the formulation of the learning rule) for resources. The PEs specialize in a different area of the input space, and their outputs can be used to represent in some way the structure of the input space.

None of the networks or learning rules studied so far explicitly use the idea of competition. Backpropagation of errors in a nonlinear network leads to a specialization of PEs, but only indirectly and at a high cost (slow convergence). RBFs, due to their local nature, specialize in different areas of the input space, but this specialization is hard-coded. The PEs in associative memories also specialize, but by using correlation, which is a linear operation. This means that only global restrictions are placed on the solutions found.

In this chapter we introduce specific topologies and learning rules that are geared towards direct competition of resources. Competition is intrinsically a nonlinear operation, and as such, its mathematical treatment lags with respect to other areas of adaptive systems. However, its appeal as a fast method to self-organize the network resources is far reaching and has been used very effectively in practice.

Grossberg introduced most of the ideas of competition in mathematically dense terms. His networks work in continuous time and are complex. Kohonen took a more engineering oriented approach to these ideas and introduced an enhanced set of principles that are easily implemented in digital systems. His approach is the one used in this text.

We divide competition in two basic types: hard and soft. Hard competition means that only one PE wins the resources. Soft competition means that there is a clear winner, but its neighbors also share a small percentage of the system resources. In neurobiology soft competition is widespread and was inspirational for the models developed so far.

Go to next section


 7.2 Competition and Winner-Take-All Networks

The typical competitive neural network consists of a layer of PEs, all receiving the same input. The PE with the best output (either maximum or minimum, depending on the criteria) will be declared the winner. The concept of choosing a winner is a fundamental issue in competitive learning. In digital computers, choosing the best PE is incredibly simple (just a search for the largest value). The concept of choosing a winner, however, often requires a global controller that compares each output with all the others. As discussed many times, global control is troublesome in distributed systems. For this and other reasons (e.g., biological plausibility) we wish to construct a network that will find the largest (or smallest) output without global control. We call this simple system a winner-take-all network. Let us assume that we have N inputs x1, x2, xN, and we want to create an N-output network that will provide only a single positive output in the PE that corresponds to the largest input, while all the outputs of all other PEs are set at zero,

(7.1)

Figure 7-1 displays a topology we can use to implement this network.

Figure 7-1 Winner-take-all network

Each PE has a semi linear nonlinearity (clipped linear region) with a self-exciting connection with a fixed weight of +1, and it is laterally connected to all the other PEs by negative weights - [image] (lateral inhibition) where 0 < [image] < 1/N. We assume that all inputs are positive, the slope of the PE is 1, and the initial condition for no input is zero output. The input is presented as an initial condition to the network, that is, it is presented for one sample and then pulled back. The lateral inhibition drives all the outputs towards zero at an exponential rate. As all the smaller PEs approach zero, the largest PE (PE k) will be less and less affected by the lateral inhibition. At this time, the self-excitation drives its output high, which enforces the zero values of the other PEs. Hence this solution is stable. Note that the PEs compete for the output activity, and only one wins it.

This network topology is unlike any of the ones we covered so far, because there is feedback among the PEs. We study feedback later, but here we point out that the output takes some time to stabilize, unlike the feedforward networks, which are instantaneous (i.e., a pattern is presented, and a stable output is obtained). This architecture provides a glimpse of the difficulty found in the implementation of the competitive principles in continuous time. Another difference is that the weights are fixed a priori, so this topology does not need training.

The function of the winner-take-all network is unlike any of our previous networks. Only one output is active at a time, which means that one output is singled out from the others according to some principle (here, the largest value). The amplitude difference at the input could be small, but at the output it is very clear, so the network amplified the differences, creating a selector mechanism that can be applied to many different problems. A typical application of the winner-take-all network is at the output of another network (such as a LAM) to select the highest output. Thus the net makes a "decision" based on the most probable answer. For example, in the previous chapter we showed an example of the Hamming network. The winner-take-all network could be used to pick the winner based on the Hamming distance between the input and the stored patterns, thus selecting the symbol that was most likely to have been transmitted.

NEUROSOLUTIONS EXAMPLE 7.1

Winner-Take-All Networks

This example shows how minute differences in the input patterns can give rise to outputs that are easily separable. In a sense, this network magnifies differences. The idea is very simple: The input signal is fed to a network that implements the winner-take-all function. The input is then removed (i.e., the network input works as the initial condition for the winner-take-all layer).

Notice the special arrangement of the lateral connections and its weights. This ensures that the lateral inhibition decreases all outputs until only a single output is significantly different than 1. The self-feedback makes that output grow until the PE saturates. Run the example and see how the convergence time depends on the value of the lateral inhibition. Observe also that these weights cannot be larger than 1/N.

NeuroSolutions Example

As mentioned before, the winner-take-all operation is trivial to implement in a digital computer and for efficency reasons does not use this feedback architecture. The ease with which competition can be implemented and formulated in digital computers has led to a simpler understanding of the principles involved.

Go to next section


 7.3 Competitive Learning

The LAM is an associator, while the winner-take-all network is a selector. Instead of using two networks in series, as discussed earlier, the goal of competitive learning is to devise a learning rule that can be applied to a single-layer topology and that tunes the PEs to different areas of the input space. Each PE works as a decoder for each area. The topology is a simple, one-layer, linear feedforward network (the nonlinearity is only at the final stage where the winner is selected). Hence the ideas of vector spaces can be applied.

Competitive learning is an unsupervised learning paradigm, so it extracts information from the input patterns alone, without the need for a desired response. As in association, we would like the information contained in the input patterns to be encoded in the weights. Correlation can do this for us, but associative memories are linear networks, so all the weights respond more or less to the full input space. An alternative way to represent the data is to let each PE specialize in a portion of the input space. The issue is, How do we keep the weights from responding to the full space?

7.3.1 The Instar Network

Grossberg proposed a PE that is able to recognize a single vector pattern based on a modified Hebbian rule that automatically accomplishes weight normalization for normalized inputs. The network is a single McCulloch-Pitts PE, (Figure 7-2)

Figure 7-2 The instar PE

but trained with the following instar rule

(7.2)

where y(n) is taken as the thresholded output that ranges from zero to one. Analyzing Eq.7.2 we conclude that the weights are updated only if the PE is active since the output multiplies (gates) the weight update. This rule is similar to Oja's (Eq. 6.13) but note that the weight is not multiplied by the output. Gating the update implies that the PE learns only when the output is active. When the PE is active the instar rule has the net effect of moving the weights toward the input in a straight line proportional to the size of [image] (Figure 7-3). Since the weights are attracted to the input position, if the input is normalized, the weights also become normalized.

Figure 7-3 Weight movement in the instar rule

A trained instar PE provides a response of 1 for data samples located near the training pattern (the bias controls the size of the neighborhood). Hence the instar rule is able to recognize one vector pattern or a class "similar" to the pattern stored in the weights. The instar is a many-input to a single-output-network that creates a 0/1 type of association.

NEUROSOLUTIONS EXAMPLE 7.2

The Instar Network

We saw that one of the disturbing facts about Hebbian learning was the divergence of the weights. One alternative not yet explored is to compute the weight update with the output of a nonlinear PE. This brings two advantages: First, the update is going to be a fraction of the update for the linear case, because y is thresholded. Second, when the output is zero, the update will be zero. These simple facts cure the divergence of the Hebbian update and do not allow the PE to forget old memories. The simple instar network receives a vector input and produces a scalar output, which responds to the activation seen during training.

NeuroSolutions Example

7.3.2 Competitive Rule with the Winner-Take-All Network

The instar rule is very similar to the principle we seek, since its weights are updated only when the PE is active (i.e. the data is close to the present weight value). It is interesting to note that this rule is similar to Hebb's rule and Oja's rule.

As in Oja's rule, the second term helps take care of the instability in Hebb's rule. When we use the instar rule with a nonlinear PE, however, infrequent patterns are preserved as long as the PE is inactive. If we use the instar rule in a winner-take-all network, we do not need to gate the Hebbian update with the PE output, because the competition does this automatically, that is, y(n) in Eq. 7.2 is

(7.3)

Hence, the competitive rule defaults to

(7.4)

where i* is the PE that wins the competition. All the other PEs keep their previous weights. Note that this rule first assumes a competition step to find the PE that is closest to the input and then updates its weights by moving them toward the input (Figure 7-3) in a straight line from the previous weight location.

The step size [image] (0 < [image] < 1) controls how large the update is at each step and requires the same considerations as other step sizes we have used previously. If the step size is close to 1, the network will converge fast, but any new input can upset the clusters. On the other hand, small [image] makes the convergence slow. A compromise is to use a variable step size during learning. Start with a large [image] and then decrease it progressively.

NEUROSOLUTIONS EXAMPLE 7.3

Competitive Learning

In this example we show the competitive learning rule applied to a single PE. The new component is called Standard Full Competitive, and it is implemented as a Synapse that changes its weights according to Eq. 7.4.

The weights of the PE move in a straight line between their current position and the current input, with the size of the movement dictated by the step size. Assuming that we have a cluster of input points, small step sizes result in slow convergence but the trajectory is smoothed so that the PE seems to move directly toward the center of the input cluster. Large step sizes create faster convergence but each individual input has a larger influence on the weights of the PE. Thus larger step sizes result in a noisier trajectory, which is a problem at the end of training, since there will be significant rattling about the center of the cluster.

NeuroSolutions Example

7.3.3 Criterion for Competition

Thus far we have not discussed which criterion is used to select the winner in a winner-take-all operation. Since we are interested in associating inputs with PEs, a proximity measure is in order, that is, the PE that is closest to the present input should win the competition. The inner product between the input and the present weights is an efficient choice since it can be computed locally by w [image] x. However, the inner product is sensitive not only to directions but also to the length of vectors: that is, it may provide the wrong metric in cases where the inputs and weights are not normalized. Such a case is illustrated in Figure 7-4. The PE that wins the competition is not PE1, which is closest to the input x, but PE2, which has the largest inner product due to an unusually high weight value.

Figure 7-4 Wrong choice of winner for the dot product

The inner product can still be used if the inputs are normalized to unit length (which will also make the weights normalized), but it may distort the data, affecting the clustering accuracy. An alternative is to implement the Euclidean distance as the metric to define the winner:

(7.5)

where the summation runs over the input dimensions. Since the square root is a computationally expensive function, this distance metric is less efficient than the inner product. Also, do not forget that while the winner is found with a maximization operator for inner products, a minimization operator is needed, with the Euclidean distance:

(7.6)

Sometimes the L1 norm (or the Manhattan metric) is preferred, since it involves only subtractions and absolute values.

NEUROSOLUTIONS EXAMPLE 7.4

Winner Metric and Performance

This example shows how the metric affects the competitive algorithm. Remember that the dot product must have its data normalized to unit length to work properly. At first this may not seem like a harsh restriction, but remember that in a 2-D space only points on the unit circle have unit length. This is a very small portion of the input space, and forcing each point to be on the unit circle may significantly warp the natural clustering of the data.

NeuroSolutions Example

Go to next section


 7.4 Clustering

The competitive rule allows a single-layer linear network to group and represent data samples that lie in a neighborhood of the input space. Each neighborhood is represented by a single output PE. This operation is commonly called clustering in pattern recognition. From the point of view of the input space, clustering is dividing the space in local regions, each of which is associated with an output PE. The input space is divided as a honeycomb. The weights of each PE represent points in the input space called prototype vectors. If we join prototype vectors by a line, its perpendicular bisector will meet other bisectors forming a division that resembles a honeycomb (Figure 7-5). Mathematically this division is called a Voronoi tessellation, or simply a tessellation. Data samples that fall inside the regions are assigned to the corresponding prototype vector. Clustering is therefore a continuous-to-discrete transformation.

Figure 7-5 Example of a tessellation

The most important engineering application of competitive learning is vector quantization. In telecommunications, data reduction is needed for economic reasons (e.g., to carry as many telephone calls as possible over a single channel or to store audio signals as efficiently as possible). An algorithm that has achieved the role of a standard in communications is vector quantization. With vector quantization, instead of transmitting the values of each data sample, the data is first categorized in clusters for which the centers, called the codebook entries, are known to the transmitter and the receiver. Then just the cluster number is transmitted instead of the data samples. At the receiver the cluster number is replaced with the codebook entry to recreate the transmitted signal. We can see that if we have enough clusters, the error between a data sample in a neighborhood and its center is small, so the fidelity is high. We gain two things with this technique: First, we do not need to send the data samples in full precision, we just send an integer. Moreover, the more-frequent samples can be encoded with small numbers so that the overall data rate is decreased.

The ultimate requirement of a vector quantizer is to have a set of clusters that minimizes the distance between the centers of each cluster and the input that falls into each cluster. Thus, for good-quality output the vector quantization must use a high quality clustering algorithm to minimize the residuals for each input. We will see the requirements of vector quantization later.

From a theoretical viewpoint, clustering is a form of nonparametric density estimation. In the absence of a desired response, the best we can do for categorization is to use the information about the input data distribution to separate inputs into groups that share the same region in data space. The basic idea of clustering is to seek regions of high sample density - data clusters- and represent their centers in the network. Cluster centers thus represent the local means of the data distribution, where the definition of local is based on the number of centers we are willing to have.

The typical non neural clustering algorithm is the K-means clustering algorithm (Duda). The idea is to find the best division of N samples by K clusters Ci such that the total distance between the clustered samples and their respective centers (that is, the total variance) is minimized. The criterion as an equation reads

(7.7)

where [image] i is the center of class i. This is similar to regression, except that the residuals in linear regression are the distance from each point to the regression line. In clustering, the residuals are the distance between each point and its cluster center. Minimizing the residuals (total variance) provides the "best" clustering. The K-means algorithm starts by randomly assigning samples to the classes Ci, computes the centers according to

(7.8)

then reassigns the samples to the nearest cluster, and reiterates the computation. One can show that J decreases at each step until it reaches a minimum.

If we interpret J as a performance criterion, its minimum can be obtained by taking the gradient with respect to the unknowns (the centers). If we use the gradient estimate proposed by the LMS, an on-line version of the algorithm will modify the centers incrementally according to

(7.9)

Note from Eq. 7.4 that this is exactly the competitive update if we link the cluster center with the jth PE weight. Competitive networks thus implement an on-line version of K-means clustering instead of the required batch adaptation of K-means.

NEUROSOLUTIONS EXAMPLE 7.5

Clustering and Voronoi Tessellation

In this example we demonstrate clustering and classification. The main data set we will use consists of data collected from a machine tool. The data is the X and Y displacement of the tool's spindle and can be clustered into three regions. These three regions roughly correspond to normal operation, chatter in the tool, and breakage. We also introduce a new display in this example. The Voronoi diagram is used to differentiate between clusters in the input space. It is created by scanning the input space and finding the assignment given by the winner-take-all network. Each PE corresponds to a cluster or region of input space (where it is the closest to all the points in that region). The Voronoi diagram shows the different regions of input space (or clusters) and is helpful in determining how the clustering will generalize to other data points. The output of such a system can be interpreted as classifying the samples, but notice that here only the information about the input data structure is used.

The Linear Scheduler is a component that schedules the step size of the competitive layer. Basically it linearly ramps the parameter of the component that it is attached to (here the step size) between an initial and final epoch count. The initial value of the parameter is read from the component. The slope is given by the value of beta: If beta is negative, the component parameter is decreased; if beta is positive the parameter is increased. Moreover, the user can select an upper and lower absolute bound for the parameter. This component should be used with all the unsupervised components.

NeuroSolutions Example

7.4.1. Clustering and Classification

It is important to understand the difference between clustering and classification. Clustering is the process of grouping input samples that are spatial neighbors. Classification involves the labeling of input samples via some external criterion. Clustering is an unsupervised process of grouping, while classification is supervised. Figure 7-6 shows an example of clustering and classification. Notice that there are four different clusters of data and two classes of data. Since clustering is determined only by the natural grouping of the input data, there are many possible clustering combinations. For example, clusters 1 and 2 could be considered one cluster, which in this case will agree with the class 1 label, but it is obvious that many other cases are possible.

Figure 7-6 Clustering versus Classification

Since clustering is unsupervised, it cannot be used directly for classification. However, in many practical applications, data from each class tends to be dense, and there is a natural valley between classes. In such cases clustering can be a preprocessor for classification. For instance, if the clustering from Figure 7-6 were used as the input to a classifier, the task would be very simple: Simply classify anything in cluster 1 and cluster 2 as class 1. What we gain are simpler classification networks. Just think of an RBF network with one Gaussian per sample or one Gaussian per cluster.

Go to next section


 7.5 Improving Competitive Learning

Competitive networks have some limitations that need to be discussed. First, we must realize that competitive learning algorithms can be only applied to one-layer structures, which means that only simple (convex) partitions can be created in the input space. Second, the number of clusters (PEs) has to be decided in advance. Third, the learning rates must be properly set, as in any adaptation procedure. We recommend that scheduling be used, initially with a large step size that is progressively reduced during training.

There are other aspects of competitive learning that are not as easily controlled. One is related to the winner-take-all nature of the competition. If a PE weight vector is far away from any of the data clusters, it may never win the competition, so its weights are never adapted. This implies that the PE is not being utilized to solve the problem ( a dead PE). These dead PEs are common when we apply the algorithm to large networks and data sets. There is a procedure called a conscience term to mitigate this problem.

The idea is that each PE should be penalized if it starts winning more than its fair share and should be helped to win otherwise (the PE's "conscience" makes it feel guilty for winning too often and will inhibit it from winning again). The fair share is estimated by the inverse of the number of PEs. Each PE keeps a count of the fraction of time it wins by

(7.10)

where o(n) is the outcome of the present competition (1 if the PE wins, 0 if it loses) and [image] is a small positive constant (0.0001). This is an on-line method of estimating each PE's win percentage. Each time a PE wins, the estimate of its winning percentage is increased, and each time it loses, its estimate is decreased. Each PE updates its own bias as

(7.11)

where [image] is a positive constant (10 is a reasonable value). The bias is the penalty term for winning too often and is subtracted from the normal distance before a winner is selected:

(7.12)

where D(.) is the Euclidean distance (if the inner product is used, change the sign of bi). If the PE was winning less than its share, bi is positive, so the chances of winning the competition increase. On the other hand, if the PE was winning more than its share, the bias is negative, and the distance is increased so that the PE will be less likely to win again. After determining the winner with this modified distance, the weights are updated the same as in conventional competitive learning.

NEUROSOLUTIONS EXAMPLE 7.6

Competitive with Conscience

In this example we introduce a new component called the Conscience Full and discuss its parameters [image] and [image].

The [image] parameter is used to determine how fast the win percentage is updated on-line. If [image] is too small, the win percentage is not updated fast enough, and PEs that are winning too often are not penalized enough. If it is too large, the win percentage rattles too much, and the conscience term has too much influence on the learning. The [image] term is used to indicate how much penalty should be assigned. The larger [image] is, the more evenly distributed the nodes should be over the input data. If [image] is too large, however, the penalty is large for even small differences in win percentage. The setting of the [image] and [image] parameters may sound difficult, but in general there are a wide range of workable values.

NeuroSolutions Example

7.5.1 Determining the Number of Clusters

One of the aspects that tremendously affects the quality of the clustering is the number of centers, which is controlled by the number of output PEs. There is nothing in the algorithm that allows us to find the best number of clusters. When the number of PEs is smaller than the number of clusters, each PE represents more than one cluster and the PE is actually placed at the center of mass of all the observations represented by the PE. This may be very different from the natural cluster centers. On the other hand, when the number of PEs is larger than the number of actual clusters, some PEs may stall (without conscience) or more than one may represent the same cluster, which means that clusters are artificially split. The designer must decide the number of clusters either by using a priori information about the problem or by trial and error. This is a common problem with this type of algorithm.

NEUROSOLUTIONS EXAMPLE 7.7

Performance and the Number of Clusters

In this example we study how the number of PEs affects the clustering performance of a competitive network. If too many PEs are selected, they split natural data modes, causing problems since the data is not divided by natural clusters. However, joining several PEs may still provide a reasonable division. If too few PEs are utilized, one PE is assigned to several data clusters, so the output of the competitive network will never represent the structure of the data well.

Conscience is always used to avoid dead PEs. To see how much the conscience term affects the results (especially with many PEs), you should also set [image] = 0 during the simulations.

NeuroSolutions Example

There is still another problem in competitive learning that does not have a quick fix. The problem is that there is no metric for how good the winner is, because the principle of competition is differential in nature. This means that the PE that wins the competition is the closest to the input, but we do not know if it is far away or close to the PE weights. To quantify the distance, we have to calibrate the output of the network, which is difficult and rarely done. Adaptive resonance theory (ART) takes care of these difficulties, as we will see later.

Go to next section


 7.6 Soft Competition

Thus far we have discussed hard competition where there is only one winner: one PE is active, and all the others are inactive. Soft competition allows not only the winner but also its neighbors (which can be defined in multiple ways) to be active. Soft competition creates a "bubble" of activity in the output space where the closest PE is the most active (highest output) and its neighbors are less active. Soft competition is widespread in the nervous system, and it implements important functions, such as specialization of resources without global control, common forms of representation, and simplicity of access to information. Similar to the winner-take-all feedback network from earlier in this chapter, a softmax network can be created using lateral feedback. In this case, however, the lateral weights vary with distance from the PE. PEs that are close actually excite one another, whereas PEs that are distant inhibit one another. Since more than one PE is active for each input, more than one PE has its weights updated for each input, although the update is typically attenuated by the PE's distance from the winner.

NEUROSOLUTIONS EXAMPLE 7.8

Effect of the Mexican Hat

In this example we briefly show how we can create a softmax network which will help in implementing soft-competition using lateral feedback. In this example we have set the lateral weights from every PE according to the distribution in the following figure:

This is called a "Mexican hat" distribution, in which each PE reinforces its nearest neighbors, inhibits PEs that are a moderate distance away, and has no effect on very distant PEs.

NeuroSolutions Example

Superficially, soft competition seems very similar to hard competition, but this similarity is misleading. Soft competition builds neighborhood relationships between the PEs, that is, they become tied by a similarity metric, unlike the PEs of the winner-take-all network. Topological mappings are possible from the space of inputs to the space of PEs. None of the previous competitive networks have this ability.

One of the important applications of topological mappings is to perform data reduction and preserve local neighborhood information. There are other ways to project the input data onto a smaller space (e.g., PCA), but projections created with soft competition preserve the fine detail of the data clusters. Therefore, if two inputs are mapped to points in the output space that are close, the two inputs will also be close in the input space. This type of mapping has important applications for classification and data visualization.

7.6.1 The Softmax as Soft Competition

We saw in Chapter 3 that for multiple-class classification problems we should use the softmax at the output layer of multilayer perceptrons. The advantage was that we could interpret the outputs as probabilities (as long the network was trained with the MSE criterion). The softmax is effectively a competitive structure that forces the sum of the activity feeding the PE to be 1. It is important to realize that this normalization of activity can be interpreted as competition, because if one PE produces a larger output, the others are forced to be lower, since the sum is constrained. In fact, any normalization, such as the one imposed in statistics by the probability density function pdf (the area under the curve has to be 1) enforces competition.

From another view, limited resources also impose competition. In biology the simple fact that a cell has access to finite energy (food) means that it has to establish a priority for what it needs to do, because any function utilizes energy. What we are saying is that the competition principle is implicitly used in many information processing systems and in statistics. What we did here was to make the principle more explicit and develop networks that learn based on competition. Competitive learning is the least studied of all the learning rules, and much more work is needed to obtain an understanding of competitive learning that equals that of gradient-descent or correlation learning.

Go to next section


 7.7 Kohonen Self-Organizing Map

The Kohonen self-organizing map (SOM) network performs a mapping from a continuous input space to a discrete output space, preserving the topological properties of the input. This means that points close to each other in the input space are mapped to the same or neighboring PEs in the output space. The basis of the Kohonen SOM network is soft competition among the PEs in the output space.

The Kohonen SOM is a fully connected, single-layer linear network (Figure 7-7). The output generally is organized in a one or two-dimensional arrangement of PEs, which are called neighborhoods.

Figure 7-7 Architecture of the SOM with a 2D output

The one-dimensional neighborhood organizes the PEs in a line of elements, so each element only has two neighbors (the preceding and the following PE). A one-dimensional SOM can be thought of as a string of PEs, where each PE is restricted to be near its two neighbors. When the SOM adapts to an input of higher dimensions, it must stretch and curl itself to cover the input space. For example, Figure 7-8 shows a 1-D SOM mapping a 2-D input space. The input is a set of random points in 2 dimensions. The PEs are represented by the solid circles in the second plot (based on their weights), and the lines connect neighboring PEs; this is the image of the string. The output PEs are arranged in this string, which is stretched and curled so that it can cover the input space yet still maintain the required neighborhood relationships – every PE is close to its neighbors.

Figure 7-8 2-D input data and 1-D SOM mapping

On the other hand, a 2-D neighborhood creates more neighbors, which has the effect of creating more flexible mappings. When we choose a 2-D neighborhood in 2-D space, we see that the PEs spread toward the data samples more rapidly. In principle, Kohonen maps could be arranged in higher-dimensional spaces but this is hardly ever made.

7.7.1 The Learning Algorithm

The idea of soft competition is crucial to understanding the function of the Kohonen SOM. The weights connecting the input to the output perform association between weights and inputs. The PE whose weight vector is closest to the present input wins the competition. This description applies also to the competitive rule discussed in Section 7.3, but SOM learning improves on this rule.

In the Kohonen SOM not only the winner of the competition but also its neighbors have their weights updated according to the competitive rule (Eq.7.2). To simplify the computation of the algorithm, the lateral inhibition network is assumed to produce a Gaussian distribution centered at the winning PE. Thus, instead of recursively computing the activity of each PE, we simply find the winning PE and assume the other PEs have an activity proportional to the Gaussian function evaluated at each PE's distance from the winner. Since we are again applying the instar-type learning rule which scales the competitive rule by the output activity of each PE, the Kohonen SOM competitive rule becomes

(7.13)

where the [image] function is a neighborhood function centered at the winning PE. Typically, both the neighborhood and the step size change with the iteration number (so they can be gradually decreased). The neighborhood function [image] is normally a Gaussian:

(7.14)

with a variance that decreases with iteration. It starts by covering almost the full map and then gets progressively reduced to a neighborhood of zero, that is, only the winning PE gets updated. Note that for the winning PE the adaptation rule defaults to the competitive update of Eq. 7.4. But for the neighbors of the winning PE, the updates are reduced exponentially by the distance to the winning PE. All other weights outside the effective range of [image] remain unchanged. As the size of the neighborhood shrinks, the network moves from "very soft" competition (nearly every PE is somewhat active) to hard competition (only the winning PE is active).

It is the spatial neighborhood that makes the Kohonen SOM so different from the other competitive neural networks. The spatial neighborhood brings the structure of the input space to the SOM output. There is evidence that the SOM creates a discrete output space where topological relations within input space neighborhoods are preserved. What this means is that the distribution of data samples in the input-space is approximately preserved. This makes the SOM useful for density function approximation. Moreover, points that are close in the input space are mapped to PEs that are also neighbors. This property is very interesting for some applications that require the preservation of a input-space metric. It is important to remember that all these properties come about just by using the information from the input. The SOM is created in an unsupervised manner.

How does the spatial neighborhood create this topological mapping? Remember that the competitive rule we are using drives the weight vectors toward the current input. Since the winning PE and its neighbors are updated at each step, the winner and all of its neighbors move toward the same position, although the neighbors move more slowly as their distance from the winning PE increases. Over time, this organizes the PEs so that neighboring PEs (in the SOM output space) share the representation of the same area in the input space (are neighbors in the input space), regardless of their initial locations.

The selection of parameters is crucial to achieve a topology-preserving map from the input space to the discrete output. Experience has shown that there are two phases in SOM learning. The first phase deals with the topological ordering of the weights, that is, where the neighborhoods are defined. Here we assume that it takes N0 iterations. During this phase the neighborhood function should start large, covering the full output space to allow PEs that respond to similar inputs to be brought together. But the neighborhood needs to shrink to a single PE or a PE and its nearest neighbors. A linear decrease in neighborhood radius specified by

(7.15)

is normally used. During this period the learning rate should be high (above 0.1) to allow the network to self-organize (and eventually untangle the map). The scheduling of [image] is normally also linear:

(7.16)

where [image] 0 is the initial learning rate and K helps specify the final learning rate.

The second phase of learning is called the convergence phase, and it is usually much longer. In this phase the PEs fine-tune to the details of the input distribution, and the learning rate should be kept small (0.01) while using the smallest neighborhood (just the PE or its nearest neighbors).

The choice of the number of PEs is done experimentally. The number of output PEs affects the accuracy of the mapping, and the training time. Increasing the number of PEs increases the resolution of the map but also dramatically increases the training time.

NEUROSOLUTIONS EXAMPLE 7.9

Kohonen Linear Neighborhood

In this example we demonstrate the operation of a one-dimensional SOM, which is implemented with a new component called Line Kohonen Synapse.

We demonstrate the importance of the neighborhood size and the topological mapping capabilities of the SOM. In this example the clusters are presented in sequence. We display the outputs of each PE in the Megascope, so the triangular types of responses (which mean that the PE wins for a given cluster) should be ordered if the map associates neighborhoods in the input space to the output PE. In the first panel we do not use neighborhood annealing, so the linear output map is ordered only by chance. Notice that once the neighborhood is properly annealed, the neighborhoods in the input are consistently mapped to the output space. Play with the annealing rate to see its effect on the proper construction of the Kohonen map.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 7.10

Kohonen 2-D Neighborhood

In this example we demonstrate the operation of a two-dimensional SOM.

We show how 2-D SOM learning is like twisting and stretching a plane to cover the input space. We present several types of input space densities to show how well the SOM works. You should try several different learning rates (and annealing rates) to understand the SOM's sensitivity to training parameters.

NeuroSolutions Example

7.7.2 SOM Properties

Input Space Approximation

The SOM is able to preserve the structure of the input space relatively well. This was a shortcoming of the competitive networks, in which the metric of the input space was lost and only the location of the clusters of the distribution were modeled. This application is very important for data reduction, and it is the goal of vector quantization.

Topology Ordering

The PEs in the SOM output are topologically ordered in the sense that neighboring PEs correspond to similar regions in the input space.

Density Matching

The SOM preserves the data densities of the input space reasonably well, that is, regions with more data points are mapped to larger regions in the output space.

NEUROSOLUTIONS EXAMPLE 7.11

SOM Topological Ordering

In this example we demonstrate the topological ordering, input space approximation, and density-matching capabilities of the SOM. This is done with different sample densities. You should run each example and modify the parameters so that you obtain the prescribed results. Note that we are training the SOM faster than normal to save time. As a result the desired properties are not always obtained.

NeuroSolutions Example

7.7.3 SOM as a Vector Quantization Algorithm

There are well established algorithms to perform vector quantization, such as the LBG algorithm. The LBG algorithm has been shown to be optimal in terms of minimizing the average distortion measure between the input sample x and the transmitted x' vector (the class center)

(7.17)

where f(x) is the pdf of the input. In the theory an internal encoding-decoding step is assumed (Figure 7-9), where the input is first coded, creating c(x) which is eventually added with noise and finally decoded to obtain x'.

Figure 7-9 Vector quantization model

The interesting point is that in order to reach optimality, two conditions are required:

• For an input sample x the code c(x) should be chosen to minimize the distortion between x and the reproducing vector x'.

• Given the code c, the reproduction vector x' should be the centroid of the input vectors that were coded by c(x).

The LBG algorithm is a batch-mode version of the Kohonen algorithm with a neighborhood size of zero. The Kohonen SOM can thus be interpreted as an optimal vector quantization algorithm.

Go to next section


 7.8 Creating Classifiers from Competitive Networks

All the unsupervised rules explained so far only utilize the information contained in the input samples. For instance, clustering is simply defined by the structure of the input data (and the number of PEs). In some cases a clustering algorithm can be used for classification, since it responds to a distance metric in the input space. Therefore, when one PE wins the competition this says that the input data belongs to a certain region of the input space, which is the purpose of classification. But classification is more general: First, classes can be constructed by more than one cluster. Second, in classification we know the labels for some of the data (the training set), so the classifier can place the discriminant function not only as a function of the input data structure, but also as a function of the class label information. If the classes are unimodal and well separated, a clustering algorithm and a classifier will yield the same results. However, when the class structure is intertwined and the classes have a multimodal structure, classification and clustering produce very different results. Nevertheless, we get the feeling that we should be able to create a classifier from a clustering algorithm.

The RBF network is a neural topology that explores this concept. Since the Gaussians are local in space, when a Gaussian PE has a high response, it means that the particular input is located in a certain area of the input space, and the output weights can learn the mapping to one of the classes. This is what we will do next, but the Gaussian layer is going to be replaced by a competitive network.

7.8.1 Learning Vector Quantization (LVQ)

Kohonen proposed the learning vector quantization algorithm (LVQ) as a way to fine-tune the boundaries of the tessellation regions using information from class labels. The idea is very simple to state. If the competitive rule produces the correct class label, then no modification to competitive learning is necessary. However, if the clustering produces the wrong assignment, then the PE weights should be "repelled" from the present cluster. This is achieved by the following rule:

(7.18)

This rule has some problems when the input data is multimodal, but the algorithm works well to fine-tune the boundaries of the clusters according to the classification labels. Kohonen proposed more sophisticated LVQ algorithms to deal with multimodal data [Kohonen].

NEUROSOLUTIONS EXAMPLE 7.12

Learning Vector Quantization

This example constructs a classification network from a competitive layer trained with LVQ. The real advantage of LVQ is that it always assigns the same PE to the same data cluster. Moreover, it is also able to fine-tune the cluster boundaries by exploiting the information of labeled data.

We have created a classification problem where the data clusters are unimodal. Competitive learning in this case produces an unacceptably high error rate (see the confusion matrix). However, when LVQ is used the cluster centers are modified according to the class label information and the results are much better.

It is important to note that LVQ does not always work as well as shown. When the data clusters are multimodal or more complex, LVQ may yield unacceptable performance. This is shown at the end of the example.

NeuroSolutions Example

7.8.2 Counterpropagation Networks

Hecht-Nielsen proposed a hierarchical feature classifier composed of a hidden competitive layer that feeds a layer of linear PEs (Figure 7-10). The competitive layer is trained first, and then the output weights are trained using the class labels (desired response) with the LMS rule, very much like the training of the RBF network.

Figure 7-10 Counterpropagation network

NEUROSOLUTIONS EXAMPLE 7.13

Counterpropagation Networks

We construct here a counterpropagation network. This network is built from a layer of competitive PEs implementing a winner-take-all function. On top of the competitive layer we place a perceptron. This part of the network has to be trained with supervised learning. Notice that the back plane goes only until the Winner-Take-All Axon, since the first layer is trained competitively.

The problem we are solving is the machine tool data of Chapter 5. Here the solution is also very good and training very fast. After convergence, the output-layer weights of the counterpropagation network simply describe the average value of the class labels. This can be utilized to create a vector quantizer.

You should experiment with the number of PEs and discover its importance in terms of classification accuracy.

NeuroSolutions Example

An interesting aspect of the counterpropagation network is that since the hidden layer has digital activations (on/off), the LMS weight update can be written as

(7.19)

since the network output is simply the weight value connecting the active hidden PE and the output PE. This means that the output-layer weights are the average value of the class label for that particular tessellation region. In other words, the network learns to implement a table lookup function, where the key value is the output weight and the key encoding is implemented by the unsupervised portion of the network.

Counter-propagation networks can fully implement a vector quantizer. The unsupervised part provides the clustering, and the supervised part can implement the code to be sent to the receiver. These networks train very fast when compared with the MLP (which can also learn how to implement a vector quantizer).

7.8.3 Grossberg's Instar-Outstar Network

At the beginning of this chapter we described the instar network. A very similar network, called the outstar network, can associate a scalar input to a vector output for pattern recall. The learning rule is obtained by switching the roles of inputs with outputs in Eq. 7.2, that is,

(7.20)

Note the similarity of Eq. 7.20 to Eq. 7.19, which trains the output of the counterpropagation network. If the output is substituted by the desired response, the result is exactly the same.

Grossberg proposed a three-layer network composed of an input layer that does normalization (using shunting activation), followed by a competitive layer linked to an outstar (Figure 7-11). The connection between the input and the competitive layer is a set of instar PEs, i.e. the winner of the competitive layer is the PE that has the weights closest to the current input. The outstar then associates that PE with a predetermined output, which normally is done with supervised learning. Hence y(n) in Eq. 7.20 is replaced with the desired response d(n). Thus, we have exactly the same function as the counterpropagation network. The difference between the two networks is the normalization of the input layer. In addition, Grossberg did all the development in continuous time for real time operation, which is more involved.

Figure 7-11 Grossberg instar-outstar network

Go to next section


 7.9 Adaptive Resonance Theory (ART)

The instar-outstar maps input data to output patterns. But can the same basic architecture be used for cluster discovery? This is possible if the desired response becomes the input, d(n) = x(n), and the output of the system becomes the competitive layer. The instar creates a winner, and the outstar maps it back to the input to check the similarity between the cluster and the data. In this way we can check whether the cluster is a good match to the input, which is impossible to do directly in competitive learning. In fact, competitive learning assigns the winner based on a min (or max) function: that is, the difference between the input and the cluster prototype (the mismatch) can be large in absolute terms. The winner can therefore be a fairly poor match for the input pattern. The instar-outstar described here solves this problem, but the network has some shortcomings for cluster discovery.

The big problem of the instar-outstar network is the instability produced by the instar learning and the limited resources. The special form of the instar rule Eq. 7.2 imposes a decay on the associations. This means that old associations tend to be overridden by new data. We could decrease the learning rate to keep old memories but this would negatively affect learning of new associations. This is the so called stability -plasticity dilemma.

To solve this dilemma, we can use the outstar portion of the net and require that the present data must be within a prespecified distance to the cluster center. If not, the sample will not be used to update the cluster weights. The problem is that we would not use all the data for training. Grossberg implemented a facility to add new PEs to the competitive layer when there is no "resonance" between the present data and the existing clusters, that is, when the input data is outside a prespecified neighborhood of the current winner. The specification of the neighborhood is controlled by a vigilance parameter.

This network has been called adaptive resonance theory (ART), and is able to dimension itself for data clustering, unlike any of the other competitive networks. Moreover, it can also be used for detecting previously unseen patterns when it creates new clusters after the learning period. There are important engineering applications for novelty filters, such as failure detection. However, setting the vigilance parameter is a delicate operation.

Go to next section


 8.1 Time Series and Computers

This chapter studies linear systems that work with time signals. Thus far we worked with several types of multidimensional data sets, but we hardly ever referred to time. Problems that do not involve time (such as the classification of patterns) are called static. Time establishes an order in the input data: that is, the data samples can be indexed by a continuous variable t (Figure 8-1), and we will call them time signals or time series. As you may expect, time provides an extra structure to the input space that can be exploited with appropriate topologies such as the delay line. Time also brings the need for other data analysis tools. Engineers long ago discovered that the linear processing of time signals can be better understood in what is called the frequency or spectral domain. We will briefly study the frequency domain and show that we can easily quantify the effect of a linear system on a signal as modification to its spectrum.

Most of the messages that we perceive from the external world with our sensory organs (such as hearing, vision, and smell) and the man-made systems such as radio and TV transmissions are phenomena that exist in time. The messages are normally coupled with physical variables (pressure waves in hearing, light waves in vision, electromagnetic waves for TV) that can be modeled as real functions x(.) of a real variable t. This means that for every value of t, x(t) is a real value D function. Time is a continuum, and the physical variables are also continuous, so these signals are called analog signals. We normally impose some simplifying constraints on x(.) to make the mathematics easier, without affecting the applicability of the conclusions to the signal of interest. For the discussion to follow it is sufficient to assume that x(.) is smooth and has finite energy:

(8.1)

Digital computers cannot work directly with analog signals (nature of analog signals). We need to sample the analog signal. The process of sampling can be visualized as "stopping" time at instant t = n0, and making a measurement x(n0) on x(t), i.e.

(8.2)

The physical device that implements sampling is called an analog-to-digital (A/D) converter. Normally the signal is sampled several times. For mathematical and implementation simplicity, sampling is done periodically. Every T seconds, called the sampling period, the analog signal x(t) is measured, producing a signal {x(nT)} called a sequence, or a time series. The time series x(nT) is a discrete version of x(t).

Figure 8-1 Analog-to-digital conversion

Notice that after the A/D converter we get a sequence of real numbers

so the independent variable of the signal was transformed from a real number to an integer. Rigorously speaking, the A/D converter also quantized the amplitudes of x(t) so that we can store the values of x(nT) in a finite number of bits, but this is not important to this discussion quantized signal.

Digital computers can work with a quantized x(nT) directly. For instance, each sample can be stored in a computer memory location of b bits (where b is the number of bits of the A/D converter), and arithmetic can be directly performed with each sample in the computer ALU (arithmetic and logic unit).

Sampling changes the nature of x(t), since the signal becomes a function of a discrete variable. The important question is: Do we lose any information when going from x(t) to x(nT)?

If we are careful, we don't. It all depends on how fast we sample the signal x(t) with respect to its fastest rate of change. Intuitively, we expect that if the analog signal x(t) is sampled at a sufficiently high rate (i.e., with a high enough density of measurements), its sampled version x(nT) should contain the same information as the original x(t). sampling for accuracy

A central result in sampling theory says that if the signal x(t) is sampled at a rate fs more than twice the value of its largest frequency, that is,

(8.3)

then x(t) can be recovered from x(nT) precisely, and the samples x(nT) contain all the information to reconstruct the analog signal. This is called the Nyquist theorem, and the smallest rate that meets this criterion for a given signal is called the Nyquist sampling rate.

It is very important to know what frequency means and how to measure the largest frequency of x(t) to apply this theorem. Nyquist theorem states that even when interested in analog signals x(t), we can study their properties using their discrete versions x(nT). This result revolutionized signal processing, because engineers could use computers to attenuate noise or otherwise modify signal properties, instead of building analog hardware. It also gave rise to the exciting field of digital signal processing (DSP). Here we concentrate on digital signal processing concepts.

Go to next section


 8.2 Vectors and Discrete Signals

Signals that have finite energy have a one-to-one correspondence with vectors. The correspondence is trivially explained for discrete signals of a finite length NT, such as the ones we are interested in here. Consider the finite duration signal {x(n)} with N samples x(n), x(n-1),.......x(n-N-1) (we drop T from the notation when no confusion arises). One way to create a vector from {x(n)} is to think of {x(n)} as a point in a N dimensional space, where every sample is the projection of {x(n)} onto the N orthonormal axes [image] 0, [image] 1,... N-1 (Figure 8-2).

Figure 8-2 A discrete signal as a vector

We can therefore represent the discrete signal {x(n)} by a vector x of N components:

(8.4)

where the superscript T means transpose. Its length (or its norm) assuming an orthonormal basis is

(8.5)

which can also be recognized as the square root of the energy of the discrete signal {x(n)}. Hence, for finite energy signals, the length of the vector is finite. A discrete signal of length N is thus a vector in an N-dimensional vector space.

8.2.1 The Delay Operator

One important question is to find the basis { [image] i} assumed in the decomposition of Eq. 8.4. When the signal x is written in terms of the projections in each basis we get

(8.6)

Comparing Eq. 8.4 with Eq. 8.6 we see that [image] which means that

(8.7)

where [image] is the Dirac delta function, which we already encountered in Chapter 5. We conclude that delta functions are the bases to represent discrete signals in the time domain. We saw in Chapter 5 how important the projection theorem was to explain function approximation. We are also seeing here how important the projection theorem is for understanding digital signals.

This reasoning leads to an equivalent way of expressing discrete time signals as combination of delta functions, if we substitute Eq.8.7 in Eq. 8.6:

(8.8)

This means that in order to represent a discrete signal of length N as a vector, we need to have access to the current and the past N-1 samples. In an on-line operation, each time tick generates another sample, therefore we have to store the past samples needed to construct the vector. A delay line is the natural topology to implement the decomposition of Eq. 8.8. The ideal delay element is a linear system, denoted by z [image] (z notation) that delays the input signal by one sample (Figure 8-3).

Figure 8-3 The ideal delay element

A delay line is a single-input, multiple-output system built from a cascade of these ideal delays. If we cascade N-1 delays together we will be able to work with the present sample n and access N-1 samples into the past (i.e., back to time n-N+1) (Figure 8-4).

Figure 8-4 Comparison of time signals with static patterns

8.2.2 Signal Space

The space whose axes are the signals at the taps of the delay line is called the reconstruction or signal space. With every time tick the signal vector slowly changes position in phase space, creating a trajectory that is called the signal trajectory (Figure 8-5).

If you recall from Chapter 5, data samples from static problems were also interpreted as vectors in a space of dimensionally equal to the size of the data components (Figure 8-4). Therefore, there seems to be no real difference between time series in signal space and data samples with no time tagging. In principle this is true, but there are some practical differences that are worth mentioning.

Let us construct a new pattern by adding a new sample. When a new sample x(n+1) of the time series comes into the delay line, all the elements of the delay line shift down one sample to make room for the arriving sample, and the last element x(n-N+1) is lost. All the intermediate values are still stored in the delay line, but in a different position (Figure 8-5). The vector constructed in this way for time n+1 is different, but not radically different from the vector at time n, which is unlike the static case where a new set of N numbers are brought from the database for each new input. Therefore a system that uses static data has a response that is totally unrelated from pattern to pattern.

On the other hand, the sequential flow of samples through the delay line produces a sequence of vectors in the reconstruction space that are related by the structure of the time series. In fact, the projection on the x axis at time n becomes the projection on the y axis at time n+1, and so on, producing a spiraling trajectory. The points visited in the reconstruction space (the signal trajectory) obviously depend on the properties of the time series and may allow a system connected to the delay-line output to extract and model the time structure of the time series.

Figure 8-5 The tap delay line and the reconstruction of the signal trajectory in phase space

Suppose that we are interested in analyzing a single, discrete, periodic signal with 1,000,000 samples. This signal is a vector in a million-dimensional space, which is an enormous space. From what we know of the curse of dimensionality, the problem seems hopeless. This is where the structure embedded in time signals comes to our rescue. The signal time structure will constrain the possible locations of the vectors in the input space, effectively limiting the position of the signal trajectory to a subspace of much lower dimensionality. The subspace where the vectors created from the time signal exist is called the signal subspace. This dimensionality reduction can be huge. For instance, the requirement to represent any periodic signal is a reconstruction space of at most the length of the period in samples. In our example, if the period is 50 samples, we need at most 50 dimensions to represent the time series, irrespective of the length of the overall signal. The signal trajectory of periodic signals becomes a closed curve with a length equal to the period, and the points visited in state space thereafter are the same.

It turns out that the actual number of dimensions to represent the signal subspace of periodic signals is normally much less than the size of the period in samples. It all depends on the complexity of the trajectory, that is, the structure of the time series. The constant signal is a point in state space. A sine wave creates the simplest signal trajectory, which is an ellipse with the principal axis aligned with the bisector of the first quadrant. The ellipse always exists in a plane. The number of points (and the eccentricity) of the ellipse changes with the number of points per period, but the shape remains the same. Therefore we only need two dimensions to represent any sinusoidal signal irrespective of the number of samples. Notice also that the trajectory does not cross itself, that is, there is a one to one mapping from the time series to the trajectory and vice versa. This shows that choosing the dimension of the reconstruction space equal to the number of samples of the period may actually be inefficient. However a very complex periodic signal (for example, a segment of white noise periodically repeated) may in fact require a reconstruction space as large as the period in samples. When the reconstruction space is not large enough, the signal trajectory may cross a previous branch. When this happens, we lose the one-to-one mapping between trajectories and time series. For example, when we reach a point where two branches cross in signal space, we do not know which exit branch to take.

NEUROSOLUTIONS EXAMPLE 8.1

Signal Space

We will use the 3-D State Space Probe to show the reconstruction of a sine wave in 3-D space. The 3-D State Space Probe uses the current sample and 2 samples from the past of the signal to plot the current point in 3 dimensions. If the input is periodic, like a sine wave, then the reconstruction periodically visits the same points in space. The structure of the time signal limits the region of the 3-D reconstruction space where the trajectory lies. For the case of the sine wave, the trajectory is an ellipse, which always exists in a plane (2-D). Even if the reconstruction is done in a higher-dimensional space, the signal trajectory still resides in a subspace. Change the sliders of the 3-D display to see fully the trajectory.

In the case of the sine wave, the reconstruction space is a plane that is oriented along the first quadrant bisector. If you change the delay used in the reconstruction, the trajectory "opens up", and the ellipse approaches a circle. Notice that the delay selection does not drastically change the overall shape of the trajectory, but the relative position of the reconstruction space varies, and the trajectory covers a wider area in the reconstruction space, since the samples becomes less correlated.

NeuroSolutions Example

One of the goals in digital signal processing is to find a reconstruction space dimension that appropriately quantifies the signal characteristics. The size of the reconstruction space determines the length N of a time window that slides over the full time series (Figure 8-6). The length of the time window may correspond to the size of the delay line and establishes the dimensionality of the reconstruction space.

Figure 8-6 Sliding time window

For processing purposes, the dimensionality of the reconstruction space should be higher than the size of the signal subspace; otherwise there is information in the original time series that is difficult or impossible to capture. Unfortunately, the choice of the time window length is a nontrivial matter, because the dimensionality of the signal subspace is difficult to quantify and also depends on the goal of the processing. This is an added difficulty both in temporal pattern recognition and signal processing. For instance, if we reconstruct the trajectory of Figure 8-5 in 2-D space, a different trajectory emerges which may make the processing of the signal more difficult (Figure 8-7). dimension of reconstruction space

Figure 8-7 Reconstruction of the same signal in a 2-D space

NEUROSOLUTIONS EXAMPLE 8.2

Signal Space and the Delay Line

This example introduces the delay line component which is called TDNNAxon in NeuroSolutions because of the famous time delay neural network. The TDNN level of the Inspector selects the number of taps (the number of delays is the number of taps minus 1 since we always use the present input) and the number of delays between consecutive taps (normally set at 1 sample, i.e. ). We can also select the number of inputs in the PE at the Axon level of the Inspector. In this example we construct a single-input delay line with 2 taps and create a reconstruction in a 2-D space with the Scatter Plot.

For the sine wave, the 3-D and 2-D reconstructions are very similar. Let us add a second sine wave component of a different frequency at the input. The trajectory for the sum of two sinewaves is still clear in 3-D but does not make sense in the 2-D projection since the signal trajectory folds. Play with the controls of the 3-D probe to visualize the 3-D trajectory. Notice that the 3-D trajectory has enough "space" to flow and never intersect a previous branch (except at the end of the period). This does not happen in the 2-D plot. So a system connected to this 2 tap delay line will have a harder time reconstructing one of the sine waves. Experiment with the period of the sine waves.

Experiment also with the reconstruction delay in both the 3-D plotter and in the 2-D reconstruction by changing the number of delays between the taps. The trajectories change shape when the delay is modified, but we can show that they are all topologically equivalent. In principle, all trajectories contain the same information. However, a smoother trajectory facilitates the extraction of time information by a system connected to the taps. The accuracy and speed of learning therefore depends on the delay.

NeuroSolutions Example

8.2.3 Dynamics versus Statistics

We have seen that mapping a time signal to the signal space effectively created a time-to-space mapping. If we take a snapshot of the signal space, we see a cloud of points, just as happened in pattern recognition. In such a case, we are obliged to resort to a statistical description of the data clusters to attain our goal (either regression or classification). But consider for a minute that you have access to the sequence that generates the samples, that is, that you are interested in the dynamics underlying the data. Data clusters that are indistinguishable from a statistical point of view become suddenly different if we can remember the sequence by which the data samples occur. In Figure 8-8, the left cluster of points seem to belong to the same class. However, if we have the two trajectories shown at right, we may conclude that they, in fact, belong to two different phenomena.

Figure 8-8 Static data cluster versus dynamic time sequence

Time helps remove ambiguity from the data. But to do this, the system must have short-term memory, that is, it must remember the past values of the data, and of course our data models must take time into consideration. This is the goal of digital signal processing and more generally of dynamics. Short-term memory helps a learning system capture more information about the time-varying input data. As a rule of thumb, if the problem we are trying to solve has dynamics, we should always use a system topology with short-term memory. This chapter provides a review of the most important concepts about the analysis and design of linear dynamical systems.

Go to next section


 8.3 The Concept of Filtering

8.3.1 The Linear Combiner

An example of processing that we study extensively in this chapter is a linear combination of the signals at the taps of the delay line (Figure 8-9). This structure is called the linear combiner or a linear finite impulse response (FIR) system. The linear combiner is a linear system .

The output of the linear combiner is calculated by

(8.9)

where N is the size of the delay line and w are parameters called weights or filter coefficients. With our interpretation of the output of the delay line as a vector of past samples, we can rewrite this expression in vector notation as the inner product

(8.10)

You may recall the same basic expression from the Hebbian networks in Chapter 6 or the linear regressor in Chapter 1. But remember, in Hebbian networks or linear regression x was a vector of values with no time tagging, while here x is a vector composed of the last N samples of the time series.

Figure 8-9 The windowing effect of the size of the delay line

8.3.2 Filtering as a Projection

The interpretation of the input-output operation remains the same for the linear combiner, the Hebbian PE or the linear regressor. The difference is the space utilized for the analysis. According to Eq. 8.10, a linear combiner produces a scalar output value. The past N inputs provide one vector in an N-dimensional space (with coordinates x(n-i)). Let us assume for simplicity that when time advances, this vector remains in an hyperplane in this N-dimensional space (the signal subspace) (Figure 8-10). The output (interpreted as a vector from the origin to the output y(n)) is a linear combination of these past inputs and must exist in the same subspace. The actual value of y(n) is the projection of x on the weight vector w, which specifies the direction of the projection (Figure 8-10).

We conclude that the linear combiner is a linear projector in signal space. If we modify the weight values w, the output of the system will change because the relative positions of the present input vector and the weight vector are not maintained, so the projection on w will change. This observation is crucial to understanding signal processing with the linear model: a linear system creates a projection of the input on a vector that is defined by the system parameters and that exists in a manifold created by the last signal samples.

Figure 8-10 The projection in the space is controlled by the weights

Notice that even with a fixed weight vector, the projection varies in time because the signal subspace is changing with time. Depending on the relative position of the weight vector and the signal trajectory, this projection may preserve much of the information of the input signal, which exists in a much larger dimensionality space, or may severely distort it. For instance, if the weight vector matches one of the signal-space axes, the output signal is always the same as the input (apart from a delay). If the signal trajectory is in a hyperplane with very different extents in each axis and we use a weight vector along the largest extent, then we also recover the signal with small error. On the other hand, if the weight vector is placed along the smallest extent, we will obtain a signal very different from the original. The role of the linear system designer is to choose the projection direction such that the desired information contained in the input signal is preserved. This is the fundamental idea of filtering. Remember that the linear combiner with N+1 weights has a memory of only N samples. Any event that happens more than N samples in the past does not have any bearing on the present linear combiner output.

NEUROSOLUTIONS EXAMPLE 8.3

Linear Combiner in Signal Space

Let us create a 3-tap delay line with the TDNNAxon to match the 3-D plot. Now let us implement a linear combiner with a Synapse and a Bias Axon. Let us set the weights by hand to (1, 1, 1) to show the output of the linear combiner in the 3-D plot. Change the weights and justify why the signal is modified (the projection changes).

This requires some thinking. Notice that if the weights are 1, 0, 0 (or combinations with just one weight different from zero), the output is exactly the same as the input (except possibly delayed). This means that projections on any of the axes preserve the input wave shape. But if we start combining different weight values (1, 1, 1) the projection is no longer along the axis, and the signal trajectory starts to be distorted (distortion may be good if we want to separate a signal from high frequency noise). The weight vector (1, 1, 1) is along the first quadrant bisector of the reconstruction space, which coincides with the major axis of the ellipse (the sine wave trajectory), so it should preserve the sine wave shape.

Add a noise component at the input. Show that (1, 1, 1) does preserve the sinewave and reduces the noise, while (0, -2, 1) produces predominantly the noise. This last vector is oriented in the orthogonal plane of the sine wave trajectory. This is the principle of filtering: orienting the projection so that particular parts of the signal trajectory are emphasized, while others are attenuated.

NeuroSolutions Example

8.3.3 Filtering Examples

Filtering is a selective and user-defined distortion of the input signal. Many applications in engineering require the selection of only partial information from the input signal. Let us study this operation in the signal space. filtering operation

Probably the simplest example of filtering is the case of a signal created by the addition of two sinusoids of different frequencies, when we want to recover just one of them. We saw that a sinusoid always provides an ellipse as a signal trajectory. When another sinusoid of a different frequency is added to the original signal, the signal trajectory becomes more complex, effectively orbiting a torus. (For simplicity we assume that the amplitude of the higher frequency sinusoid is smaller than the amplitude of the lower frequency one and that the difference in frequency is large.) One of the fundamental differences is that the trajectory is no longer 2-D, that is, signals with more frequencies have more complex trajectories that do not exist in a 2-D space..

Figure 8-11 Illustration of the composite signal trajectory of two sinewaves

Let us assume that we want to recover the sinusoid of lowest frequency (or to filter out the high-frequency sinusoid). Just by imagining the signal trajectory (Figure 8-11) we can see that if the weight vector is chosen as the bisector of the first quadrant (corresponding to the principal axis of the torus) which is (1, 1, 1) for a 3-D space, we will recover the lowest-frequency sine wave. The highest-frequency sine wave will be attenuated, since the samples of the composite signal have zero projection on this weight vector, going up and down the plane of the orbit. There are many other weight vectors for the linear combiner that provide only the lowest-frequency sine wave. These filters are called lowpass filters. Can you think of another choice of w?

Likewise, if we find a weight vector that is perpendicular to the midsection of the torus, we obtain an output that contains only the variation produced by the highest frequency sinewave. Such weight vector is for instance (1, -2, 1), and the linear combiner with these values is called a highpass filter. Can you find another choice for w to implement a highpass filter?

NEUROSOLUTIONS EXAMPLE 8.4

Filtering in Signal Space: The Two Sine-wave Case

In this example we use the previous NeuroSolutions breadboard and show that orienting the weight vector emphasizes different parts of the signal trajectory. Let us use two sine waves of different frequencies as the input to our system. With a 3-tap system the discrimination between the different components is not very large. However, if we use higher-dimensional spaces (i.e. tap delay lines with more taps), the discrimination can be as high as we want: we can completely remove one of the frequencies. A simple lowpass filter has all its weights equal to 1/N, while a simple highpass filter with even number of taps has alternating coefficients (e.g. 1, -1, 1, -1) . These choices provide vectors along the first quadrant bisector and one of its orthogonal directions respectively.

NeuroSolutions Example

Filtering with the linear combiner can always be interpreted as in the previous example, but not all the signals are as easy as the addition of two sine waves. For instance, a triangular wave produces a signal trajectory that looks like a rectangle, but the small sides are effectively tilted at an angle, going up and down of the main plane of the orbit. Therefore, a lowpass filter distorts this orbit, rounding off the corners of the rectangle, since they will be projected to the main plane of the orbit. Likewise, the highpass filter accentuates the corners that go up and down from the planar orbit, creating a waveform that looks like a rectangular wave.

NEUROSOLUTIONS EXAMPLE 8.5

A Triangular Wave in 3-D Space

Let us change the input to a triangular wave and analyze the trajectory. Change the filter weights to see the difference in the projection. If the dimension of the reconstruction space is increased, the projection results are more dramatic, but we lose the ability to visualize the resulting trajectories.

NeuroSolutions Example

For a square wave you should now be able to determine what is going to happen. During the time that the wave is a positive constant the signal trajectory collapses to the point (1,1,1) in 3-D. However, when the signal changes from positive to negative the trajectory goes through the points (1, 1, -1), (1, -1, -1) to (-1, -1, -1). Likewise, when the wave goes from negative to positive the points visited are (-1, -1, 1), (-1, 1, 1), and again (1, 1, 1). Notice how different this is from the smooth trajectory produced by the sinewave. Using a linear combiner with a (1, 1, 1) weight vector, we obtain a smoother signal, which resembles an ellipse of the same period as the square wave. If the projection uses the weight vector (0, -2, 1), a trajectory with many discontinuities appears.

NEUROSOLUTIONS EXAMPLE 8.6

Square Wave in 3-D Space

Let us change the input to a square wave and analyze the trajectory. Change the filter weights to see the difference in the projection. If the dimension of the reconstruction space is increased, the projection results are more dramatic, but we lose the ability to visualize the resulting trajectories.

NeuroSolutions Example

This signal space interpretation provides a qualitative explanation for the function of the linear combiner. We saw that the linear combiner projects the input signal onto a vector that exists in a linear space. The direction of this vector is controlled by the weights of the linear combiner. Depending on the relative direction of the weight vector and the signal trajectory, some of the features of the original signal are preserved and some others are attenuated. This is the idea of filtering. The problem with this interpretation is that it is difficult to quantify exactly what is going to happen to the input signal, which motivates the need for other analysis tools.

Go to next section


 8.4 Time Domain Analysis of Linear Systems

Although the vector space methodology leads to a conceptual understanding of a linear combiner, it is not very usable for the systematic quantification of how the linear system affects the input signal properties. The widespread use of linear systems in engineering is due to the existence of easy and systematic methods to quantify their properties. Signal processing developed two basic methodologies to study linear system properties: the time domain approach and the frequency domain approach. We start with the time domain methodology.

8.4.1 The Impulse Response

A very important quantity to describe a linear system is the impulse response. As the name indicates, the impulse response h(n) is the output of the system when the input is a delta function (also called an impulse) applied at time zero, [image] (n). In linear systems the time that the response takes to stabilize for a constant input is called the transient time, and the system output during that time is called the transient response. The impulse response is sometimes also called the transient response.

The impulse response completely describes a linear system. Nonlinear systems do not have this property, so with a nonlinear system we have to either know the equation for the map or probe all the points of the input space to find the corresponding output (as we did in Chapter 3 with the discriminant probe). For the linear combiner the impulse response is, from Eq. 8.9,

(8.11)

which means that the impulse response of the linear combiner is simply a time function with sample values equal to the weights h(i) = wi (the first time sample is equal to the first weight, etc.). Hence, for an N-delay FIR filter the impulse response will have at most N+1 nonzero values, that is, it always will be of finite extent, hence the name finite impulse response (FIR) for this type of linear system. These properties are easily verified by considering that a single value of one will progressively move through the tap delay line of the linear combiner.

NEUROSOLUTIONS EXAMPLE 8.7

Impulse Response of an FIR Filter

The impulse response of a FIR filter is very easy to visualize. Just enter by hand some values for the weights of the linear combiner, and take note of them. Then choose the impulse function as the input to the filter and run the data sample by sample with the step exemplar button. Place a Matrix Viewer at the output. The output of the linear combiner matches the values you entered as the weights producing a time response. This is easy to understand if we envisage the impulse being multiplied by the weights and being moved forward every time tick. Note that the time response is related to the weights by h(i) = wi . This makes sense, since at time n = 0 the output reproduces the first weight w0, at n = 1 the output reproduces the second weight w1, and so on.

NeuroSolutions Example

8.4.2 Convolution

The importance of the impulse response is the followingfact: the response y(n) of a linear system to an ARBITRARY input x(n) can always be calculated by the convolution of the input x(n) with the system impulse response h(n):

(8.12)

where the [image] indicates convolution. This is a very important property, because knowing the impulse response (for instance after measuring the response of the system to an impulse only once), we can calculate beforehand how the system will respond to any other known input. We just need to program the operation of convolution in a computer to predict the response of the system to hypothetical inputs.

Notice that the impulse response in Eq. 8.12 appears reversed in time since the summation index i has a negative sign (or, equivalently the input appears reversed in time). This time reversal is a mathematical necessity to translate the normal operation of the system when it receives a time signal. Let us look at Figure 8-9 to see what is happening.

Assume that all the taps of the filter are initialized to zero. The first sample of the input x(0) multiplies the first filter coefficient so the output at n = 0 is equal to y(0) = w0 x(0); for the next time tick (n = 1) the sample x(0) is moved to the next tap and multiplies w1, while x(1) multiplies w0. So the output is y(1) = w0x(1) + w1x(0). For the next time tick, x(0) multiples w2, and the output becomes y(2)= w0x(2) + w1x(1) + w2x(0).Notice how this sum of products coincides with Eq. 8.12 for the same points. The process continues on for every time tick. convolution operation

The convolution for FIR filters can still be computed easily by hand, and it is an excellent opportunity to understand the concept fully. For FIRs the limits in Eq. 8.12 are finite (0 and N) when the input is applied at time n = 0. The convolution of a signal segment of length M with a FIR of length N is at most M + N - 1 samples long.

NEUROSOLUTIONS EXAMPLE 8.8

More on the Impulse Response of a FIR Filter

The same process explained in the previous example works for any input, but since the input normally has more than one sample different from zero we have to multiply and add the contributions from each tap. This is formulated as the convolution sum shown in Eq. 8.12 and explained in Appendix A.

We now do the convolution in NeuroSolutions. Notice that the breadboard has two parts: The left part helps us create arbitrary waveforms by using the impulse function and letting the FIR filter create the wave shape (by choosing the weights). Note that we are using this left FIR as an arbitrary signal generator. The right part of the breadboard implements our FIR system, for which we want to compute the output using convolution.

Use sample-by-sample execution and copy the values produced by NeuroSolutions to a piece of paper. Now compare this with the previous "know more box" which explains the convolution to understand the process. Knowing the input wave shape and the FIR impulse response, you should be able to derive the output of the system. The output of NeuroSolutions is your golden standard: your results should match the ones obtained in the simulator.

In particular, you should understand why you have to reverse in time one of the waveforms (either the input or the impulse response) and find out why the output is of length equal to the sum of the lengths minus 1. Just picture in your mind the FIR filter receiving the samples and computing the output with the block diagram.

Some of the important properties of linear systems, such as: that the output appears only after the input is applied (causality) should be clear from this example. In addition, the output peaks after a few samples (i.e., there is always a delay in the response of the system). If the FIR impulse response is symmetric (called linear phase), it should peak at (N-1)/2 (if N is odd).

NeuroSolutions Example

A problem with the convolution is that it is not a simple, sample-by-sample operation as the addition or multiplication of samples. We see from Eq. 8.12 that to compute the response y(n) for each time tick n, a very large (eventually infinite) number of products have to be added together (as many as the length of the impulse response). We need very fast digital computers or alternative descriptions.

8.4.3 Matched Filters

An important engineering problem is the search for transients of a given shape in a noisy background, which is called detection. Areas where detection is important are radar, sonar, and biomedical applications. One way to create a linear system that can be used for detection is to include the a priori knowledge (the transient wave shape) as the weights of a FIR filter. As you may expect from the convolution, the procedure calls for the reversal in time of the transient shape to specify the filter weights. This filter is called the matched filter precisely because its impulse response is the transient shape reversed in time. If we look at the convolution equation (Eq.8.12) we can see that the output of the matched filter is the sum of the squares of the waveform samples when the impulse response is the waveform reversed in time. This happens to be the autocorrelation function evaluated at lag zero.

The matched filter is an optimal filter in the sense that it maximizes the output signal energy to the noise power at one point in time:

where N is the length of the transient and the denominator is the variance of the noise, which is assumed white in this formulation. The point in time where the output peaks is the last sample of the transient, since at that time the output is R(0), which represents the power in the transient and is the largest value of the autocorrelation function.

The matched filter output can be thresholded and used as a detector when the transients appear in background noise. The amplitude of the transient may be buried in noise, but the matched filter is able to find the correlation in time and transform it into an amplitude value that peaks above the noise floor. Therefore, by observing the matched filter output and finding the peaks, we can obtain the location where the transients occur. This is the best possible linear system for this application, and it resembles the eigenfilter and the Hamming net discussed in Chapter 6, except that now the patterns exist in time. Matched filters are used frequently in communications for optimum receivers.

NEUROSOLUTIONS EXAMPLE 8.9

Matched Filtering

Consider the case of a transmitter that creates a pulse of energy of a known shape, but we do not know when the pulse arrives. To make matters worse, the transmission is noisy, so we receive not only the transient but a great deal of noise. How can we reliably find when the pulse is received?

To solve this problem let us first create the matched filter. This is easy, since we just need to encode by hand the weights of the linear combiner as the samples of the transient (reversed in time). Now we can simply observe the output of the matched filter with a threshold to find where the transients occurred. Note that the setting of the threshold depends on the noise power. If it is set too low, we get too many detections (the ones that do not correspond to the transient are called false alarms, or false postives), while if the threshold is set too high, we may miss some pulse occurrences (called missed detections, or false negatives). The figure of merit for such a system is precisely the plot of the detection probability against the number of false alarms as a function of the threshold (which is called the receiver operating characteristics (ROC) curve). The ROC is the best way to evaluate a detector.

NeuroSolutions Example

Go to next section


 8.5 Recurrent Systems and Stability

Thus far, we have focused on the linear combiner as a finite impulse response (FIR) filter. A second class of systems is composed of infinite impulse response (IIR) filters, and they contain recurrent/feedback connections. An important aspect of IIR systems is that they are not inherently stable like the FIR systems. This section will discuss recurrent systems and their stability.

8.5.1 First Order Recurrent Systems

Let us study the following discrete recurrent system:

(8.13)

where 1- [image] is a coefficient called the feedback coefficient, y(.) is the output and x(.) is the input. This system is called recurrent because the output at time n depends on the output at a previous time (in this case the last output). Equation 8.13 is called a difference equation and it is related to differential equations in continuous time (continuous and digital filters). The block diagram of this system is

Figure 8-12 First order recurrent system

and it clearly shows the feedback connection from the output to the input adder. Note how easy it is to write the difference equation from the block diagram (or vice versa). The initial condition y(0) needs to be specified to compute Eq. 8.13 (normally y(0) = 0). Let us evaluate the impulse response of this system (x(n) = [image] (n)). Just by following the block diagram we can immediately see that

(8.14)

The first remark is to note that the impulse response is of infinite extent. This system therefore does not belong to the same class as the linear combiner. Linear systems that have infinite impulse response are called IIR (infinite impulse response). They must have some type of internal feedback mechanism. Another important observation is that the impulse response shape is controlled by the feedback parameter. If 0 < [image] < 1, the impulse response will decay exponentially toward zero with a decay rate given by the feedback coefficient. It is exactly zero only when n = [image], but for all practical purposes it becomes zero after a finite time n0. The closer [image] is to 1, the slower will be the decay of the impulse response (Figure 8-13). For [image] < 0 or [image] [image] > 2, the response will diverge (for zero initial conditions), that is, it will become larger and larger. Note that this case is impossible to obtain with FIR systems.

Figure 8-13 Impulse response of the first order recurrent system

NEUROSOLUTIONS EXAMPLE 8.10

IIR Filters

We introduce one more NeuroSolutions component that creates the first order recurrent system of Figure 8-12. This system has the one-sample delay in the feedback loop, and the feedback parameter can be set in the Inspector.

We use an impulse at the input and display the response in a MegaScope. The goal is to show that the response is controlled by the feedback parameter. By making the parameter close to 1, the response slowly decays to zero. For values above 2 or less than 0, the response explodes. Let us also use the 3-D probe to see that the response linearly approaches the point (0, 0, 0) in state space when the feedback parameter is between 0 and 1. Finally, input a sine wave to see that the response takes some time to stabilize. The time that the response takes to approximately reach the final value is still called the transient response.

NeuroSolutions Example

The behavior of the first and third cases of Figure 8-13 is qualitatively different. In the first case the response to the delta function decays to zero and the system will follow any input applied thereafter to it, while in the third case the response to the transient never disappears The goal in signal processing is to have the system respond to inputs (eventually taking away unwanted portions such as noise or amplifying the signal, as when we listen to music), so we would like the system to always have a finite-duration transient response . Hence we have to guarantee that the response to transients will die away, which can be done by the control of the feedback parameter.

8.5.2 Definition of Stability

There are many definitions of stability, but the bounded-input bounded-output (BIBO) definition is the most commonly used for linear systems. Let us define BIBO stability. A system is BIBO stable if it responds with a finite-amplitude signal to any input that is also of finite amplitude. From Eq. 8.12 this means that the sum of the impulse response values has to be finite. Hence, the linear combiner with N taps is intrinsically stable.

If the definition of stability is applied to the recurrent system, we see that the system will be stable for 0 < [image] < 1, unstable for [image] < 0 and [image] [image] > 2. A stable linear system has an impulse response that approaches zero so that its sum is finite. For [image] = 0 the stability depends on the initial condition, and it is called marginally-stable for this reason. Hence we see that recurrency brings new properties to linear systems (and also new worries, such as stability).

In the FIR case the impulse response of a Nth-order FIR is the filter weight values ( it takes at most N samples to reach zero). In recurrent systems the impulse response is of infinite extent, and the relationship between successive values is controlled by just a few parameters (the feedback parameters). In the system of Figure 8-12 the stability is captured by the value of the feedback parameter (1- [image]). The absolute value of the feedback parameter must be less than 1 for stability, i.e. |1- [image] |<1.

NEUROSOLUTIONS EXAMPLE 8.11

Second Order Recurrent System

Let us build a second-order linear recurrent system in NeuroSolutions. We have to use a TDNNAxon with two taps and a Full Synapse that feeds the taps back to the input of the Axon. When the interconnection is established, the system complains that an infinite loop is created and we have to include an extra delay in the feedback. Hence a two-tap TDNN connected in this way creates a recurrent system described by the following difference equation

The first weight in the Full Synapse is a and the second is b. You can verify that the system is stable only for a subset of the values a, b. There are several possible regimes of the impulse response. A decaying exponential is obtained for b > 0 and [image]. A decaying, oscillating response is obtained for b < 0, [image] 4b [image] > [image] a² [image] and a² + 2b = 2. For a sine wave is obtained (the system is marginally stable). For all the other values the response is unstable. We will see later how these values come about.

It is instructive to visualize in state space the stable oscillating regime. The system output spirals down to the origin of state space and stays there.

NeuroSolutions Example

Go to next section


 8.6 Frequency Domain Analysis

8.6.1 Signal Spectrum

The impulse response is a very powerful method to describe linear systems, but it is not very practical for the following reasons:

  • When the linear system is described by its impulse response, the system output cannot be computed with sample by sample evaluations (because it requires a convolution between the input and the impulse response).

  • The response of a linear system (with an N-sample length impulse response) to a N-sample length input signal using convolution requires O(N²) multiplications.

  • The time domain description is not very appropriate to quantify the effect of linear systems in terms of filtering, which is at the core of linear signal processing.

An important goal is to seek methods that preserve the descriptive power of the impulse response but can be directly used to describe filtering and to compute the output of the system as a point by point operation and hopefully more efficiently than [image].

Such an alternative technique exists and is based on the idea of Fourier Analysis. In Fourier analysis we decompose the time signal in energy per frequency and call this representation the spectrum. The description of a signal by means of its spectrum is called spectral analysis. Without knowing it, we already covered the fundamental theory that leads to the definition of the spectrum when we studied function approximation in Chapter 5.

Our goal is to seek a new representation for time signals that will make the processing by linear systems easy to understand and apply. Since we are interested in linear systems that are linear operators, we know that their eigenfunctions are complex exponentials. eigendecomposition explained

8.6.2 Decomposition in Phasors- Fourier Analysis

It is interesting to decompose discrete signals in to complex exponentials of the form

(8.15)

where s is a complex variable, n is an integer, a and [image] [image] are real quantities, and T is the sampling period. The quantity w is inversely proportional to time, and it is called the frequency. Frequency is measured in radians per second (or alternatively in Hertz). is called the frequency resolution and can be defined as the smallest frequency increment that can be represented in NT seconds, that is, [image] [image] = 2 [image] /NT . What this means is that if we observe a signal for a long time (large NT), we can discriminate minor frequency changes. If the window of observation is short, our frequency resolution is coarse. The resolution [image] [image] is also the slowest frequency that can be reliably measured in a window of NT seconds.

The spectrum is by definition the decomposition of the signal {x(n)} onto the basis set formed by the complex exponentials [image] with zero real part (a = 0), which are called phasors (concept of phasor). Basically, a phasor is a single frequency component that describes an ellipse in signal space, as we saw in the first example of this chapter. Decomposition in phasors is called Fourier or harmonic analysis.

We saw that a signal x(n) with a finite length NT can be thought as a vector in N-dimensional space. Therefore we need exactly N phasors (each rotating at a different frequency), given by

(8.16)

to span the same space. It is easy to show that the complex exponentials are orthonormal (i.e., they are an orthogonal set of vectors with unit magnitude).

We now have two alternative discrete representations for the signal x(n), one using delta functions, yielding the time domain representation, and the other using phasors or single- frequency vectors. Do we lose any information when x(n) is represented in this new orthogonal basis of the same size as the time domain basis? The length of a vector is independent of the orthonormal basis, so we do not loose any information. The signal is simply decomposed into different elemental components, in other words, the bases are no longer { [image] (n-i)} but become phasors as shown in Eq. 8.13, which lead to easier interpretation in many cases. Parseval's theorem

The projection of {x(n)} on each of the phasors is represented by X(k), where k indicates the phasor used (1 < k < N), that is, k is the frequency index (Figure 8-14).

Figure 8-14 Decomposition of a time sequence in delays and in complex exponentials

X(k) is called the kth Fourier coefficient and the set of all X(k) is called the discrete Fourier series (DFS) of x(n), or the spectrum of x(n).

Unlike x(n), X(k) is a complex function, which can be decomposed into its magnitude, called the magnitude spectrum [image] X(k) [image] , and its phase, called the phase spectrum [image]. We can represent x(n) in the time domain or equivalently by the two plots, the magnitude spectrum and the phase spectrum (Figure 8-15). Notice that the magnitude spectrum is symmetric with respect to N/2, while the phase spectrum is antisymmetric as long as x(n) is a real signal.

Figure 8-15 Time series and its DFT (magnitude and phase)

8.6.3 The Discrete Fourier Series

One of the relevant aspects of Fourier or harmonic analysis is that there are analytic expressions to compute X(k) from x(n), and vice versa. These are called the discrete Fourier series (DFS) pair:

(8.17)

We already saw these expressions in Chapter 5 when we discussed function approximation without knowing all these implications. There we computed the Fourier coefficients X(k) as the inner product of the signal with the bases. Let us interpret the meaning of the spectrum X(k) by looking at the second equation in the pair. Each basis is a phasor (which really corresponds to a sinusoid) with different frequencies n [image] [image], multiples of the smallest frequency [image] [image] ( [image] = 2 [image] /NT) that can be represented in the time window. According to Eq.8.17 we get back x(n) by adding all the phasors multiplied by the projections in each phasor, which are the Fourier coefficients X(k).

Therefore the spectrum is telling us how much of the energy of the windowed input signal x(n) exists at each phasor frequency. For instance, a sine wave of normalized frequency k1 (k1 corresponds to the frequency w = 2 [image] k1/NT) has a spectrum with one nonzero coefficient X(k1) exactly at the frequency k1 of the sine wave (with a value equal to the amplitude of the sine wave). If the signal is composed of two sine waves, the spectrum contains two lines at the respective sine wave frequencies. We can see that, at least for sinusoidal signals, the DFS gives a much more compact representation than the time representation (the signal amplitude changes according to a complicated relation over the period). However, a square wave that is such an easy signal to describe in time (only two amplitudes) requires an infinite number of amplitudes when decomposed in phasors. properties of DFS

Some important remarks are in order. Notice that the DFS is periodic with N (the number of samples in the time window). Its magnitude is symmetric with respect to sample N/2, while the phase spectrum is antisymmetric. The value k = N/2 corresponds to the highest frequency that can be represented with the chosen sampling frequency (1/2fs). Values of k above N/2 correspond to the negative frequencies and can be ignored from the interpretation of real signals (since the magnitude spectrum is symmetric). The separation between consecutive values of k is called the frequency resolution, and is given by

(8.18)

in Hertz (or 2 [image] /NT in radians per second). We can see that the higher the number of points used in the time signal (longer windows), the better is the spectral resolution.

Computers can easily compute the DFS through a fast algorithm called the fast Fourier transform (FFT). how to use the FFT

NEUROSOLUTIONS EXAMPLE 8.12

The Fast Fourier Transform (FFT)

NeuroSolutions has a probe that computes and displays the magnitude of the FFT of a segment of data. We choose the variable we want to graph, place a data buffer on the access point, and place the FFT Probe on top of it. In the Inspector you must select the size of the window (if smaller than the data buffer), the size of the FFT (which must be a power of 2, i.e. 32, 64, 128, 256, etc), the number of independent FFTs that will be averaged together to decrease the variance of the estimation, and finally the percentage overlap between the successive windows. This probe is more than a simple FFT: it is a consistent spectrum estimator (called the Welch spectrum estimator). But if we select 1 as the number of FFTs, this will give us just the spectrum. The probe displays only the spectral magnitude.

If we input a sine wave (30 samples/period) and compute the 64 point FFT it will show two lines symmetric with respect to the midpoint of the display. Note that the horizontal axis of the FFT display is frequency, in a linear scale, from 0 to fs. The middle of the display is fs/2. The right half of the display corresponds to the negative frequencies and should be the same as the left side (for real functions). This is the reason we sometimes just display the first half of the spectrum. The left peak of the spectrum should correspond to the frequency of the input sine wave (the right peak is just its negative frequency counterpart). If you create a 2- samples per cycle sinewave (the highest possible frequency) you should see just one spectral line at fs/2.

Input 2 sine waves of different frequencies (6 and 30 samples/period). The display shows four lines representing 2 frequencies. Notice that it is much simpler to interpret the FFT plot than the time signal. Let us now select two frequencies that have contiguous periods (16 and 17 samples/period). The 64 point FFT will display just two peaks instead of 4, because it does not have enough resolution to discriminate the two peaks (use Eq. 8.17). You have to increase the FFT length to 512 to be able to resolve the two peaks (use Eq. 8.17 again).

NeuroSolutions Example

There is a one-to-one relation between signals in time and their spectra; that is, representations of signals in one domain or in the other are perfectly equivalent. A nice property of the signal spectrum is that the signal components along each one of the bases are orthogonal to each other, while in the time domain the signal components along each axis are not orthogonal (when the bases are the delays). For many problems we do not even need to compute the phase spectrum: we simply work with the magnitude. The importance of spectral analysis can be better appreciated if we blend it with the analysis of linear systems. relationship between DFS, FT, DFT, LT and Z

Go to next section


8.7 The Z-Transform and the System Transfer Function

 8.7 The Z Transform and the System Transfer Function

We just presented the basic concepts of spectral analysis applied to signals. Can we apply the same principles to systems, and are there any advantages? A system is described by its impulse response, which can also be interpreted as a signal, so we can apply the ideas of the transform domain to the impulse response, but this route is neither elegant nor efficient (it requires a priori knowledge of the impulse response).

In linear system theory the important aspect is to quantify the output signal given the input and a description of the system. The linear system output can be computed by either:

  • The convolution of the impulse response with the input, or

  • Generating the output sample by sample by means of the difference equation

The difference equation (Eq. 8.9 or Eq. 8.13) immediately gives an algorithm to compute the system output, but it is not very good at quantifying what is happening to the input signal characteristics, due to the fact that both the system and the input are mixed in the same equation. It would be wonderful if we could enhance our understanding of the difference equation by decoupling the system function from the input. This is exactly what the Z transform does.

The Z transform simplifies the study of discrete time linear systems because it transforms difference equations into algebraic ones, providing a theoretical framework for working with delay operators. The definition of the Z transform of a discrete sequence x(n) is

(8.19)

where z is a complex variable. To exemplify the power of the method, let us take the Z transform of the one sample delay [image] (n-1):

(8.20)

Eq. 8.20 states that the Z transform of the delay operator is the infinite sum of terms of the time function multiplied by z¯ [image]. It turns out that [image] (n-1) has a nonzero value of 1 only when n = 1, so all the terms of the sum are zero except the term n = 1, which is evaluated as z [image]. The delay operator we defined previously as the fundamental building block for the memory element is the Z transform of the one-sample delay. Now we understand why we used the label z [image] for the block of Figure 8-3.

If we apply the definition of the Z transform (Eq. 8.19) to the linear combiner (Eq. 8.9), we get

(8.21)

where Y(z) and X(z) are the Z transforms of the output y(n) and of the input x(n), respectively. This expression clearly shows that the effect of the linear combiner on the input x(n) can be individually studied in the Z domain by the properties of the polynomial in z of order N (a purely algebraic form).

(8.22)

The quantity H(z) is called the system transfer function, and it is the Z transform of the impulse response (Eq. 8.22 shows this for the FIR case). Note that the Z transform also naturally decoupled the effect of the system from its input, since the Z transform of the output is a product of the two (Eq.8.21).

Similar to the impulse response, the system transfer function completely describes the behavior of the linear system, but with some advantages. Notice that Eq. 8.21 computes the Z transform of the output of the linear combiner, but now we obtain the Z transform of the output by a point-by-point multiplication of the Z transform of the input with the system transfer function. Taking the inverse Z-transform we get back y(n).

A comparison of Eq. 8.21 with Eq. 8.12 shows that convolution in the time domain was mapped to multiplication in the Z domain. This is one of the reasons that the transform domain representation of linear systems is so widely used. The transfer function was derived for the linear combiner but can be extended to any linear system. types of discrete time linear systems

Go to next section


 8.8 The Frequency Response

We already saw in Chapter 5 that a linear system accepts the complex exponentials as eigenfunctions. The implications of this fact are far reaching, because we can construct the output of ANY linear system H by independently computing the system response to each complex exponential of the input and summing the result. Here we are interested in making this decomposition for phasors, which are complex exponentials with unit module (a = 0). As we saw in the box "Eigendecomposition Explained"

(8.23)

where the [image] i (complex numbers) represent the effect of the linear system at each frequency. We call the system effect at each frequency [image], the system frequency response. The system frequency response is complex, so it is described by its magnitude frequency response and the phase response.

Equation 8.23 means that ANY linear system H affects the phase and the amplitude of each frequency of the input signal and that the effect can be computed independently at each frequency. This, if the input is expanded in phasors (the DFS) a simple frequency-by-frequency multiplication by the frequency response will provide the DFS of the output:

(8.24)

This is a very important relation, because it lets us predict what is going to happen to our input signal when it goes through a linear system, provided that we know [image] and the spectrum of our input signal beforehand. circular convolution

Notice how clear Eq. 8.24 is for the analysis of the effect of the linear system (just a point-by-point multiplication in the magnitude and an addition in the phase responses). Alternatively, it allows us to specify through design the frequency response of the system to achieve our processing goal. To obtain the steady-state time response y(n) we just need to take the inverse DFS (Eq.8.17). transfer function versus frequency response

NEUROSOLUTIONS EXAMPLE 8.13

Analysis of the Output in the Frequency Domain

Let us experimentally verify Eq. 8.24 using the filters we worked with previously, the linear combiner and the first-order recurrent system. We input a square wave and compare the spectrum of the output with the spectrum of the input and the frequency response of the system. We should be able to see that the spectrum of the output is a sample-by-sample multiplication of the spectrum of the input with the system frequency response.

Since the spectrum of the impulse is flat, we should be able to see why the impulse response is so important: We only need to compute the spectrum of the output to obtain the system frequency response.

NeuroSolutions Example

The big advantage of the frequency response is that it can be computed with the FFT, while the system transfer function requires the computation of the Z transform, which is much more difficult. But remember that the frequency response provides only partial information about the system in the form of its steady-state behavior (e.g., no information about the transient response is obtained from the frequency response). In a lot of applications, the steady-state response is enough for the design.

NEUROSOLUTIONS EXAMPLE 8.14

Minimum Phase Systems

Although the frequency response is very important, we should realize that it provides only partial information about the system response. For instance, the frequency response is insensitive to a change in the phase of the system response. We show this by moving the zero of a linear combiner from inside the unit circle to outside, but in such a way as to preserve the distance to the unit circle (which is called the mirror location). We see that the frequency response does not change, but the impulse response is different. One system is called minimum phase (the one with the zero inside the unit circle), and the corresponding filter with the zero outside (in the mirror location) is called the maximum phase.

NeuroSolutions Example

Go to next section


 8.9 Frequency Response and Poles and Zeros

We saw the intrinsic relationship between frequency response and the system transfer function. One aspect that engineers use frequently in practice is to sketch the frequency response from the knowledge of the poles and zeros of the transfer function. This is important because from the difference equation one can easily obtain the system transfer function and from there the location of the poles and zeros. For instance, from Eq. 8.13 we obtain

which immediately tell us that the transfer function has a zero at z = 0 and a pole at z = (1- [image]) (remember that the pole is the zero of the denominator polynomial). With this knowledge can we sketch approximately the frequency response without taking the FFT of h(n)?

The shape of the frequency response [image] is uniquely specified by the location of the poles and zeros of the transfer function H(z). When the frequency [image] varies from 0 to 2 [image] the analog frequencies are scanned from 0 to fs (Figure 8-16): that is, we have all the knowledge of how the system responds to each frequency. scales in the unit circle

Figure 8-16 The Z plane and the unit circle

The frequency response can be graphically obtained by considering the response of the system as the height of a "tent" that is placed around the unit circle. A system pole is like a stick that supports the tent from the inside, while a zero is like a stake for the tent. The height of the stick is the inverse of the difference from the pole to the unit circle, while the length of the stake "rope" is proportional to the distance from the zero to the unit circle.

Depending on the location of the poles and the zeros different "tent" configurations are obtained. The height of the tent at a given frequency location is the amplitude of the system frequency response; a high value at a given frequency means that this frequency will appear enhanced at the output, while a value close to zero means that the frequency will be attenuated. second order systems

In DSP language, the singularities of the transfer function (poles and zeros) sculpt the system frequency response. When H(z) has a zero at a certain angular frequency, the frequency response [image] has a valley (attenuation) at that frequency, since the numerator of the frequency response [image] is small. On the other hand, when H(z) has a pole at a given radial frequency, the frequency response [image] has a peak (gain) around that frequency (Figure 8-17), since its denominator has a small value, which corresponds to a large value of the inverse.

Figure 8-17 Relation between the pole zero plot and the frequency response

The closer the singularities (poles or zeros) are to the unit circle, the more noticeable are their effects (sharp peaks and narrow troughs, respectively) on the frequency response. (dominant pole approximation) The relationship between singularities and their effect on the frequency response is very important, because after knowing the pole-zero plot, we can immediately qualitatively predict the modifications that will occur to the input signal after processing, as we discuss next.

NEUROSOLUTIONS EXAMPLE 8.15

Frequency Response of the Second-Order System

This example is aimed at enhancing our understanding of poles and zeros and their effect on the frequency response. We use for this illustration the second-order resonator and the second-order anti-resonator. The former as you recall, has a pair of poles, and the latter has a pair of zeros. We can then expect that the resonator will control the peaks of the frequency response, while the anti-resonator will control the placement of the zeros. It is important that you understand the difference between these two types of effects to the frequency response.

We start with the anti-resonator. A simple 3-tap FIR will do the job and implements the simple equation

y(n) = x(n) + ax(n-1) + bx(n-2)

which has roots (zeros of the transfer function) in the Z domain at

When the zeros are complex (conjugate), it is preferable to think that b = r², and a=2rcos [image].

Change the values of a and b and see the effect on the transfer function. Note in particular that when the zero is at the unit circle, the response at that frequency is exactly zero. Keep the angular direction the same and change the modulus of the zero to observe the trough in the frequency response become more and more pronounced. Finally, place zeros in the positive real axis and also in the negative real axis to see the shape of the frequency response.

NeuroSolutions Example

Next let us work with the resonator with the same parameters a and b. A simple 2-delay system with feedback is sufficient to implement the resonator, with equation

y(n) = -ay(n-1) - by(n-2) + x(n)

which has roots (poles of the transfer function) in the Z domain at

When the poles are complex (conjugate) it is preferable to consider that [image] b = r², and -a=2rcos

Change the values of a and b and see the effect on the transfer function. In this case you have to be much more careful, since for many values of a and b the system will be unstable, so you will not be able to compute the FFT. But you should also find out the effect of the poles in controlling the height and location of the peaks of the frequency response. You should use the approximate formulas given in Appendix A on resonators to estimate the magnitude of the response and see whether it matches the NeuroSolutions result.

The effects of poles and zeros are ADDITIVE in the frequency response, so just by studying these two simple cases, you can generate ANY frequency response shape by implementing the required number of poles and zeros. (There are different ways of doing this, but probably the easiest here is to cascade the resonators and the anti-resonators or both. You can do this in NeuroSolutions.)

NeuroSolutions Example

Go to next section


 9.1 Introduction

In this chapter we study a different way of designing filters. Instead of synthesizing filters according to user specifications, as we did in Chapter 8, we assume that the desired response is available. Hence we can use the ideas of adaptation to automatically produce the required functions without ever synthesizing the filter transfer function. For obvious reasons this methodology is called adaptive filtering.

An adaptive filter is a special type of neural network that is linear, which means that it can be studied in linear vector spaces. Our present knowledge of linear adaptive systems can be framed in relatively simple mathematics, when compared with that required for general (nonlinear) neural networks. Excellent textbooks on adaptive filters are (Widrow) and (Haykin). We will see that adaptive filters are nothing but regressors in the signal space. The output of each tap of the delay line is considered a component of the input signal, and the goal is to regress it onto the desired signal. Normally, we do not use a bias in linear filters to preserve the properties of linearity, as discussed in Chapter 8.

We also extend to time the theory of Hebbian networks covered in Chapter 6. This is hardly ever done, but we will see that very interesting functions can be created with temporal PCA networks.

Although much of the theory in this chapter is a restatement of the theory from Chapter 1, this chapter shows how adaptive filters (the linear combiner and temporal PCA) can be applied to various signal-processing functions. Applications that we cover are prediction, model based spectral analysis, system identification, inverse modeling, interference cancellation, echo cancellation, adaptive inverse control, maximum eigenfilters, PCA in time, spectral analysis by eigendecomposition, noise cancellation, and bind source separation. As you can see, adaptive filters can be applied to solve may different problems using adaptive concepts.

Go to next section


 9.2 The Adaptive Linear Combiner and Linear Regression

The linear combiner was discussed in Section 8.3. It is built from a delay line of D-1 delays (D taps) and an adder, represented by the equation

(9.1)

The impulse response of this system is specified by the weights wi, and its region of support is D samples long. Hence, this system has a finite impulse response (FIR). The output of the linear combiner is a linear combination of the present and past D-1 inputs. Depending on the filter weights, we can create any of the filter types discussed in Chapter 8 (lowpass, highpass, bandpass, or bandstop). If the filter order is large enough, any transfer function can be constructed with the linear combiner. Filter design by specification is applicable when the signal and the noise are very well characterized and the signal and noise spectra do not overlap. In such cases the design procedure is very well suited to solve the filtering problem.

Unfortunately, not all problems of interest fall in this category. Sometimes the signal and the noise spectra overlap, or the signal has characteristics that change slowly over time. In such cases predefining the filter coefficients is not appropriate. More generally, filtering involves removing or accentuating certain frequencies but we may have different goals such as signal modeling, prediction, and so on. This is the reason alternatives to designing filters by specification have been developed. Here we study methods to adaptively determine the filter coefficients from the data.

The adaptive linear combiner is a linear system with adaptive weights. Instead of fixing its weights according to some design criterion (i.e., filter synthesis) we are back to the framework of adaptive theory. A desired response is presented, and the filter weights are modified so that a minimum of a cost function is achieved (Figure 9-1).

Figure 9-1 Paradigm to adapt the weights of the linear combiner

This block diagram is identical to the one presented for regression or classification, but it is more similar to regression than classification because normally the desired response is also a discrete signal with many amplitude values instead of {1, -1 (or 0)} as in classification. We can therefore interpret the function of the adaptive linear combiner as a linear regressor (without bias) from the input time series to the desired time series. Note that the order of the regressor is established by the number of taps in the delay line. In spite of the similarity with regression, which warrants a review of Chapter 1, we will formulate the problem of finding the optimal filter coefficients as done in adaptive filter theory to enhance our understanding and specifically address time series analysis.

Go to next section


 9.3 Optimum Filter Weights

The pioneering work in optimal filtering theory was done concurrently by Wiener and Kolmogorov. They solved the following problem in continuous time (i.e., in infinite vector spaces called Hilbert spaces): Given a signal x(t) contaminated by noise n(t), find the best linear system that is able to approximate another signal d(t). d(t) could be any signal, including the signal x(t) advanced by [image] seconds. In this most general case, the linear system would work as a predictor and filter. Here we cover only the discrete time case (finite vector spaces).

The problem is set up exactly like the regression problem. Let us define the error signal as the difference between the desired response d(n) and the system output y(n):

(9.2)

when the input to the linear combiner with D weights is the signal x(n). We assume that all these signals are statistically stationary, that is, they are random vectors, but their statistical properties do not change over time. We define the mean square error (MSE) as the expected value of the error square,

(9.3)

This means that the optimization criterion is the power of the error, or the L2 norm of the error. We want to choose the filter weights to minimize J. To find the optimal weight values, we derive the cost with respect to the unknowns (the filter weights) and set it to zero:

(9.4)

Substituting Eq. 9.2 in Eq. 9.3 with y given by Eq.9.1, we obtain a set of D equations in D unknowns called the Wiener-Hopf equations

(9.5)

The solution, using vector notation, is the well known equation

(9.6)

which has the same form as the solution of the regression problem. The structure of the solution is the same; however, R and p have a slightly different meaning here when compared with the regression case. They represent the time autocorrelation of the input and time cross-correlation functions between the input and the desired response.

We define the autocorrelation function in time as

(9.7)

and the cross-correlation function as

(9.8)

These definitions use the statistical operator E[.]. In practice, the statistical operator is substituted by the temporal operator A[.].

(9.9)

If ergodicity holds, the autocorrelation and cross-correlation functions computed with temporal operators approach their statistical counterparts. Commonly, the limiting operation is also dropped, which means that the estimation is done within a window of M samples. The autocorrelation matrix becomes

(9.10)

The autocorrelation function for lags k1, k2 is obtained by cross multiplying the time series x(n-k1) with a shifted version of itself x(n-k2)

(9.11a)

This is repeated for each lag up to D-1 to obtain all the entries of Eq. 9.10. The subindex n means that the computation is done for the time series around sample n. The cross-correlation vector p is obtained in basically the same way, but now by cross multiplying the desired signal with shifted versions of the input:

(9.11b)

M should be at least 10 times larger than the lag D of the autocorrelation function to obtain a reasonable estimate for the longest lag.

We can show (Widrow and Stearns) that with the optimal weights given by Eq. 9.6, the error [image] (n) becomes decorrelated with the system output, hence it is also decorrelated with respect to the inputs, i.e.

(9.12)

This gives a very powerful interpretation of the Wiener solution: The Wiener filter finds the weights such that the system output (which has to exist in the space spanned by the input, since it is an inner product with x(n-i)) is the orthogonal projection of the desired signal onto the input space (Figure 9-2).

Figure 9-2 The Wiener solution in vector spaces

The orthogonal projection minimizes the error (the distance) between the desired response and the system output. Kolmogorov developed Eq. 9.12 independently to solve the minimization of MSE and its geometric interpretation.

Instead of analytically computing the optimal solution, we can search for the optimal set of weights, as discussed in Chapter 1. It turns out that the cost in the space of the weights is a paraboloid of equation

(9.13)

where R and p are the estimates of the autocorrelation and cross-correlation function respectively, w is the weight vector, and d(n) is the desired time signal. An efficient search procedure to find the single minimum cost is to use gradient information. Under this paradigm the weights are modified proportionally to the negative of the gradient estimated at each point

(9.14)

where [image] is a step size parameter and n is the iteration index. The most useful estimator of the gradient was proposed by Widrow. Instead of estimating the gradient through a statistical average, Widrow proposed the direct use of the instantaneous value of the gradient at each sample, that is,

(9.15)

This gives rise to the famous LMS algorithm to adapt the weights:

(9.16)

where w(n+1) is the vector of weights at iteration n+1, [image] n is the error at iteration n, and xn is the input vector at iteration n. There are better algorithms to adapt the linear combiner but the LMS is still widely used due to its excellent compromise of simplicity and robustness versus computational complexity. RLS algorithm

NEUROSOLUTIONS EXAMPLE 9.1

Design of Filters Using Adaptation

In this example we explore the power of adaptive filters. A very good example is to create filters that pick the harmonics of a square wave. As we know, the square wave has a Fourier spectrum characterized by spectral lines at the odd harmonics of the period of the wave. We can see this easily by placing an FFT probe on top of the Axon being fed with a square wave.

The goal of the adaptive filter is to individually pick the harmonics. The desired response is a sine wave of the particular harmonic. The adaptive filter is in fact implementing a bandpass function, but instead of using one of the filter synthesis programs to come up with the coefficients, we use the desired response and the LMS algorithm to automatically design the filter for us. This is a good alternative in some cases.

Change the frequency of the desired signal to see how fast the weights adapt to the change. We can stop the simulation at any time and freeze the weights to compute the frequency response.

With adaptive filters, we are not confined to creating the conventional filters we have studied in Chapter 8. We can smoothly design the frequency response. For instance, we can transform a square wave to a triangular wave of the same period. Find the frequency response of such a system using NeuroSolutions.

NeuroSolutions Example

9.3.1 Wiener Filter as a Predictor

One interesting application of the Wiener filter is when the desired response d(n) is the input advanced in time, that is, x(n+n0). This special case is something magical, because its goal is to "predict the future" of a time series. But notice that the filter is simply computing the best approximation for a future value of the time series, given the past samples. Since time signals have a time structure, that is, they do not change radically from sample to sample, it is possible to locally extrapolate what the next sample is from the past N samples. Wiener was able to formulate prediction mathematically, which was a landmark result.

NEUROSOLUTIONS EXAMPLE 9.2

Adaptive Filters for Prediction

Prediction is just a special type of filtering. In fact, prediction implements a type of lowpass filter that depends on the short-term time structure of the time series. The desired response in prediction is the signal advanced by a number of samples (let us start with one). The predictor has to look into the past of the signal and linearly combine the samples to obtain a good approximation to the position of the next sample. In a sense, we are linearizing the trajectory around the present sample (the tangent space) and extrapolating to find the next value.

Let us use a smooth signal first (the sum of two sine waves). If you let the system run, you quickly find that the error becomes very small if the FIR has at least 8 taps with a delay of 2 between taps. Alternatively, you will need 16 taps and a delay of 1 to get comparable results, but notice that the system has twice as many coefficients. Notice that the frequency response is in fact a lowpass type of filter. Prediction is in fact a kind of extrapolation.

If you now use a square wave with period 32, you find out that the system does not work well at all. Can you discover why? The square wave is discontinuous, so there is no way the predictor can anticipate the abrupt change in amplitude by just looking at 8 samples of the recent past of the signal. To predict the waveform, a full cycle must be present in the delay line (at least 32 taps with a unity delay). With this choice, the predictor works very well. Note once again that it is sufficient to increase the delay between taps to 4, and keep the number of coefficients at 8 (there is no need to have 32 weights….).

Another case where the predictor performs poorly is when there is white noise in the data. Since the samples of white noise are uncorrelated from sample to sample, there is no information in the past of the signal to help predict the future values. In general signals that have a very complex time structure are difficult to predict with small models. We can get an idea of the predictability of the signal if we visualize it in state space and find abrupt jumps in the plot.

You are now ready to bring in your pet data set to predict it. (We do not take responsibility if you try to predict the stock market).

NeuroSolutions Example

9.3.2 The Wiener Filter as a Function Approximator

Chapter 5 was devoted to presenting a unified view of regression and classification as special cases of a broader problem called function approximation. But remember that, at the time, only static problems were being addressed. Now we know how to handle time functions with filters, and in particular we know how to optimally set the linear combiner's parameters to approximate the desired time series. Can we extend the views of function approximation to the Wiener filter?

The answer is affirmative, and it is rather important for framing the application domain of linear combiners. If we compare Figure 9-1 with Figure 5-1, we can see the similarity immediately. Here we assume that the desired signal d(n) is a fixed but unknown function of the input x(n), that is, d(n) = f(x(n)). The role of the Wiener filter is precisely to discover a linear approximation to the unknown function f(.).

The noticeable difference here is the use of time functions, while in Chapter 5 the data was multidimensional, but static. However, in Section 8.2 we provided a mapping from time to a vector space called the signal space, which essentially showed that a segment of a time series can be always mapped to a point of a vector space of appropriate dimensionality. The difference between time functions and static data therefore washes away if we consider that the approximation is being performed in the signal space. Another minor difference is that we do not use a bias in the linear combiner (to comply with the definition of a linear system).

We can then conclude that the linear combiner is doing regression in the signal space. Looking at the Wiener filter operation from the point of view of function approximation, we see that it is a linear function approximator. The bases of the signal space are the input signal and its delayed versions (i.e., they are time varying), so the linear combiner is finding the best projection of the desired signal onto the input signal space. The best projection must lie in the input signal space since it is a linear combination of the bases.

This function approximation property can be put to good use in engineering. We already saw the application of Wiener filters to prediction. In section 9.6 we discuss system identification and noise cancellation. system identification fundamentals

Go to next section


9.4 Properties of the Adaptive Solution

 9.4 Properties of the Iterative Solution

The properties of the iterative solution to find the minimum of J have been extensively studied in Chapter 1, and they also apply to the adaptation of the linear combiner. The fundamental difference is the nature of the autocorrelation and cross-correlation functions which now are related to time information. Here we summarize only the most important ideas.

9.4.1 Largest Step Size

The largest step size [image] max to guarantee convergence to the optimum value is

(9.17)

where [image] max is the largest eigenvalue of the input time autocorrelation function R. In practice we choose

(9.18)

where [image] 0 has to be set according to the problem (0.01~0.1) and the trace of R can be estimated by the signal power at the taps

The trace of R is the sum of the eigenvalues, so tr[R] is an upper bound for [image] max. Notice that if the input power changes, the step size changes automatically. This is called the normalized step size and is the preferred way to adapt the linear combiner.

NEUROSOLUTIONS EXAMPLE 9.3

Choosing the Step Size

Please check the convergence/divergence of the linear combiner as a function of the step size. We provide here the breadboard for the linear combiner for your convenience, but since Examples 9.3 through 9.5 are duplications of the ones in Chapter 1, we will not elaborate the details.

NeuroSolutions Example

9.4.2 Speed of Adaptation

In steepest descent, each weight converges to the minimum with a different time constant given by the inverse of the respective eigenvalue. The mode with the longest time constant is

(9.19)

The search time is limited by the adaptation of the slowest mode. The existence of many time constants is readily apparent in the learning curve. This means that the largest step size is limited by the largest eigenvalue (Eq.9.17), and the speed of adaptation is limited by the smallest eigenvalue (Eq. 9.19). Hence the eigenvalue spread of the input autocorrelation function defines the performance of the steepest-descent algorithms (LMS in particular).

For time signals the speed of adaptation is coupled with the actual time it takes to adapt the linear combiner, because when we multiply the number of samples by the sampling period, we obtain time in seconds. A good estimate is 4 [image] max.

NEUROSOLUTIONS EXAMPLE 9.4

Speed of Convergence

Please check the speed of convergence of the linear combiner as a function of the step size.

NeuroSolutions Example

9.4.3 Rattling

The choice of the step size in LMS is a compromise between the speed of adaptation and misadjustment. The faster the algorithm converges, the higher is the misadjustment, that is, the excess error with respect to the theoretical minimum error. The misadjustment can be approximated by

(9.20)

If we assume that the eigenvalue spread of R is small, we can say that the algorithm converges in

(9.21)

This gives the following rule of thumb: for a misadjustment of 10 percent, the algorithm converges in a number of samples equal to approximately 10 times the number of weights. We can therefore think of translating speed of adaptation to actual external time (in seconds) by multiplying the number of samples by the sampling period. In many of applications, the adaptation time of the linear combiner is reasonable, yielding systems that can track changes in the signal dynamics.

NEUROSOLUTIONS EXAMPLES 9.5

Misadjustment

Please check the rattling of the linear combiner as a function of the step size.

NeuroSolutions Example

9.4.4 Neural Versus Adaptive Systems

There is a major difference between the neural network models covered so far and the adaptation of the linear combiner. In ANNs we always work with a training set and test the performance on an independent test set. We may have to reuse the training set samples many times until the ANN converges. But notice that once the ANN is trained, its weights are fixed for the test set. Thus, although the ANN has trainable weights, they are frozen during the testing. Therefore, the ANN is no longer adaptable after training.

Adaptive filters are utilized in a different way. They are continuously adapted, that is, the weights are always being modified according to the LMS rule. There is no training set -test set division when working with adaptive filters. When the data is first sent through an adaptive filter, the performance of the system is far from optimal, but the system parameters quickly find reasonable values (convergence in about 10D samples). The essence of this feature is the use of the input signal and its delayed versions as the bases for the approximation.

An advantage of this procedure is that the system is able to track changes in the structure of the time series, which does not occur with the training procedure applied to ANNs. However, the system forgets old data and can be driven from its optimal parameter setting if outliers (noise spikes) occur. It will recover rather quickly, however, due to the good tracking properties of LMS in a linear topology.

There is clearly a trade-off between trainable versus adaptive weights, or between neural and adaptive systems. A neural system seeks to model the data as long-term memory in its weights, while the linear combiner is basically a tracker, that is, it has only short-term memory. For general use, an intelligent system needs both. This is not easy to do. Within the linear model, the Kalman filter solves this problem by using the state information. The state is estimated from the data, and it corresponds to the long-term memory of the system. When the system cannot predict the input, it decides to update the output for tracking. Unfortunately, we still do not have a general framework to accomplish a similar decomposition for the nonlinear model.

NEUROSOLUTIONS EXAMPLE 9.6

Tracking Ability

This example shows the tracking ability of the linear combiner. Tracking is a new concept that makes sense only in time series. The goal is to be able to change the filter weights so that a change in the desired response or the relationship input to desired signal is followed. This is a very important characteristics in real-world problems where the data may slowly change (and sometimes not that slowly).

Tracking is intrinsically a compromise. On one hand, we would like the linear combiner to reach the bottom of the bowl with low misadjustement, which means a small step size. On the other hand, if the location of the minimum changes with time, we would like to have a sufficiently large step size to follow it to the new location. The trade-off can only be solved experimentally.

NeuroSolutions Example

Go to next section


9.5. Hebbian Networks for Time Processing

 9.5 Hebbian Networks for Time Processing

The Hebbian networks discussed in Chapter 6 can be extended to time processing if a delay line is incorporated at their input (Figure 9-3).

Figure 9-3 Hebbian PE extended to time

The input signal x(n) flows into the delay line, and the Hebbian PE receives not only the present input but also D-1 past samples. As in its static counterpart, the PE estimates the eigenvectors and eigenvalues of the autocorrelation of the input, but now the input is a signal in time. Karhunen and Loeve did the original work in this area, and temporal PCA is called the Karhunen Loeve transform (KLT) in their honor. The mathematics are very difficult for continuous time signals (the original formulation), but we already know all the mathematics needed to treat the case of discrete signals of finite length.

We will call the discrete version of the KLT temporal PCA. The solution of temporal PCA is also an eigendecomposition given by

(9.22)

or in matrix form

(9.23)

where [image] i and ei are the eigenvalues and eigenvectors, respectively, of the autocorrelation matrix, R given by Eq. 9.10.

The temporal PCA equations proceed analogously to the static PCA case. Since the eigenvectors are orthonormal,

(9.24)

they form a complete basis in the D dimensional space. Thus the vector x(n) can be synthesized without error by an expansion over the D eigenvectors:

9.25

with E = [e0, e1,…,eD [image] 1]. The matrix E is deterministic and full rank. The coefficients ui(n) of the expansion can be found from

9.26

The ui(n) are called the principal components. We have seen in Chapter 6 that Sanger's rule is an on-line algorithm to compute this decomposition, so we can adapt the parameters of the network of Figure 9-3 with Sanger's rule to compute the temporal PCA. In this network the eigenvectors e i are the weight vectors w i. The power of the outputs are the principal components ui(n). We can recover the input x(n) from the weight vectors if we use Eq. 9.25.

9.5.1 Properties of Temporal PCA

When viewed as a set of eigenfilters, the eigenvectors of the autocorrelation matrix have very interesting properties. In the limit of long windows, the KLT expansion reduces to a Fourier basis, and the eigenvalues become the power spectrum (see Van Trees). For a finite number of delays, the relationship between eigenvalues and the power spectrum is not exact but is a good starting point. Temporal PCA becomes a set of FIR bandpass filters with center frequencies attracted to the peaks of the spectrum.

Less well known is the relationship between the ordered eigenvalues and spectral peaks. Assume that the eigenvalues of R are ordered from largest to smallest, [image] 0 = [image] 1 =.…= [image] D-1 as the number of delays approaches infinity, the maximum eigenvalue of R approaches the maximum value of the power spectrum of x(n), and the power spectrum of the corresponding maximum eigenfilter becomes an impulse centered at the corresponding maximum frequency of the power spectrum, that is,

where X(w) is the power spectrum of x(n). In fact, the second eigenvalue and eigenvector also exhibit the same properties. This can best be explained in terms of the orthogonality of sine and cosine waves. For any pure sinusoidal basis, a second orthogonal basis of the same frequency can be found through a 90-degree phase shift of the original basis. For finite but sufficiently large D, the first two eigenvectors act as bandpass filters around the peak in the signal spectrum. The passband becomes increasingly narrow as the number of taps is increased. Generalizing for the other components, we can say that eigenfilters can be approximately regarded as filters matched to peaks in the signal spectrum.

A multiple-output Hebbian network trained with Sanger's rule computes the PCA components on-line, on a sample-by-sample basis. If instead of Hebbian learning we use the anti-Hebbian learning, then a new set of functions (novelty filters) are possible as we briefly mentioned in Chapter 6.

Go to next section


 9.6 Applications of the Adaptive Linear Combiner

It is very instructive to address the applications of the adaptive linear combiner. Here we present simulations that use the adaptive linear combiner for prediction (time series identification), interference canceling, echo-cancellation, line enhancement, system identification, and adaptive controls. How can the same system be used for so many applications? We will see that the principle involved - exploiting correlation between the desired response and the filter output- is always the same, but the adaptive filter is used in different arrangements, producing very different end results.

9.6.1 Prediction

In prediction the desired signal is the input advanced by no samples. Normally no = 1, which is called single-step prediction. We study single step prediction here, but multi-step prediction (no > 1) is the same, except that the task becomes increasingly more difficult for larger no.

Prediction is widely applied in forecasting and modeling. Forecasting is more easily understood, and it is the task we explain first. In forecasting problems, we know a time series up to the present time n. We wish to know what the next sample, x(n+1), of the time series is going to be. The idea is that if we can train a system to predict the former values of the time series (i.e., if the time series for sample k < n is fed to the system and the request is to predict sample k + 1), then the system will also accurately predict sample n + 1.

It is obvious that we have to assume that the structure of the time series does not change over time (i.e,. the signal is stationary), otherwise, the goal is illusory. In many of applications this model is reasonable, and even if the signal time structure changes, we assume that it changes slowly, so that the system can continuously track the changes.

Figure 9-4 shows the block diagram to implement single-step prediction. The input signal is delayed by one sample before it is fed to the linear combiner. The desired signal is the current value of the input x(n). The input of the linear combiner is therefore delayed one sample with respect to the desired response.

Figure 9-4 Block diagram to implement single-step prediction

NEUROSOLUTIONS EXAMPLE 9.7

Prediction of Time Series

This breadboard shows the prediction of a complicated time series, the sunspot data. Sunspot activity is an example of a phenomenon we cannot control but that affects us greatly (loss of radio communications, weather, etc.), so the only option is to look ahead and see what is going to happen. The sunspot data is very complex (possibly chaotic), but short-term prediction with the linear combiner is possible.

Let us select the size of the tap delay line as 16. This is the variable that controls the size of the model, so it has to be experimentally determined. As we saw, larger models normally work better, but this depends on the type of data. Here, for instance, larger models do not improve performance because the data is chaotic. Larger models simple utilize more information from the past to establish the value of the next sample. But if the time series changes rapidly and it is aperiodic, this is of no help (remember the square wave?). You should experiment with this variable in the breadboard.

The accuracy of the prediction is not uniform along the training set. To see this, modify the learning from batch (which uses the full data) to custom with 500 samples per update in the Back Controller. You also have to modify the reporting on the L2 Criterion from batch to weight update. The accuracy drops rapidly for multistep prediction. You can create multistep prediction by going to the desired response file and customizing it (create a segment starting at samples 2, 3, etc.) . Watch the MSE for the different cases.

NeuroSolutions Example

The prediction of the future of the time series seems an impossible problem due to the causality inherent in any physical system. However, if we look at it from the point of view of signal space, prediction is really a simple problem of locally fitting the best hyperplane to the signal trajectory. The source of the error can be due to the fact that the hyperplane is not a good fit to the trajectory, or that the characteristics of the signal have changed from the present to the future.

9.6.2 Prediction for Modeling

We can interpret prediction as the identification of the linear system that produced the time series x(n). Let us analyze the block diagram of Figure 9-5. Here we assume that there exists a hidden linear model M(z) that generates the time series x(n) when its input is white noise. (White noise is a random signal that contains equal amounts of every possible frequency, i.e., its FFT has a flat spectrum.)

Figure 9-5 Prediction as modeling the system that produced the time series

When we use the linear adaptive filter H(z) to predict the next sample and the prediction is successful the output error is also white noise, that is, E(z) = k. Under this condition, it is easy to show that the block D(z) has to be equal to k/M(z), since,

9.27

which means that the zeros of H(z) correspond to the poles of the unknown system M(z). By using the adaptive linear combiner (which is a moving average (MA) model) we are modeling M(z) as an autoregressive (AR) model. This inversion of models is rather important and should be remembered. Hence this scheme produces very good results when the time series has spectra with sharp peaks and broad valleys, which is indicative of an all-pole model. When the time series spectrum has narrow valleys, a linear combiner of small order is not very effective.

The coefficients of the linear combiner store the extracted information from the time series, in the sense that different time series normally yield different AR coefficients (note, however, that two time series can yield the same AR coefficients as long as they have the same autocorrelation function). Therefore the AR coefficients can be used to characterize the time series. For example, in speech recognition, the model parameters are used as input to a classifier.

The order of the linear combiner is related to the complexity of the time series. Each spectral peak requires a pair of poles, so it will require two weights in the linear combiner. The spectral "tilt", how the energy decays across frequency must also be modeled and requires several other (real) poles, which also imply extra weights. By counting the number of peaks in the spectrum (and increasing this number somewhat to model spectrum tilt), we can estimate the order of the linear combiner. If more accurate estimation is required, then Akaike or Rissanen criteria explained in Chapter 5 should be utilized.

NEUROSOLUTIONS EXAMPLE 9.8

Prediction for Modeling

One application of prediction is to derive the signal generation model. We can apply this approach to extract parameters of real-world data and use them to quantify the system that produced the data (called signal modeling) or as features for pattern classification.

For signal modeling, we use the same breadboard as in Example 9.7. We let the system train and look at the frequency response of the system, which is obtained by sending an impulse through the model and taking the FFT. After training, the filter weights and the parameters of the system that generated our input signal should be very similar.

Next we illustrate the use of the linear combiner for feature extraction. We start with a 6-tap delay line, and the data to be used is the sustained sound "a." We let the system train and look at the coefficients. Then we modify the file to the sustained sound "i" to see the difference in the coefficients. Notice also how different the two frequency responses are, but since they are FIR, we can see broad peaks. If you look closely at the spectra of the input and the frequency response, they seem to be inverted.

Train the system several times, and see that the weights are similar. Now change the training to batch, and see how much smoother the overall spectrum gets. Increase the size of the delay line and observe as the frequency response becomes more wavy, with more detail. The weights of the linear combiner can be used to classify the speech segment. They are, in fact, used in speech recognition and they are called the linear predictive coding (LPC) coefficients.

NeuroSolutions Example

Let us point out the ingenuity of the prediction formulation for system identification. Notice that we are using a supervised framework, but we have just a single time series, which is exactly what characterizes unsupervised (self-organizing) frameworks. Prediction is one of the cases (the other being autoassociation studied in Chapter 6) where a supervised framework is utilized to model the input data.

9.6.3 Model-Based Spectral Analysis

The prediction framework also provides a very powerful method of doing spectral analysis, by computing the spectrum of a time series using an implicit model. Eq. 9.27 basically explains it all. Note that if the input to M(z) is white (and normalized power), then the spectrum of the time series is X(z) = M(z), so we can also say that the Z transform of the time series is

(9.28)

when a linear combiner with D delays is used for the model. Notice that the spectrum of the time series can be computed by making [image], where L is the number of frequency samples we want to obtain in our spectral representation. This spectrum estimator assumes an AR model for the time series and produces very peaky spectra (in fact, distorting the spectrum for signals that have a different generation mechanism). It is frequently used in spectral analysis when one wants to quantify the presence of closely packed sine waves and the data segments are of short duration.

When LMS is the learning algorithm to obtain the weights of the model the precision of the technique is poor due to the difficulties of gradient descent learning (rattling and coupled modes). Note that the weights define the position of the poles of the inverse system, so even if all but one of the coefficients are at their optimal values, ALL the poles will be misplaced. This high sensitivity to the placement of the zeros in inverse modeling is well known. Therefore in AR-based spectral analysis we normally prefer the RLS algorithm or the analytic solution (called the Levinson-Durbin algorithm, see Haykin).

NEUROSOLUTIONS EXAMPLE 9.9

Prediction for Spectral Analysis

In the previous example we were a step away from doing model-based spectral estimation. Conceptually, what we need to do is to invert the MA model obtained during training to create a system given by Eq. 9.28. Note that this can be achieved easily if we create a TDNNAxon of order N in cascade with a Full Synapse and feedback its output to the TDNNAxon. All the weights determined during training should be copied (entered by hand) into the Full Synapse.

The difficulty comes in the quality of the model obtained during LMS training. Notice that for model-based spectral analysis we are creating a recurrent system, that is, a system with poles that may be unstable, so any minor discrepancy in the linear combiner weights may leads to unstable spectral models. Particularly for speech, where the signal has a peaky spectrum, this is almost always the case, unless the step size of the LMS is well controlled (also try batch learning, which tends to be more accurate). Try to display the spectrum obtained during the training of the "a" and "i" sounds in the previous example.

NeuroSolutions Example

Model-based spectral analysis is one application that is much more demanding than prediction. In fact, there are many similar sets of coefficients that provide a small prediction error. However, only one of these will be the actual model for the system that generated the signal. Prediction uses an input-output model, while in model-based spectral analysis we are also interested in finding the parameters of the system.

9.6.4 System Identification

In system identification the goal is to model the input-output transfer function of an unknown plant by injecting a common signal to both the plant and the adaptive linear combiner. The output of the plant becomes the desired response for the linear combiner.

Figure 9-6 Adaptive filters for system identification

The goal of the task is to minimize the output error, and since the error is e(n) = d(n) -y(n) or the difference between the plant output and the model, minimizing the error makes the model output as close as possible to the plant output. The importance of input-output modeling is that it completely describes the external behavior of the unknown plant.

When the output of the adaptive linear combiner approximates the desired response, it produces an equivalent input-output model for the plant. The error can go to zero only if the plant is exactly modeled by the linear combiner. In this case, this means that only moving average plants can be modeled exactly (if the order of the linear combiner is large enough). In all other cases, the modeling is approximate. Normally, the injected signal to the plant is white noise or a signal of broad spectrum to excite all the modes of the plant.

NEUROSOLUTIONS EXAMPLE 9.10

System Identification

The data for this example was created in the following way. In a different breadboard we stored both the input white noise and the output of a plant that is a very simple and known system (all weights equal to 1/N with N=6). The input is assigned to the input of our linear combiner, and the desired response file contains the output of the plant. We are effectively using the block diagram of Fig 9.5.

Now the idea is to train the system having as a variable the size of the model. We place both Mega Scopes and FFTs at the inputs, outputs, and desired responses so that we can have an idea of the system behavior during adaptation. We also have a probe that shows the frequency response of the model. We see that after adaptation the weights they are very similar to the plant parameters, which proves that the identification was successful.

We have done this assuming that we knew the order of the unknown system. Now we should change the number of taps to a smaller and a larger value and see the effect in the error and in the values of the weights. We should also try a more difficult plant with poles and zeros to see that in fact the error is higher. Increasing the order of our model, however is able to decrease the modeling error.

NeuroSolutions Example

Note that the linear combiner and its parameters are simply an input-output model of the unknown system. This is the reason it is called black-box modeling. When little is known about the plant, the procedure is reasonable. Alternatively, physical modeling can yield better results by providing the functional form of the plant, but it requires much more knowledge about the real-world system, which may not be available.

Remember that system identification involves the choice of a family of models. The family is defined by the topology of the model. In this case the family is the moving average (MA) models. But we will see later (and in the next chapter) other choices. The choice of the element within the family is defined by the parameterization, that is, the model order and the actual value of the coefficients. As you can expect from the description, this is not a unique procedure: there are many possible models for a given unknown plant. We should use Occam's razor argument to pick the smallest model that meets our performance criterion. In Chapter 5 we presented several methods to find the best model order, but in linear systems Akaike's criterion is probably the most widely used.

9.6.5 Inverse Modeling

The idea of inverse modeling is very important in many applications and complements the direct modeling approach that we just discussed. In inverse adaptive modeling the model system is in series with the unknown plant, and the input to the plant is also the desired response of the model (Figure 9-7).

Figure 9-7 Inverse modeling

Assuming that the output error is small, we can see that y(n) is quite similar to the input s(n), in other words, that the combination of the unknown plant and the model cancel each other. In terms of transfer functions this means

(9.29)

Inverse modeling with a linear combiner produces an autoregressive model (AR) of the plant. The modeling is perfect only if the plant is an all-pole system and the model has the same size as the plant. Otherwise, the overall plant input-output can only be approximated with the inverse model and may require large model orders for reasonable results. Another concern is the type of input. In practice we would like to have an input that is able to excite all the important modes of the plant, otherwise they will not show up at the output, which means that they will not be modeled. A practical approach is to add small amounts of white noise, which is called "dither" noise, to the plant input.

NEUROSOLUTIONS EXAMPLE 9.11

Inverse Modeling

Let us start with a very simple case of a first-order recurrent system as our plant. This case can be perfectly identified by a two-tap FIR filter. Take note of the value of the recurrent parameter.

Use a square wave at the input to produce a resonable excitation of the plant. Let the system adapt and notice that the error goes to zero, meaning we obtained perfect modeling. Read back the coefficients and observe that they exactly match the recurrent system parameters.

Now let us use a more complex signal-generation model. We will use a system that has poles and zeros (ARMA), here one pole and one zero. The pole can be exactly matched with the linear combiner; however, the zero of the model can only be modeled by a system with poles, so the linear combiner can only approximate the plant model. We know that in principle an infinite linear combiner order is needed, but in practice the error here is very small after an order of 7 or so (it depends on the value of the feedback coefficient - try several and observe the dependency).

Now change the input (and desired signal) to a sine wave and verify that the error goes quickly to zero; however, the coefficients are different from the previous values. Disable learning in the Controller to fix the weights, and modify the input back to a square wave. The waveform is distorted, that is, the identification was not successful.

The reason can be found in the inappropriate excitation of the plant (the output contains only the response to a single frequency), which makes the inverse problem very difficult to solve. (It is easy to find weights to match the amplitude and phase of the sine wave, but this does not mean that the full plant response has been identified.) Add noise and observe that the coefficients approach those of the plant.

You might think that the impulse is a good input signal, since it has all the frequencies. But effectively this is not so when a sample-by-sample adaptation is used to modify the system parameters since in the time domain the waveform is impulsive. This means that the weights will jump up and down due to the impulsive error, which does not leas to a smooth convergence towards the optimal values. White noise has also a flat spectrum and it is a much better choice for adaptive systems since the time waveform is a blend of all frequencies at all times. Try these two choices with this breadboard.

NeuroSolutions Example

In practical cases we have to handle two conditions: The plant, being a physical system may include delays, and there may be plant noise (Figure 9-8).

Figure 9-8 Practical inverse modeling for plants with noise and delays

Nevertheless, the same basic scheme can be utilized, except that we include a delay in the forward path, by delaying the desired signal k samples. A good rule of thumb is to set k as one half of the model order. The inclusion of plant noise may seriously affect the determination of the inverse, but when the noise is white reasonable results can still be obtained.

NEUROSOLUTIONS EXAMPLE 9.12

Inverse Modeling of Non-Minimum Phase Plants

This is basically the same breadboard as in the previous example, but now the system is a little more complex. It is now an autoregressive moving average (ARMA) system with a transfer function

First, make a = 1, b = 0.5 and c = 0.5 as before. Let us input a square wave and create a model with two taps. As you can see, the MSE is quite high, and the output waveform is far from the square wave. Now let us increase the order of the model to 5. Run the example again, and see that the error decreased quite a bit and the output is much closer to the square wave, even with the white noise added to the input. We can increase the model even further (to order 10), but notice that the improvement is not as drastic. The output of the linear combiner has many ripples, so although the MSE is slightly better in terms of waveform fitting the improvement is questionable.

Turn off the noise. Go back to 5 taps and change the FIR coefficients to b = 1.5. Note that the pole is the same and the zero is in the mirror location with respect to the unit circle (i.e., the frequency responses of the previous and this system are the same). Run the example and observe that the response is much worse than before. What happened? Looking closely at the input scopes, you will notice that now there is an appreciable delay between the output and the input waveforms, much larger than before. We have effectively created with b=1.5 a nonminimum phase plant. The linear combiner is faced with a much harder problem since its desired response rises earlier than the signal applid to its input. We are asking the system to be antecipatory, which is impossible for any causal system.

This is where incorporating a delay between the input to the cascade and the desired response, makes sense. Let us delay the desired response by creating a phase of -18 [image] (two samples). Run the example and verify that the MSE is again low and that the output looks much closer to a square wave. Put the noise back on and experiment with the order of the model.

Notice how much harder inverse modeling is than the direct modeling of Example 9.10.

NeuroSolutions Example

Important applications of inverse modeling are adaptive equalization of communication channels and adaptive inverse control.

9.6.6 Interference Canceling

Interference canceling is very useful when the time series of interest is disturbed with an additive narrowband interference, such as the power-line 60-Hz frequency (50-Hz in Europe). In the modern world we are surrounded by power lines, and so the 60-Hz interference is pervasive. Biological signals are very weak (from microvolts to millivolts), and the body works as an antenna at these frequencies, so data collection in medicine is very vulnerable to the 60-Hz noise contamination. The goal is to find a way to filter out the 60-Hz from the collected signal. Traditional filtering ideas would develop a filter with a single input that would allow the signal through and attenuate the noise (notch filter). Interference canceling requires an extra collecting probe that only receives the 60-Hz signal from the subject. The idea is to use an adaptive filter to subtract the 60-Hz from the collected physiological signal (Figure 9-9).

Figure 9-9 Removal of 60 Hz interference from brain waves

For narrowband interference the adaptive filter H(z) requires only two weights, because its job is to find the best amplitude and phase of the 60-Hz sinusoidal noise reference input to match the 60 Hz noise collected with the brain wave probe. Assuming that the brain waves are uncorrelated with the noise, the MSE will be minimized when the noise is removed from the brain waves.

Sometimes, no other probe is available to collect the 60-Hz signal, but we can artificially generate 60-Hz signal and assume that the line frequency is sufficiently close to 60 Hz (the line frequency varies with the load on the power system). It can be shown that in this case the adaptive filter (after adaptation) is simply a constant-coefficient notch filter, but here we have not designing the filter explicitly: we used the power of adaptation to automatically find the weights.

NEUROSOLUTIONS EXAMPLE 9.13

Interference Canceling

We start with an example with synthetic data to explain the methodology. Let us assume that we have a square wave signal that was added to a sine wave of a known frequency, but we do not know the phase nor the amplitude of the sine wave. How can we subtract it from the square wave?

One idea is to create a two-tap adaptive filter that receives as the input a sine wave of the same frequency as the interference. This filter receives as its desired response the mixed square wave + sine wave signal. What do you expect the adaptive filter will do? It will try to modify its weights so that it will produce at its output a sine wave of the same phase and amplitude as the interference. This is the best the adaptive filter can do to decrease the difference between its output and the desired response. But notice that when the filter adapts to this solution, the error becomes very close to the square wave that we would like to recover. This is the principle of noise canceling with adaptive filters.

Let us create a simple breadboard with a two-tap linear combiner that receives as input the sine wave and that gets as the desired response the square wave with the sine interference. We place a MegaScope on the breadboard so that we can see the error, the desired response (mixed signal), the input, and the output of the adaptive filter (the signals are displayed in this order). We also place an FFT probe to show the spectrum of the error and the output. Notice the system adapting. The error begins very close to the desired response, but when the system adapts, the error approaches the square wave because the system output approaches the sine wave that was added to the square wave. This can also be seen in the spectrum of the error, which starts with a low-frequency double peak and approaches the spectrum of the square wave toward the end of adaptation.

This example illustrates the power of adaptive filters for noise canceling, but notice that this only works well when the interferences are narrowband signals that we hope do not overlap with the signal spectrum. To see how crucial this is, just modify the frequency of the sine wave to coincide with the fundamental frequency of the square wave (you have also to change the frequency of the input of the filter to match the expected new frequency). The filter now removes all the energy at the fundamental frequency of the square wave, as can be seen in the spectrum of the error. Hence the error is no longer the expected square wave.

The electroencephalogram (EEG) is a signal collected with electrodes from the scalp that monitors brain activity. The signal is rather tiny (in the range of a few to hundreds of microvolts) and has a spectrum between 0.5 to 50 Hz. Very high quality (low noise and high common-mode rejection ratio) amplifiers are required to amplify the EEG, but even the best amplifiers are unable to successfully remove 60-Hz interference. Either we must collect the EEG in shielded rooms (which are very expensive), or we have to find a signal processing technique to cancel the 60-Hz interference. This is where adaptive filters can be used, as explained in the text.

Here we have a segment of the EEG with 60-Hz interference. The signal is sampled at 500 Hz, so a 60-Hz wave has approximately 8 samples per cycle. Since we did not collect any other channel, we artificially generate a sine wave of 8 samples per cycle to input to the adaptive filter.. Run the system and observe how the EEG, which is very irregular becomes cleaner. Observe the output of the system, which contains only the sine wave.

NeuroSolutions Example

The same basic idea can be used to for instance filter DC (zero frequency) from a data set by creating a highpass filter. For this case the input to the adaptive filter (a simple adaptive constant is sufficient) is a DC value such as 1. The value of the step size controls the cut off filter of the highpass filter.

NEUROSOLUTIONS EXAMPLE 9.14

Adaptive DC Removal

This example is just another application of the cancellation idea that may be very useful for preprocessing. Many times the data we collect from the outside world has a DC bias due to instrumentation problems, and we would like to use just the AC component. This is particularly true for filtering, since a lot of signal processing algorithms assume zero DC data (such as PCA).

If we use a single-tap adaptive filter with a constant value at the input and our data is presented as the desired response we effectively create a DC remover or highpass filter.

Notice that the selection of the step size also defines the cutoff frequency of the filter. This is clear from the visualization of the spectrum of the filter output. Let us modify the step size to see the effect on the operation. If we increase the step size, the output of the filter becomes more and more like the EEG. This is because the step size effectively controls the bandwidth of the adaptive filter. We can understand this if we consider that with a larger step size the system can change the weights faster, so it is able to subtract more energy from the desired response. On the other hand, if we decrease the step size, the filter takes a little longer to converge but gets only the DC value. We can see this easily in the spectrum of the filter output.

In neural networks the step size just controls how fast the system adapts, but in adaptive filtering the step size also controls the bandwidth of the system (since the adaptation of the adaptive filter happens in time, so there is a physical meaning to what signal feature the weights adapt).

NeuroSolutions Example

Another interesting application of interference canceling is the removal of noise from speech in high-noise environments such as factories, plane cabins, and so on. The idea is basically the same. The noisy speech is collected with a microphone, providing the primary signal source, and an extra microphone captures only the noise.

In these more general cases the adaptive filter has to be of larger size, since the noise source is normally far from a narrowband signal. For each two taps, the filter is able to suppress a single harmonic of the interference. The method is effective when the noise conditions are regular (stationary) or vary slowly and have marked harmonic content (such as noise produced by rotating machinery). In cases of broadband noise, only the low frequencies are reasonably canceled.

NEUROSOLUTIONS EXAMPLE 9.15

Real Example of Noise Canceling

This example shows the power of interference canceling applied to more complex signals: speech and a washing machine noise. Our speaker is Homer Simpson, and his voice mixed with noise from a washing machine is presented to the reference input of the system (s + n). At the input of the linear combiner we present just the noise (n). The linear combiner is specified as size 5 since the washing machine noise has a broader spectrum, but still has a salient peak. The linear combiner tries to cancel the noise in the desired response, leaving only Homer's voice. We present the signals and their spectra on scopes for visualization of the progression of learning.

After presenting the speech segment we listen to the error, which contains Homer's voice. We attached a DLL to the error output of the criterion to call the Windows 95 routine that plays sound files. Notice that in the beginning Homer's voice is mixed with lots of noise, but after a few iterations it becomes intelligible. Can you understand what he is saying?

NeuroSolutions Example

The same principle of noise cancellation can be applied to different problems, for instance, to the enhancement of low-level sinewaves in high noise backgrounds. Only one signal source is utilized (Figure 9-10). The signal is split into the primary input and the reference input, which is delayed to decorrelate the noise but preserve the signal correlation, since we assume it is periodic.

Figure 9-10 Line enhancer with the LMS

NEUROSOLUTIONS EXAMPLE 9.16

Finding Periodic Signals in Noise

This example explores the same basic principles of noise cancellation, but the application is different. Here we would like to find weak sine waves in high-noise backgrounds. We delay the input from the desired response to decorrelate the noise. However, this will not decorrelate the signal, since the sine is still correlated from period to period (it is periodic). The adaptive filter tries to correlate the desired response with its input. Although we are searching for a sine wave (2 weights should suffice) we recommend to use a slightly larger model (4 to 6 weights), since the samples will be very noisy.

Experiment with the delay, the learning rate, and the filter size and observe the differences in the spectra.

NeuroSolutions Example

9.6.7 Echo-Cancellation

Echo cancellation and adaptive equalization are probably two of the most widely applied interference canceling schemes due to their importance in communications. Here we will explain echo cancellation. It turns out that our telephone channels are not ideally terminated; when signals travel from the source to the destination they are reflected back to the source. This is the familiar echo that we can hear in long-distance phone calls. The echo is annoying and decreases the intelligibility of speech, so phone companies built special devices called echo cancelers to attenuate the echo.

The culprit of this problem is a transformer (called the hybrid) that receives a two-wire input from the phone and creates a four-wire output (the long-distance trunk) to send our conversation multiplexed with others. The net effect is that when caller A speaks, caller B receives the voice, but there is a return path between the hybrids of callers A and B that injects caller A voice in the outgoing port, and it bounces back to caller A phone, appearing delayed in time (Figure 9-11). For satellite communications this delay for the long echo path reaches 400 ms and is very annoying. (The local echo is not disturbing because our ear filters it automatically by means of the precedence effect.) When caller B talks, the same thing occurs in the reverse direction.

Figure 9-11 The source of echo in phone lines

A possible implementation of an echo canceler is to use an adaptive linear combiner in parallel with each hybrid, as shown in Figure 9-12 to subtract the echo from the incoming signal. The adaptive filter at the left filters the echo for caller A, and effectively identifies the system composed by the channels and the hybrid of caller B. For a sampling rate of 8 KHz a linear combiner with at least 128 taps is required.

Figure 9-12 Echo cancelers using adaptive filters

More recently, with the widespread use of modems for data transmission in our homes and offices a similar problem occurs, but now the disturbing echo is from reflections in the local subscriber loop (the short-path echo). The echo is much more troublesome in data transmission because the data rates are very high, which means that any modification of the shape of the waveforms (as produced by adding a 0 with a 1) can be interpreted as the wrong symbol, so the message must be repeated.

Adaptive filters can also be used to attenuate the digital echo quite effectively, and all modems operating above 4800 baud have a built-in adaptive filter (Figure 9-13). The idea is to have the filter subtract the echo by learning the echo-path transfer function. The signal that goes back to the modem is basically echo free.

Figure 9-13 Echo-cancellation with adaptive filter

The efficiency of the echo canceler is measured in ERLE (echo return loss enhancement) defined as

,

which computes how much the echo was attenuated.

NEUROSOLUTIONS EXAMPLE 9.17

Echo-Cancellation

The data used in this example is fabricated, but represents realistic conditions found in the real world. We generated a pulse train to model a bit stream created by a modem. This signal is then passed through a transfer function that models the lossy path of a 2-4 wire transformer as

The output signal is then added to the incoming signal to the modem, which here is modeled by a simple square wave. Instead of receiving just the square wave, the modem sees the addition of the square wave plus the echo of the outgoing bit stream.

The role of the adaptive filter is to subtract the echo from the square wave. To do this, the adaptive filter receives as the input the bit stream sent out by the modem, and tries to figure out the echo (by identifying the hybrid lossy transfer function). We have placed MegaScopes and FFT probes to observe the adaptation process.

Notice that the advantage of this technique is that it will always adapt to changes in the echo transfer function, which are known to change from line to line and with the load in the lines.

NeuroSolutions Example

9.6.8 Adaptive Inverse Control

Another important application of adaptive filters is in control engineering. The idea of controlling an unknown plant is to design a system (controller) that will produce a prespecified plant output.

Probably the most widely used method of plant control is feedback. The controller receives the command input and provides an input to the plant. The plant output is measured and fed back to the controller so that it can relate changes in the plant input with the changes produced at the plant output. Feedback control is useful, but the application of the method is not always straightforward due to physical constraints. Other methods exist, such as inverse modeling (Figure 9-14).

Figure 9-14 Inverse modeling with plant drift and a delay

In inverse modeling the adaptive filter H(z) is placed in cascade with the unknown plant, and the output of the combination seeks to match the input signal (or its delayed version). This block diagram is basically the inverse modeling that we described in Section 9.6.5. When the error is small, we see that the adaptive filter transfer function is the inverse of the plant (Eq.9.29). The delay is necessary because the plant may either have delays or be nonminimum phase, that is, may have zeros outside the unit circle, which may cause the system model to be poor or unstable (although not in this case, since we are using a linear combiner). The inclusion of the delay improves the inverse model.

With the inverse model available, we can now construct a controller between the command input and the plant just by duplicating the adaptive filter used in the inverse modeling and copying its weights, as shown in Figure 9-15.

Figure 9-15 Adaptive inverse model control

With this arrangement the plant output will closely follow the command input applied to the controller, since we "predistort" the command to the plant by the controller. This predistortion is exactly what the plant needs to follow the command input. This theoretical picture is complicated by two primary factors: nonminimum phase plants and plant drift, which can be handled, but are not discussed here.

NEUROSOLUTIONS EXAMPLE 9.18

Adaptive Inverse Control

Here we will try to control a model of a patent's lung while under anesthesia. The controller will determine the internal settings of a mechanical ventilator that helps patient breathe. We will use adaptive inverse control for this task.

Humans breathe with negative pressure by exerting their diaphragam and chest muscles to inhale air. Mechanical ventilation is positive pressure, actually blowing air into the lungs and it is almost exclusively controlled with a proportional flow control valve (PFCV), which controls the flow rate of the ventilator.

The model of the lung implemented here is a simple, first-order dynamical system modeling lung compliance and resistance. When the left and right compliances and resistances are equal, the system can be modeled as its electrical equivalent shown below:

The goal is to control the pressure at the patient's airway (at the end of the ventilator tubing). There is resistive pressure, which is a function of the flow (when flow ceases the resistive pressure goes to zero), and compliant pressure, which is based on the amount of air in the lung and the compliance ("stretchiness") of the lung, which yields

The purpose of the ventilator is to apply a square wave of pressure to the lungs. However, due to the lung compliance and resistance, what actual flow waveform should the ventilator produce to create a square wave of pressure in the lung? This is exactly what our implementation will figure out. The input to the controller is the square wave of pressure (we just model the rising slope here). Connected to the output of the controller is the "lung", here modeled as an Axon. The output of the "lung" is then fed to the adaptive part of the controller which identifies the inverse lung model. Its parameters are copied to the controller, which predistorts the command input so that the desired behavior is obtained.

This scheme can even be refined to include a reference model that will relax the condition that the plant input and the desired response of the inverse model be the same (eventually delayed).

Now let us see how we can implement such a complex scheme in NeuroSolutions. First, there are the controller and its function. We use a linear combiner followed by a logistic nonlinearity. The need for the nonlinearity arises because the real controller (the ventilator) can produce only positive signals (positive ventilator pressure). The weights of the controller are copied from the inverse model that follows the plant. A DLL transmits the coefficients. The input to the controller is a square wave added with white noise. We use white noise so that we always provide a driving force for the adaptation (the square wave is featureless except during the jumps and, just like the sine wave of our previous example, is not sufficient to drive all the modes of the plant).

The lung dynamics given by the preceding equation are modeled in the Axon. The Axon is an input-output device, so through DLLs we can create arbitrarily complex programs in C language that model real systems. For each input to the Axon the C program creates the corresponding model output.

The inverse model is created by a linear combiner followed by the same logistic nonlinearity. The desired response to the inverse model is received from the input to the lung through a DLL. Let us see how this works.

NeuroSolutions Example

Go to next section


 9.7 Applications of Temporal PCA Networks

The ideas and learning algorithms covered in Chapter 6 for Hebbian networks can also be extended to time signals with minor modifications in the topology. We present here an example of an eigenfilter, the decomposition into principal components of a discrete signal, a subspace method for sinusoid detection in broadband noise, and one example of blind source separation of speech signals.

9.7.1 Maximum Eigenfilter

Suppose that we have a signal that is generated by a subtractive process where some frequencies are eliminated, which happens, for instance, in the force signal collected from the cutting head of rotating machinery. Such signals have spectra with a large concentration of energy in several ranges of frequencies but have nulls, which show their moving average (MA) generation. Instead of applying the previously discussed spectral analysis method using prediction which implies an AR model, we use temporal PCA here. PCA in time produces MA models that seek the peaks of the spectrum. The single-output system is called the maximum eigenfilter and seeks the largest peak in the spectrum but assumes at the same time a moving average model. Basically the maximum eigenfilter finds a projection vector in the input space where the output power is maximized (see Chapter 6 discussion). It maximizes the Raleigh quotient.

We can train a temporal PCA network with a single output using Oja's algorithm to solve this problem. The output power will be the largest eigenvalue over time, while the filter weights correspond to the eigenvector. The larger the delay line, the more coefficients the filter will be and hence the sharper will be the eigenfilter main lobe. The matched filter is an eigenfilter for transients.

NEUROSOLUTIONS EXAMPLE 9.19

Maximum Eigenfilter

Let us create a synthetic signal to exemplify the largest eigenfilter model and then apply it to real world data. We create a signal through a subtraction process. Consider a pulse of 4 nonzero samples (-0.2, 1, -0.5, -0.3) that repeats itself every 15 samples. Let us now add 4 versions of the same signal but delayed in time by equal amounts, and add noise. We can consider this signal as a model for force pulse produced by a rotating machine with 5 teeth, and what we measure with a force sensor is the overall waveform.

The spectrum of this signal has many peaks, and the goal is to find the fundamental component. We can use the maximum eigenfilter to solve this problem. Let us create a delay line with 15 taps and use Oja's rule to train the system (just one output). We see that the output grabs the largest peak. Now we freeze the weights (by setting the learning rate to zero), and send an impulse through to compute the system frequency response. Since this is a FIR filter (implementing an MA model), we expect to see a frequency response with a large lobe and many side lobes sculped by zeros. We can increase the number of taps in the delay line and see that the output spectrum gets sharper as we would expect from the theory.

The second example is to extract the fundamental component of the sunspot time series. As we have seen, the time series is very noisy but it has a quasi-periodic oscillation. Let us use a tap delay line of 20 and train the eigenfilter with Oja's rule. We see the waveform progressively becoming cleaner showing that the high frequency information is being removed. If we freeze the weights and send a pulse through, we see the frequency response of the lowpass filter which was automatically obtained to preserve most information about the largest mode in the spectrum. This type of processing can be considered pre-processing for the subsequent analysis, and it is one of the attributes of eigenfilters.

NeuroSolutions Example

9.7.2 Optimal Filter Banks

If the network of the previous example is extended with more outputs (Figure 9-16) and Sanger's rule is utilized in the training, an optimal filter bank is achieved.

Figure 9-16 Optimal Filter Bank

The size of the time window determines the accuracy and the size of the autocorrelation function estimator. The number of output PEs establishes the number of principal components extracted. The optimal filter bank maximizes the projected power to a subspace.

When the input signal has the maximum power concentrated at low frequencies (lowpass spectrum), and there is a monotonic decay of power across frequency, the temporal PCA defaults to a bank of optimally designed bandpass filters, which for very large windows approximate the spectral decomposition achieved with the FFT. In temporal PCA, pairs of neighboring PEs tune to the same basic band since a given frequency band can be covered by two orthogonal components of the same basic frequency (as the sine and cosine waveforms). As we can expect, the bandpass filters are FIR, that is, PCA in time implements a moving average (MA) model.

We can easily observe the frequency response of each eigenfilter if we stop training and input a delta function to the system. The analysis of the decomposition is not straightforward, except for the limit of long windows or for signals that have a line spectra such as the square wave. Closely analyzing the solution for this case, we see that the zeros of lower order eigenfilters are placed over the top of the peaks of higher order eigenfilters, which can be expected from the orthogonality condition of the PCA components.

When the input signal spectrum is not lowpass or the limit of long windows is not verified, this simple analysis no longer holds. The principal component still captures the largest spectral mode (which is broad and may be anywhere in the spectrum). The lower-order eigenfilters have to compete for the other spectral modes, but now there is no guarantee that the eigenfilters will be unimodal and ordered. Nevertheless, the system still finds the best (ordered by variance) orthogonal projection of the input.

NEUROSOLUTIONS EXAMPLE 9.20

Optimal Filter Bank

Let us use a square wave as an example of a signal that has a lowpass, monotonically decreasing spectrum. If the PCA decomposition in time is implemented with Sanger's rule, we see that the system self-organizes in such a fashion that pairs of eigenfilters grab one harmonic of the square wave. If we stop the training and estimate the frequency response, we see that the eigenfilters are effectively organized in pairs of increasing frequency. Note also that the zeros of the higher-order eigenfilters are placed on top of the preceding eigenfilters bandpass frequencies.

Now let us change the input to a speech segment of the sound "ah" as in "hot". The first three formants should appear at f1= 730 Hz, f2 = 1090 Hz, and f3 = 2440 Hz. We use a PCA network with 20 input taps and 6 outputs. After training you see the first two eigenfilters peak at the first formant, the second two at the second formant, and the last 2 picking up some low frequency activity but also some higher frequency activity. If we change the FFT display to log, we can see this feature more clearly. Notice that this decomposition produces very interesting features in the time domain.

NeuroSolutions Example

9.7.3 Spectral Analysis by Eigendecomposition

Spectral estimation based on autoregressive models loses accuracy rapidly when the noise in the data increases. Alternatively, the eigenvectors of the autocorrelation function can be used to find sine waves in white noise. The idea is that in the eigendecomposition of the autocorrelation function, the input signal energy is divided into orthogonal components. Since sine waves are the natural modes of the PCA decomposition for long observation windows, one of the eigenvectors is associated with the sine wave, while the noise projects everywhere.

NEUROSOLUTIONS EXAMPLE 9.21

Noise Removal by PCA

Let us utilize the previous breadboard but now change the input source to a single sine wave with a large amount of additive noise. We decrease the number of outputs to 3 to show exactly what we described in the text. We chose the number 3 because a sine wave will appear in two eigenfilters, while the third eigenfilter will simply show the noise.

We start with the sine wave amplitude of 1, 10 samples per cycle, and add uniform noise of variance 2. The sine wave is only faintly visible in the time signal, but the spectrum still shows a very clear peak. The network has no problem of providing a nice sine wave in the first two eigenfilters. If we now decrease the amplitude of the sine to 0.3, we do not see the sine wave even in the spectrum, but the system still trains to produce the sine wave. For an amplitude of 0.1 the system is unable to find the sine wave, thus there is a limit to this technique. We can increase the window to 50 samples and decrease the learning rate. Most of the time we can see a spectral line in the spectrum of the first eigenfilter that can be traced to the sine wave.

NeuroSolutions Example

This same idea can be extended to multiple sine waves in noise or other man-made signals. The directions that contain only the white noise components generally correspond to smaller eigenvalues. If we truncate the eigendecomposition when there is a large drop in the magnitude of eigenvalues, we thus retain the signal components (signal subspace) and attenuate the noise.

An alternative uses only the noise subspace (normally represented by the last PCA component) to find the sinusoidal components. The noise subspace is orthogonal to the signal subspace. If we use the minor component of the PCA as a filter, we guarantee that it will have zeros in the unit circle where the signal components are located. We can copy the weights of the minor component to a recurrent system implementing the inverse system, whose frequency response has peaks where the sinusoids are located (since the zeros give rise to poles of the inverse model). We already did this when we discussed model based spectral analysis, except that here we are using the eigenfilter that corresponds to the noise subspace as the model. It has been shown that this method is highly accurate for determining the frequency of sinusoids (the MUSIC algorithm Kay). One of the problems of this method is that it requires knowledge of the number of sinusoids included in the signal to appropriately set the size of the PCA space to at least one larger than twice the number of sinusoids.

NEUROSOLUTIONS EXAMPLE 9.22

Spectral Analysis by Eigendecomposition

We can use the previous breadboard to train the PCA in the case of sine wave plus noise case. We use the anti-Oja rule, which is simply anti-Hebbian normalized to unit norm weights. Hence the network has a single output, and we select a 5 tap network. In principle we only need a 3 tap filter but the lack of resolution will be noticeable. We train the system using the Scheduler for the step size. Once the PCA has converged, we copy the weights of the weight matrix and call it H(z). These weights represent the eigenfilter that represents the noise subspace, and its frequency response is guaranteed to have zeros at the frequency of the sinusoid embedded in noise. According to the theory of noise subspaces, the spectral estimator should be 1/H(z).

To create this transfer function we create a recurrent system in NeuroSolutions by feeding back the output of a linear combiner to itself. The recurrent system transfer function is

where G(z) is feedforward transfer function. To obtain the spectral estimator, with this breadboard, we have to make G(z) = 1- H(z). This can be accomplished easily if we exchange the signs of all the coefficients of H(z) and add 1 to the leading term (i.e. , the coefficient for lag zero). Run the network and observe that the spectrum peaks at the desired frequency. With this method the frequency can be estimated very accurately.

NeuroSolutions Example

9.7.4 Blind Source Separation

Blind source separation is an important problem in science and in engineering. It can be formulated in the following way: Suppose that we have M sources s1(n), s2(n),…,sM(n) assumed statistically independent, which are unknown, and mixed by a linear but unknown system A, i.e. X=AS . Here we deal only with the case that the mixing is instantaneous, that is, the system is an unknown scalar mixing matrix A. We do not know the sources nor the mixing matrix, but we collect M signals x1(n), x2(n),…,xM(n) from the output of the mixing system (i.e., we have as many mixed signals as sources). The goal is to find a system W that is able to demix the signals, i.e. Y=WX with y1(n) [image] s1(n) (Figure 9-17). This is a difficult problem since we do not know the sources nor the mixing matrix.

Figure 9-17 The problem of blind source separation

To illustrate this problem, let us assume that we are in a room with many people talking at the same time (called the cocktail party effect). Our auditory cortex is able to focus on a given speaker and distinguish his or her voice from the others. Our senses are basically doing blind source separation because we do not know the source (the speech is unknown to us) and because the room acoustics and the placement of the subject with respect to the others and to us is unknown. Humans can do this separation with just two ears, but here we assume that we have as many measurements x(n) as sources s(n).

We use the ideas of anti-Hebbian learning coupled with a tap delay line to create a system that is able to demix the signals. How can this simple idea be used to perform blind source separation? For the sake of simplicity, let us use just two sources. We will use a recurrent system built from two anti-Hebbian networks like the ones in the Figure 9-18. Principe et al

Figure 9-18 The solution based on novelty filters

By assumption, s1(n) and s2(n) are orthogonal (since we assumed that they are independent). When we start the adaptation using anti-Hebbian learning, y2(n) will push y1(n) away from it, while y1(n) will make y2(n) rotate away from its direction. As the figure shows, they will stop the rotation when y1(n) is orthogonal to y2(n), which coincides (apart from a premutation and scaling) with the directions of s1(n) and s2(n). This is the principle that we utilize to do blind source separation.

NEUROSOLUTIONS EXAMPLE 9.23

Blind Source Separation with Anti-Hebbian Learning

This breadboard is complex, not because of the task, but because of the implementation in NeuroSolutions. There are two identical parallel channels that are interconnected as shown in Figure 9-18. Each is a tap delay line followed by a set of weights, which are adapted using anti-Hebbian learning with weight normalization. The normalization of the weight vector to unit length is necessary to avoid convergence to the trivial solution (zero weights). This is accomplished by writing a DLL for the adaptation.

The first two Axons in each channel are there just for synchronization purposes. The two mixed signals are applied to the input of the network, and the output is plotted in scopes. We also create a sound file from the top channel, so that we can appreciate the evolution of the demixing while the system is training. In the beginning it is impossible to understand the sentence. After 15 iterations over the data set, this system is able to demix the two speech signals.

Notice that filtering would be impossible, since the two speech waveforms share the same frequency band. The demixing uses the statistical properties of the two waveforms to achieve the separation, without knowing the sources nor the mixing matrix.

NeuroSolutions Example

Go to next section


 9.8 Conclusions

In this chapter we covered adaptive filters. We showed that an adaptive filter is a linear regressor in signal space because it tries to fit a line to a signal trajectory. We showed that we can adapt linear filters with our well-known LMS procedure.

Another class of linear systems that we covered is the temporal PCA network. Borrowing the ideas of PCA from Chapter 6 we can extend them to time by including a tap delay line in the PCA architecture. Temporal PCA computes eigendecompositions in time. We can therefore design maximum eigenfilters, novelty filters, and KLTs.

Probably the most important part of Chapter 9 is the application section. Here we tried to show many possible uses of adaptive filters and their advantages in digital signal processing. We cover a wide range of applications, from prediction, system identification, noise cancellation, to inverse adaptive controls. We also showed how the temporal PCA networks can be used in spectral analysis, in blind source separation, and for eigenfilters.

Go to next section


 9.9 Exercises

9.1 Use the breadboard of Example 9.7 to predict the signal stored in the file prob9.1 contained in folder chapter9 of the data directory (CDROM).

9.2 Use the breadboard of Example 9.8 to create the model of the data for problem 1.

9.3 Use the breadboard of Example 9.9 to estimate the power spectrum of the signal stored in the file prob9.1 contained in folder chapter9 of the data directory (CDROM).

9.4 Use the breadboard of Example 9.10 to identify the system where the output is stored in the file prob9.1 contained in folder chapter9 of the data directory (CDROM).

9.5 Use the breadboard of Example 9.12 to create an inverse model of plant whose output is stored in the file prob9.1 contained in folder chapter9 of the data directory (CDROM).

9.6 Use the breadboard of Example 9.13 to attenuate the noise in the sound file prob9.6 contained in folder chapter9 of the data directory (CDROM).

9.7 Use the breadboard of Example 9.16 to find a periodic component in the signal stored in the file prob9.7 contained in folder chapter9 of the data directory (CDROM).

9.8 Use the breadboard of Example 9.17 to cancel the echo in the signal stored in the file prob9.8 contained in folder chapter9 of the data directory (CDROM).

9.9 Use the breadboard of Example 9.19 to create the maximum eigenfilter for the signal in the file prob9.9 contained in folder chapter9 of the data directory (CDROM).

9.10 Use the breadboard of Example 9.20 to create a PCA decomposition of the signal stored in the file prob9.10 contained in folder chapter9 of the data directory (CDROM).

9.11 Use the breadboard of Example 9.21 to remove the noise from the signal in the file prob9.11 contained in folder chapter9 of the data directory (CDROM).

9.12 Use the breadboard of Example 9.22 to estimate the power spectrum of the signal in the file prob9.12 contained in folder chapter9 of the data directory (CDROM).

9.13 Use the breadboard of Example 9. 23 to separate the speech signals in the file prob9.13 contained in folder chapter9 of the data directory (CDROM).

Go to next section


 10.1 Static versus Dynamic Systems

Chapter 9 was the first chapter to cover systems (dynamic systems) that process time information. Before that chapter, all the learning systems we studied iplemented static input-output transformations (static mappers). The system response was computed instantaneously from the available input pattern and did not change if the input was kept constant. This is an important subset of learning systems, but, as we saw in Chapters 8 and 9 there are systems that do not have instantaneous responses to inputs. They need some time to stabilize their outputs for a given input excitation.

A linear combiner has memory, that is, it preserves the past values of the input internally in the tap delay line. When an input is applied, the input samples need to propagate through the filter topology. The response of the FIR for a constant input will stabilize only after a time equal to the number of delay line taps (the transient response). Static mappers have zero transient time, while the linear combiner has a finite transient response.

Another important class of systems with nonzero transient response is the recurrent topology that we discussed in Chapter 8. Recurrent topologies have feedback, where the system output or the output of an intermediate PE is fed back to the input or some internal PE. The feedback brings the influence of the past input (or past output) to the response at the present time so it is also a form of memory. If the system has feedback, the response in general takes time to stabilize. For some values of the system parameters the response may never stabilize, so we see that new system properties (stability) emerge for recurrent systems. Both these systems (memory based or recurrent) are called dynamical networks. All analog systems are dynamic, because their responses take a finite time to reach a steady value (if at all). This naturally creates the concept of time scale and introduces new properties when compared with static mappers. In Chapters 8 and 9 we restricted the study to linear dynamic networks, but here we will expand the scope to include nonlinear dynamic networks.

The weights in a static mapper, such as the multilayer perceptron, also represent memory about the data it has been trained with. This type of memory should not be confused with the memory we discussed for the linear combiner. Therefore we call long-term memory the information contained in the weights, and short-term memory the past information available as data within the neural network topology. short-term versus long-term memory

In digital simulations we have the power to artificially control time. The sampling frequency links the samples to time, but we can, for most practical applications, think of the data as a sequence of numbers and ignore the underlying time scale. Moreover, the availability of short-term memory structures used at the system input allows the system to work with many samples at a time, as if the input was static. In this sense short-term memory structures have the power to transform dynamic phenomena into multidimensional static patterns. We saw this in Chapter 8 where the output of a delay line of D taps was shown to be equivalent to a point (a static pattern) in a space of size D. An important question is: Are dynamic models preferable to static models? This is a difficult question to answer. Conventional statistics requires static data, or at least properties that do not change over time (stationary signals). A signal in time is a function, so functional analysis is required for their study, which complicates matters. It is fair to say that our mathematical tools are much more developed for static than for dynamic models, which may explain the generalized use of static modeling in neurocomputing.

On the other hand, we see that the processing power of biological systems is still more accurate and robust than most man-made systems, and we have to pause and wonder why. If there is any advantage in time processing, biological systems, which evolved embedded in time signals, must have learned how to use time for information processing. It is therefore important to look at time processing as a promising, albeit more complicated, alternative to the more conventional theory of static machines. When compared with static systems, the weights of dynamic systems are capable of codifying enhanced information about the input (through filtering in the short-term memory window). In fact, we can consider a static mapper as a special case of a dynamic system with a window length of one sample. static versus dynamic modeling

NEUROSOLUTIONS EXAMPLE 10.1

Time Disambiguates Data Clusters

Let us create the problem we discussed above: a single input datafile containing the amplitudes of 2 alternating sine waves of different frequencies and equal amplitudes. Let us assume that one frequency is one class and the other frequency belongs to another class. If we try to separate the two segments with a single-input, one-hidden layer MLP, the class separation must be based on the amplitude of each sample. It is easy to convince ourselves that the task is impossible since the two classes have exactly the same amplitude mean and range. We will show this with the Scatter Plot.

Now let us substitute the input Axon by the simple, first-order, recurrent lowpass filter we introduced in Chapter 8. Notice that the number of inputs and system topology is the same, however, when we run the system the separation becomes possible, with some error. Notice that the errors are near the DC value.

What is the new information that the recurrent PE brings to the processing? The recurrent PE adds its past value to the present input, bringing information from the past samples to the present processing task. This shows that the recurrent system is exploring information from the signal time structure and that this information is crucial to solve the problem.

If we change the value of the time constant to 0.5, the MLP is unable to differentiate the two classes. How much information we use from the past (given by the value of the feedback parameter) is crucial to separate the classes.

Note that if we knew that the signals were sine waves other simpler methods could be used to make the separation (a simple zero-crossing detector could do the classification). But here we are using the black-box approach, assuming no knowledge about the patterns to solve the problem.

NeuroSolutions Example

Dynamic neural networks are topologies designed to explicitly include time relationships in the input-output mappings. Time-lagged feedforward networks (TLFNs) are a special type of dynamic network that integrate linear filter structures inside a feedforward neural network to extend the nonlinear mapping capabilities of the network with a representation of time. In this context, the delay line of the linear filter is called a short-term memory mechanism . In TLFNs a representation of time is created inside the learning machine, as opposed to windowing the input signal. We start by studying several linear short-term memory mechanisms and combine them with nonlinear PEs in restricted topologies called focused . Focused TLFNs have memory only at the input layer and can still be adapted with static backpropagation, so we can use our existing tools and knowledge to apply focused TLFNs for time processing. Then, we study the function of memory PEs and show that they define the basis for the local projection space. The delay line is compared with the context PE, and a unifying memory PE called the gamma PE is introduced.

Go to next section


 10.2 Extracting Information in Time

As we saw in earlier chapters, the static classification problem is well understood. It can be formulated as a special case of an arbitrary mapping between two vector spaces (function approximation) with indicator functions. In static pattern recognition the dimension of the input data vector defines the size of the pattern space (e.g., if the experiment has two independent variables, the size of the input space is 2-D). In static pattern recognition the data clusters are described statistically (mean and variance or more rigorously with the probability mass function). The classification problem remains the same when we shuffle the presentation of the data to the learning machine, because we assume that there is no ordering in the data clusters (i.e., we assume no internal sequencing in the data). In static pattern recognition the way the data is drawn from the input space is irrelevant.

Let us now assume that, instead of static inputs we seek to recognize time patterns from an M dimensional, locally stationary signal x(n). Locally stationary means that we assume that the signal is stationary for the length of observation as in Figure 10-1.

Figure 10-1 Processing of time varying signals with an external window

In temporal problems the measurements from the world are no longer an independent set of input samples, but functions of time. A single sensor produces a sequence of measurements that are linked by an ordered relation, the signal time structure. If we change the order of the samples we are distorting the time signal x(n), and changing its frequency content, so sample order must be preserved in temporal processing.

An important question is how to select the length of time (called the window length) to represent the properties of x(n). As we saw in Chapter 8, the signal time structure limits the dimensionality of the manifold where the signal trajectory exists. We can normally work with small dimensional spaces without loss of information. The problem is that we have to find the appropriate reconstruction space size by selecting the length T of the time window. Moreover, we can use this projection to enhance the difference between the signal we are interested and all the others (i.e., by adapting the network coefficients). This is the basic idea of filtering.

NEUROSOLUTIONS EXAMPLE 10.2

Disambiguate Data with the Tap Delay Line

For the same problem of Example 10.1, let us modify the input PE to a tap delay line. Enter the values of the filter weights (all 1, which corresponds to a lowpass filter). Start with 3 taps and find out that the system never learns. Then go to 5 taps and verify that the system learns fairly well. Then go to 15 and see that the number of errors decrease. Reduce the number of taps but proportionally increase the delay between taps to find that what matters is not the number of weights in the system, but the length of the time window.

Next let the weights of the FIR adapt automatically. Now find out whether the system found the same solution. It did not. This means that there are other solutions that minimize the error, which points out the importance of filtering.

NeuroSolutions Example

To expolit the signal's time structure, the learning machine must have access to the time dimension. Since physical systems are causal, the search is restricted to the past of the signal. Physical structures that store the past of a signal are called short-term memories, or simply memory structures. Basically, a short-term memory structure transforms a sequence of samples into a point in the reconstruction space.

In Chapter 9 the tap delay line was introduced and was used in conjunction with a simple adder to build the linear combiner. In this case we can either consider the memory as a window over the data or part of the learning machine. With the former view the linear combiner is just a regressor, which does not offer the right perspective for understanding filtering. Moreover, the first-order recurrent system also has memory but this memory cannot naturally be described in terms of a windowing operation. Therefore a more rigorous approach must be taken.

In this chapter we incorporate memory structures inside the learning machine. This means that instead of using a window over the input data, we will create PEs dedicated to storing either the history of the input signal or the PE activations (Figure 10-2).

Figure 10-2 The new paradigm for processing time signals

By bringing the memory inside the learning machine, we expect to improve the efficiency of the representation of time because the learning machine can:

  • Choose the size of the time window that best suites the processing task.

  • Choose the weighting of the data samples in the window in order to decrease the output error.

  • Receive only the present sample from the external world, just like the biological archetype.

The added advantage of temporal processing is that the learning system can utilize filtering (frequency domain information) to reach the processing goal. It is also obvious that the learning machine is now required to perform a more complex function, because it has to codify time information. Fully recurrent networks have this ability, but they are complex. In this chapter, we study topologies of intermediate complexity between that of the static networks and fully recurrent networks.

Go to next section


 10.3 The Focused Time Delay Neural Network (TDNN)

We already have the knowledge to create, train, and understand our first dynamic neural network by joining the linear combiner and the multilayer perceptron. More specifically we replace the adder from the linear combiner with several nonlinear PEs in a feedforward arrangement. Alternatively, we replace the input PEs of an MLP with a tap delay line. It is called the focused time delay neural network (TDNN). The topology is called focused because the memory is only at the input layer (Figure 10-3). TDNNs with short-term memory distributed across the network were first applied by Waibel for speech recognition.

Figure 10-3 A focused TDNN with one hidden layer and a tap delay line with k + 1 taps

The delay line of the focused TDNN stores the past samples of the input. The combination of the tap delay line and the weights that connect the taps to the PEs of the first hidden layer are simply linear combiners followed by a static nonlinearity. The first layer of the focused TDNN is therefore a filtering layer, with as many adaptive filters as PEs in the first hidden layer. The outputs of the linear combiners are passed through a nonlinearity (of the hidden-layer PE) and are then further processed by the subsequent layers of the MLP to achieve one of the following tasks:

  • In classification the goal is to find weights that separate the signal trajectories which correspond to different time patterns.

  • In system identification the goal is to find the weights that produce a network output that best matches the present output of the system by combining the information of the present and a predefined number of past samples (given by the size of the tap delay line).

  • In prediction the goal is to approximate the next sample as a nonlinear combination of past input samples (given, once again, by the size of the tap delay line).

For all practical purposes we can think of the focused TDNN as an MLP that uses not only the current sample of the time series but also a certain number of past samples. We can design an MLP equivalent to the focused TDNN if we use multiple samples from the time series instead of a single input and the tap delay line. Effectively we place a rectangular window over the time series and choose an MLP with as many inputs as samples in the window. Note, however, that in this interpretation the memory becomes external to the network.

The problems of designing TDNN topologies are the same as discussed for the MLP, with the addition of choosing the size of the tap delay line (also called the memory layer). We will discuss later a more rigorous way to address this issue. Here it suffices to say that the size of the memory layer depends on the number of past samples that are needed to describe the input characteristics in time. This number depends on the characteristics of the input and the task, so it has to be determined in a case by case basis. Before we are able to apply the focused TDNNs to solve problems we have to address how to train them.

10.3.1 Training the Focused TDNN

One of the appeals of the focused TDNN of Figure 10-3 is that it can still be trained with static backpropagation, provided that a desired signal is available at each time step. The reason is that the tap delay line at the input does not have any free parameters, so the only adaptive parameters are in the static feedforward path. To train these weights we simply use the backpropagation algorithm of Chapter 3 without any modifications. We have already successfully used static backpropagation to train the network weights for Examples 10.1 and 10.2. We further discuss the training of dynamic neural networks in the next chapter.

10.3.2 Applications of the focused TDNN

The focused TDNN topology has been successfully used in nonlinear system identification, time series prediction, and temporal pattern recognition. We have demonstrated this topology with temporal pattern recognition for the simple case of discriminating between two sine wave segments. We will now create a more challenging task of phoneme classification.

NEUROSOLUTIONS EXAMPLE 10.3

Temporal Pattern Recognition with the Focused TDNN

In this example we will exemplify the power of the focused TDNN to classify phonemes. A phoneme is the fundamental building block for the sounds that make up our spoken language. The phoneme is the sound equivalent of a letter in our written alphabet. The problem with phonemes is that they exist in time. The phonemes are produced by puffs of air coming out of the lungs that make our vocal cords vibrate (sometimes) and are shaped by the resonances of our vocal tract. When we collect sounds with a microphone we translate the sound into electrical voltages, and we can see them on an oscilloscope. Each phoneme corresponds to a different waveform, so if we want a TDNN to differentiate among phonemes, we have to do time processing.

Speech researchers found that a good way to characterize phonemes is to model the time series with an AR model. In Chapter 9 we saw that through prediction we can adapt the linear combiner to predict the next time series sample. The coefficients of the predictor are called linear predictive coding (LPC) coefficients and are used here to characterize each segment of the speech waveform. Every 10 ms (roughly 128 samples) we read the coefficients of the linear combiner, obtaining a set of 11 LPC coefficients. THus the speech time series is translated into a sequence of LPC vectors (of size 11), which will be input to the focused TDNN.

Now the problem of training the TDNN is relatively easy to explain. When the sound we want to classify appears in the time series, we give a desired response of 1 to the TDNN. When the phoneme is absent, the desired response should be 0. We therefore have to create different desired signals in time to train the TDNN to recognize phonemes. Here the desired response will be a square wave of 0 (phoneme absent) and 1 (phoneme present). This is one of the differences between temporal and static pattern recognition: the desired response is also a time signal.

The other big difference between static and temporal pattern recognition is that here the pattern has an unknown and variable duration. We have to set the size of the tap delay line so that it looks back in time to recognize the time structure of the phoneme, which is translated here into a sequence of LPC vectors. This is not a trivial problem. If the window is too small, there may not be enough data from the past and performance will suffer. On the other hand, if the window is too long, there will be many more weights in the network which will result in slower training. The size of the window is thus a design parameter. In the TDNN we have to physically modify the network topology to adjust the length of the window in time. In this example, the LPC coefficients contain some of the temporal information, thus we only need 3 taps for our tap delay line.

In the TDNN we can still use static backpropagation to train the system. There are no recurrent connections – the memory simply creates a larger static set of inputs to the standard MLP. This is one of the advantages of this topology; the weight gradients are instantaneous because the net is static. Let us train the system and see how it works.

As with the examples of previous chapters, you should modify the parameters of the system to see how well it works, and when it fails. Understanding why a system fails is normally much more enlightening than when everything goes right. In particular, you should observe the difference between dynamic and static pattern recognition. Just set the tap delay line to have a single tap, so that the classification is done with only one LPC vector. Try to train the system. You will see that the performance drops, that is, by including time information the classifier is improved.

NeuroSolutions Example

Next, the focused TDNN is applied to nonlinear system identification, extending the linear combiner to nonlinear models.

NEUROSOLUTIONS EXAMPLE 10.4

Nonlinear System Identification with the Focused TDNN

This example studies system identification. To illustrate the power of the TDNN, we are going to model a nonlinear plant and compare the results of the TDNN with those of the linear combiner with the same number of delays. The plant is given by the following equations

The input is

The block diagram for system identification is still the same as in the linear case. A common input is given to the TDNN and the plant, and the desired response for the TDNN is obtained from the plant output. We have stored both the input and the plant response in files to make it compatible with NeuroSolutions. In this case the input is a sum of two sinusoids. We put Megascopes at the input and at both the model and plant outputs to see how the learning progresses.

We start with the linear combiner of order 10. Run the simulator and observe that the error immediately stalls at a very large value. Increasing the size of the linear combiner does not help because of the nonlinear nature of the plant. Put a FFT at the input and output and observe that new frequencies are created by the plant; hence a linear system is hopeless in this case.

Let us now see how the focused TDNN does. It is amazing that a TDNN with 7 PEs in the hidden layer does an almost perfect job, and the training is very fast. Modify the size of the hidden layer and of the input layer to better understand their role in the solution. For instance, is there a size of the tap delay line below which the system does not train, irrespective of how many PEs we use in the hidden layer? Conversely, is there a minimum number of hidden PEs below which, no matter how large the time window, the system never trains?

NeuroSolutions Example

Finally, we apply the focused TDNN to chaotic time series prediction.

NEUROSOLUTIONS EXAMPLE 10.5

Nonlinear Prediction with the Focused TDNN

This example deals with nonlinear prediction. As explained in Chapter 9, prediction is a form of system identification in which both the input and desired signal come from the same source but at different time intervals (the input is delayed, normally by one sample). With this introduction we can see that if the system that created our time series is not linear, the linear combiner will not be able to perform as well as the TDNN.

The example that we prepared is a time series that is produced by a nonlinear dynamical system, the Mackey-Glass system, given by

As we can immediately see, the system is nonlinear (notice the power of 10 in the denominator), and with a delayed feedback (the delay [image]), so we can expect a very complex oscillation. Notice that the equations are written in continuous time, so we have to discretize the system. Here we use a fourth order Runge-Kutta integration, downsample it by 6, normalize to [-1, 1] and use a delay of 30 (this is called the MG30 time series). MG30 is a chaotic time series, because the waveform is always different and unpredictable in the long run.

Once we have the time series, we just need to delay the input to the TDNN by 1 and run the breadboard. We compare the performance of the TDNN with that of the linear combiner of the same memory size. As we see the TDNN is superior in capturing the fine detail of the waveform, but the linear combiner does an excellent job in approximating the major features of the waveform, which clearly shows why linear systems have been in use for so long and still satisfy most of the requirements.

You should experiment with the size of the delay line and the size of the hidden layer to better understand the breadboard. You can also delay the input by more than one sample and see how much more difficult the task becomes. The linear system breaks down quickly while the TDNN still works reasonably well.

If the waveform is chaotic, that is, unpredictable in the long run, does it make sense to try to predict it? As we can see, there is a deterministic equation that relates the samples, so locally there is a structure that can be captured by the predictor. One of the interests in chaotic time series prediction is related to the difficulty of the task, which provides an excellent ground to test predictors. Moreover, many signals from the real world that we usually model as noise may in fact be chaotic, and we may be able to model them.

NeuroSolutions Example

The focused TDNN is a great compromise between simplicity and processing power, but it is not the only dynamic neural network topology as we saw in Example 10.1 with the first order lowpass filter. We are interested in formulating a theory for the processing of time signals with neural networks, so we will develop a new neural component explicitly designed to remember the past, which we call the memory PE. We will see that the ideas of filtering explained in Chapter 8 are crucial to understand the memory PEs.

Go to next section


 10.4 The Memory PE

Figure 10-4 shows a block diagram of the new memory PE, where g(.) is a delay function.

Figure 10-4 The memory PE

The memory PE receives in general many inputs xi(n) and produces multiple outputs [image] which are delayed versions of y0(n) the combined input,

(10.1)

Analyzing Eq. 10.1 we conclude that these short-term memory structures can be studied by linear adaptive filter theory if g(.) is a linear operator. It is important to emphasize that the memory PE is a short-term memory mechanism, to make clear the distinction from the network weights, which represent the long-term memory of the network.

The memory PE has a biological interpretation (Figure 10-5). When the biological neuron receives multiple connections from the same axon at different levels of its dendritic tree, the signal propagation toward the soma produces different time delays.

Figure 10-5 Similarity between biological neuron and the memory PE feeding a M-P PE

The cell responds to the average field, so the output is a weighted sum of delayed versions of the original excitation. Although these aspects are not fully understood, the dendritic tree works as a signal representation stage. If we represent the cell by a McCulloch-Pitts PE and the dendritic tree by a single-input memory PE, the combination of the two PEs by means of a set of weights creates a processing block that incorporates the same basic principles of signal representation and nonlinear response. Figure 10-5 shows the similarity of the biological neuron with the memory PE feeding an M-P PE.

10.4.1 The Delay-Line PE

When the memory PE is built from a delay line, we call it a delay line PE, and it implements memory by delay, that is, by simply holding past samples of the input signal. The delay line PE is the memory structure utilized in the TDNN.

The delay line PE is a multidimensional single-input, multiple-output linear system where the tap outputs are delayed by one sample from the previous tap (Figure 10-6)

Figure 10-6 The delay line PE

The delay line PE with D outputs can be represented by

(10.2)

where y(n) is the combined input to the PE. Comparing Eq. 10.2 with Eq. 10.1 we verify that g(.) is the delta function operator [image] (n-1). Note that a delay-line PE with D outputs has D-1 delays. At sample time n, the output of the delay-line PE of size D only stores samples that occurred after (and including) time (n-D), that is, D-1 samples into the past. Any input event occurring before time n-D is not represented at the output. The delay line PE with D-1 delays implements a time window of length D samples positioned at the current sample.

10.4.2 The Context PE

The first-order recurrent system of Figure 8.11 (and Eq. 8.13) can be used to implement a linear context PE for general use in neural networks. This type of memory is called memory by feedback and the memory PE is called a context PE. It is only necessary to equip it with multiple inputs and eventually a bias

(10.3)

which is depicted in Figure 10-7. Note that the combined input is multiplied by [image] for normalization purposes.

Figure 10-7 A linear context PE and its representation

We normally represent this PE as in the right diagram of Figure 10-7, where the delay is not apparent, but remember that context PEs always have a delay of one sample in the feedback loop.

NEUROSOLUTIONS EXAMPLE 10.6

Context PEs Revisited

Let us revisite the first example of this chapter, and analyze the reason why the system with a context PE at the input was able to solve the discrimination task reasonably well. Note that from now on we will replace the context PE with the integrator PE (they differ only in a normalization factor).

Attach a Megascope to the output of the recurrent PE to see why the network is capable of classifying the two signals. Although the two sine waves at the input have the same amplitude, at the output of the recurrent PE they have DIFFERENT amplitudes because the recurrent PE is a lowpass filter and it filters (attenuates the amplitude) the high frequency sine wave more. These two different amplitudes are enough for the MLP to create a decision surface between the two amplitudes and distinguish the two regimes. Notice that with this explanation we also understand the errors clearly. When the low-frequency sine wave passes close to zero the MLP is fooled and assigns those samples to the high frequency sine wave.

NeuroSolutions Example

10.4.3 Memory Depth and the Length of the Impulse Response

The signal processing function of these two types of memory is to hold information from the past. The question that we have to ask is how long and how accurate is the information kept in the memory structure, which naturally leads to the concepts of memory depth and resolution. In the case of linear memories, the memory properties are related to the characteristics of the impulse response of the memory PE. Memory by feedback and memory by delays hold the information from the past differently.

The delay line PE represents the past of the signal y(n) in a discontinuous way. For delays within the memory size D the past is represented exactly, but beyond this limit it is totally forgotten. The only way to represent events further back in time is to extend the size of the delay line which requires a modification of the topology of the PE. Figure 10-8 shows an impulse at n = 0 and the progression of time by the position of the window advancing along the time line. Abruptly at n = D +1 the response of the memory PE loses all its information from the impulse. Note that D is the length of the impulse response, which is also called the region of support of the response.

Figure 10-8 Response of the delay line PE

D is exactly the number of samples for which the output of the delay line PE "remembers" the impulse at n = 0. Let us explain this terminology. When the output of the PE is different from zero, there is evidence at the output that an event (in this case an impulse) occurred somewhere within D samples. So a D-tap delay line PE remembers D samples in the past or has a memory depth of D-1 samples.

NEUROSOLUTIONS EXAMPLE 10.7

Memory Depth of the Tap-Delay Line

Let us construct a delay line PE in NeuroSolutions and show that the effect of an impulse disappears after a number of samples given by the number of taps. This explains why in the previous example the TDNN "starts" to work abruptly when the size of the memory reaches a critical value. The tap delay line has an "all-or nothing" type of memory.

NeuroSolutions Example

Let us now study the context PE. Again, suppose that an impulse is applied at time n = 0 to two context PEs H1 and H2, given by Eq. 10.3 with [image] 2 < [image] 1 (0< [image] 2, [image] 1 <1). In Figure 10-9 the impulse response appears reversed in time, as dictated by the convolution operation. We measure the system response y(n) at time nt > 0 by multiplying the input by the reversed impulse response sample-by-sample and summing the result. The response is practically zero for H1 and nonzero for H2. You can visualize how this system responds at nt to the delta function at n = 0 by measuring the value of the impulse response at n = 0. By means of the impulse response, the information of the event occurring at n = 0 was brought forward in time to n = nt. In the first case there is a trace of a response (a memory trace), while in the second case the system simply "forgot" what happened at n = 0.

Figure 10-9 Response of two different lowpass filters to an impulse

NEUROSOLUTIONS EXAMPLE 10.8

Memory Depth of the Context PE

Let us show the memory trace in NeuroSolutions. When an impulse is presented to the memory, the output decays exponentially in time. Note that the region of support of the impulse response is infinite, but for all practical purposes it is finite, since the amplitude becomes negligible after a finite time. Also note that when you change the feedback parameter, the region of support of the memory changes. This is the beauty of these memories for neurocomputing: if the system can adapt the feedback parameter it can choose the value that minimizes the output MSE and work with the best possible memory depth WITHOUT changing the topology. In the TDNN, to change the region of support, a change in the topology is required.

NeuroSolutions Example

In the context PE the memory depth can be increased without any topological modifications by decreasing the feedback parameter [image]. This is unlike the case of the delay line PE. The recurrent system of Figure 10-7, with a single delay unit displays an impulse response that decays toward zero with a time constant [image]:

(10.4)

In reality the region of support of the impulse response is infinite. But after some time (taken normally as 4 time constants) the response becomes so small that any noise can corrupt the system output. The equivalent power under this decaying exponential and the rectangular window of N samples is obtained with a number of samples equal to half-of the time constant, N = 2/ [image].

Notice that the past samples are not preserved exactly, as in the case of the delay line PE. In the context PE the output is an addition of the present input to a weighted version of the past output (see Eq. 10.3), so the information from the past is progressively distorted. This is the reason that we sometimes talk about a memory trace, or a modified version of the input.

The memory trace in a context PE gracefully degrades in time, unlike the response of the delay line PE, where the transition from memory to forgetting is abrupt. Moreover, the memory depth of the context PE is controlled by the system parameter [image]. In an adaptive system framework, recurrent systems are advantageous, since the system can control its own memory depth by changing the parameter instead of the topology. But notice that the flexibility of this memory PE is very limited, since the system can control only the rate of decay of the memory trace.

10.4.4 Memory PE Properties

With this explanation we are ready to distill two definitions that characterize linear memory PEs. We define a linear memory PE as an (eventually multidimensional) single-input-multiple-output system whose transformation kernel g(n) is a causal and normalized impulse response:

(10.5)

We define memory depth M as the center of mass (first moment in time) of the impulse response gD(n) of the last memory tap D. That is

(10.6)

where [image] and * is the convolution operation.

Similarly, we define the memory resolution R as the number of taps per sample (or unit time). With this definition the memory depth and the resolution are coupled i.e.

This has interesting implications, since for a given number of taps D, an increase in R (resolution) implies a decrease in M (depth). In other words, the only way to increase depth and resolution at the same time is to increase D, the number of taps of the memory.

Applying these definitions to the tap delay line shows that M = D and that R = 1. For the context PE the memory depth is 1/ [image], and the resolution is [image]. memory depth of the context PE

Go to next section


 10.5 The Memory Filter

Let us examine the signal processing operation produced by attaching a single input memory PE to a sigmoidal PE through a set of weights (Figure 10-10).

Figure 10-10 Block combining the memory PE with the next sigmoidal PE

If we restrict our analysis to the subsystem created by the memory PE and the output of the linear part (net(n)) of the nonlinear PE, we can recognize a topology similar to the linear combiner. The delayed versions of the input signal x(n) created by the memory PE are then multiplied by weights wk and summed to obtain net(n). Hence, we call this arrangement a memory filter, and its function is to project the input signal x(n) into a linear manifold defined by the memory traces yi(n). The signal net(n) is the result of the projection of the input vector x(n) on the weigth vector w. Wan called this special configuration of the memory filter a FIR synapse.

The fundamental conclusion from this observation is that, besides the qualitative properties of memory depth and resolution we can use the theory and tools of filtering explained in Chapter 8 to study the effects of the memory PE on the mappings.

NEUROSOLUTIONS EXAMPLE 10.9

Interpretation of Focused TDNNs as Nonlinear Memory Filters

Let us go back to the topology of Example 10.2 and interpret what is happening in the input layer. Start with a 3-tap delay line and 2 hidden PEs. We place a 3-D probe at one of the channels of the input so that we can see the two signals in signal space. We also place a Megacope and a Scatter Plot at the preactivity of the hidden layer such that we can see the outputs of the two memory filters. The Megascope shows the projections produced by the memory filters. We will start with fixed coefficients: one of the filters will be a lowpass (all ones) and the other an highpass (1, -2, 1).

The Scatter Plot shows us exactly the set of points that the MLP is trying to separate, so this Scatter Plot shows the pattern space. We have configured the display so that it first shows one class and then the other class. The goal of the MLP training is to place discriminant functions for classification in the space created by the output of the memory filters.

Let us experiment with the number of delays between consecutive taps, [image]. Start with [image] =1. The two signal trajectories are narrow ellipses, which makes them difficult to separate visually in 3-D space. The projections in the decision space are on top of each other, so the MLP has difficulty in placing the 2 discriminant functions on this space. The classification results are poor.

Now increase [image] to 3. Notice that the signal trajectories open up in 3-D space (now the ellipses use more of the signal space, making the differences in trajectories clearer). Notice now that the projections with the lowpass and highpass filters that form the decision space are further apart, so it should be easier for the MLP to do the placement of its discriminant functions. The errors are exactly at the point that the two projections cross each other. Notice that the linear projections (implemented with the linear memory filters) do not have the power to move these trajectories to different parts of the decision space. They are both anchored at the origin.

Finally, we can let the system learn all of the parameters, including the weights of the memory filters. Do this with the same 3-tap memory and triple delay. The system found a good solution as shown by the MSE and the output. Let us now look at the weights of the memory filters. Amazingly enough, the optimal weights do not correspond to our solution of lowpass and highpass filters.

We can see clearly from the output of the filters in the scope that the solution found is to zero one of the waveforms while letting the other through. (If you want to visualize the frequency response of each filter just copy the weights to a piece of paper and go to Example 7.16 to plot the frequency response.) This can be achieved if the filters are such that they place one of the zeros at the frequency of one the sine wave. This particular solution works for this case of a single frequency input, so the solution found actually is pretty smart and minimal. We need one filter to do this, that is, one hidden layer PE, which means that a perceptron can solve this problem. Adaptive systems tend to find the most obvious solution all the time….

Substitute the input sine waves by triangular waves of the same frequencies, and redo the problem to see what solution is found.

NeuroSolutions Example

10.5.1 Memory Traces as a Projection of the Input Signal

The linear context PE is a projection operator, in many ways similar to a constrained memory filter. In fact, the output of the context PE is the inner product (convolution) of the input with the impulse response of the PE, as shown in Figure 10-11.

Figure 10-11 Dependence of the output on the parameter

The projection is onto the impulse response vector, which is equivalent to the projection onto the FIR weight vector (since the weight vector is effectively the impulse response). But unlike the linear combiner, the impulse response of the context PE is restricted to a decaying exponential. The value of the PE output y(n) becomes a function of the single feedback parameter [image]. When [image] varies, the relative position between the impulse response vector, and the input vector changes. We conclude that the feedback parameter has the ability to control how much of the past is retained at the context PE output. But note that the context PE is restricted to a lowpass filter, while the delay line PE with the weights to the next layer may implement a FIR filter of arbitrary type. There are some more subtle differences. Selecting [image] effectively changes the size of the projection space and orientation of the projection. projection with the context PE

Go to next section


 10.6 Design of the Memory Space

As we have seen in the section on memory PEs there are basically two types of memory mechanisms: memory by delay and memory by feedback. We seek to find the most general linear delay operator (special case of the ARMA model) where the memory traces yk(n) would be recursively computed from the previous memory trace yk-1(n). This memory PE is the generalized feedforward memory PE (Figure 10-12). We can show that the defining relationship for the generalized feedforward memory PE is (see Principe et al)

(10.7)

where * is the convolution operation, g(n) is a causal time function, and k is the tap index. Since this is a recursive equation we have to provide a value for g0(n). In the Z domain Eq. 10.7 reads

(10.8)

where k is the stage in the cascade and we are denoting transforms by capital letters. This relationship means that the next memory trace is constructed from the previous memory trace by convolution with the same function g(n), the memory kernel yet unspecified. Different choices of g(n) will provide different choices for the projection space axes.

Figure 10-12 The generalized feedforward structure

When we apply the input x(n) to the generalized feedforward memory PE, the tap signals yk(n) become

(10.9)

the convolution of yk-1(n) with the memory kernel. For k = 0 we have

(10.10)

where g0(n) may be specified separately. The projection [image] of the input signal is obtained by linearly weighting the tap signals according to

(10.11)

and shown in Figure 10-13.

Figure 10-13 The memory filter created by the generalized feedforward memory PE

The most obvious choice for the basis is to use directly the past samples of the input signal x(n), i.e. the kth tap signal becomes yk(n) = x(n-k). From Eq. 10.9 it is easy to show that this choice corresponds to

(10.12)

and we are back to the delta function operator used in the tap delay line (in this case g0(n) is also a delta function [image] (n)). The memory depth is strictly controlled by D, i.e. the memory traces store the past D samples of the input. The TDNN uses exactly this choice of basis.

The context PE equation in our notation is

(10.13)

where 1- [image] is the feedback parameter. It is easy to show that we can write

(10.14)

When compared with Eq. 10.9 the context PE becomes a special case of a single tap (D = 1) generalized feedforward structure with

(10.15)

and y0(n), the memory trace, becomes a processed version through convolution of the input samples.

Go to next section


 10.7 The Gamma Memory PE

The gamma memory PE is a special case of the generalized feedforward memory PE where

(10.16)

and g0(n) = [image] (n). Figure 10-14 shows the block diagram for the gamma memory PE. The gamma memory is basically a cascade of lowpass filters with the same time constant 1- [image]. As can be seen from Figure 10-14, when the number of stages D = 1, the gamma memory defaults to the context unit, and when the feedback parameter [image] = 1, the gamma memory becomes the tap delay line (deVries). As a result, the gamma memory contains as special cases the context PE, and the delay line PE used in TDNN or in the FIR synapses. Hence the gamma memory appears as a unifying structure for the most common connectionist memories.

Figure 10-14 The gamma memory PE

Figure 10-15 shows the family of impulse responses from the input to tap k in the gamma memory PE, which is mathematically given by

(10.17)

These functions are discrete versions of the integrands of the gamma function, hence the name given to the memory PE. They form a complete basis in L2 space (i.e., we can approximate a finite energy signal arbitrarily closely as a weighted sum of these functions). In the context of memory structures, they can be interpreted as impulse responses of the memory PE from the input to the different taps. These impulse responses peak at k/ [image] when k >0.

Figure 10-15 Impulse response of the gamma kernel from k = 1 to k = 4 for [image] = 0.8

In the frequency domain Eq. 10.17 can be alternatively written

(10.18)

Alternatively, the tap signals of the gamma memory can be interpreted as lowpass filtered versions (memory traces) of the input. The filters are all equal, with a pole at zp = 1 [image] [image]. The transfer function from the input to the kth tap thus has a pole of multiplicity k. This means that the memory traces for deeper taps correspond to progressively filtered versions of the input. Signals are therefore progressively more delayed but also more filtered.

An important property of this family is that the time axis is scaled by the parameter [image], which means that there is a change in time scale from the input to the memory traces (uniform time warping). When this parameter is adapted with the information of the output MSE, the neural system can choose the best time scale to represent the input signal information. The memory depth and resolution for the gamma memory are respectively

(10.19)

The gamma memory has the powerful property of decoupling the memory depth M from the order of the memory D. This is a very important property that is not shared by the delay line. Just consider that a given application requires a memory of 100 samples, but three free parameters are sufficient to model the memory traces. A tap delay line will need 100 taps, and equivalently 100 parameters. This is inefficient, because we know only three are sufficient. During adaptation in a perfect world, the system must discover this and set 97 weights to zero. In reality noise may make all the parameters different from zero which will produce poor performance.

The context PE is able to decouple the memory depth from the free parameters of the system, but it can only model one of the modes of the input since it has a single parameter (projection to a one dimensional space). The gamma PE produces a D-dimensional space to represent the input, which is much more powerful. The gamma PE is more versatile than the tap delay line PE because there is an extra parameter in the representation that controls the time axis scale. This means that the memory PE can represent in D taps N samples into the past (N >D ), that is, the dimensionality of the input space is reduced when the gamma kernel is used to build the memory filter. Hence the gamma PE normally leads to a much more parsimonious representation of time information.

It is important to remember that recursive memories may become unstable. The stability of the gamma memory PE is guaranteed by limiting the value of the feedback parameter [image] to values 0< [image] < 2.

10.7.1 The Gamma Filter

The linear topology weighting the taps of the gamma memory creates a very interesting filter called the gamma filter (Figure 10-16, Principe). The gamma filter has a feedforward topology, but it is an IIR filter with a multiple pole z = 1- [image]. The gamma filter is very similar to the linear combiner in the sense that it creates a projection space of size equal to the dimensionality of the filter and whose output is the linear projection on the weight vector (which exists in the projection space). However, the gamma filter has several important advantages.

Figure 10-16 The gamma filter

The bases are no longer the input signal and its delayed versions, but are the convolution of the input with the gamma bases. These bases decouple the memory depth from the memory resolution, unlike the linear combiner. This has potential importance in system identification for real-world systems, which tend to have very long (exponentially decaying) impulse responses. In such cases the gamma filter is able to provide a smaller fitting error than the linear combiner for a given filter size.

Moreover, with the gamma filter the output MSE becomes a function of the [image] parameter, that is, under the framework of adaptation, the gamma filter has the ability to choose the best compromise between memory depth and resolution for the task at hand. It does this without any topological modification (as would be required with the tap delay line) simply by changing the feedback parameter. Unfortunately, the performance surface is nonconvex so the search can be caught in local minima. The adaptation of the gamma filter feedback parameter will be explained in the next Chapter. gamma as an extension of Wiener filters

NEUROSOLUTIONS EXAMPLE 10.10

Gamma Filter for System Identification

This example presents the system identification problem solved in Chapter 9 with the linear combiner. As we have shown the delay line is a special case of the gamma filter when [image] [image] = 1, so if we start with [image] [image] = 1 we are mimicking the linear combiner. This is an important case, because we can see what we gain by using the gamma memory instead of the more conventional delay line. The gamma memory PE is shown in the figure: remember that now it has one parameter which is the feedback parameter ( [image]).

Run the example with [image] [image] = 1 and write down the error. Now enter different values of [image] = 0.8, 0.6, 0.4, 0.2 and see how the MSE changes. You will see that for this problem [image] [image] = 0.4 provides the smallest error. The beauty of the gamma memory is that it allows the same topology to provide different projection spans, which is impossible to do with the tap delay line. For the gamma memory the bases are a convolution of the input with the kernel g(n), while for the tap delay line they are simply the delayed versions of the input. Notice that for this problem the difference in terms of MSE is appreciable.

How can we decrease the error with the linear combiner? The solution is to increase the size of the tap delay line (the filter order). For this problem the required memory depth is 3/ 0.4 ~ 7 samples (Eq. 10.18), so let us try a seventh order FIR and see the results. As we can expect, this linear combiner works better, since it has longer memory depth (and more coefficients which is not always better under noisy conditions). But the issue is that we have to guess a priori the size of the memory. Try a sixth order FIR and see that the results are much worse than the seventh order FIR.

NeuroSolutions Example

Go to next section


 10.8 Time Lagged Feedforward Networks

A time lagged feedforward network (TLFN) is a feedforward arrangement of memory PEs and nonlinear PE (the TDNN is an example of a TLFN). The short-term memory in TLFNs can be of any type and distributed in any layer. The advantage of TLFNs is that they share some of the nice properties of feedforward neural networks (such as trivial stability), but they can capture the information present in the input time signals. In this section we study a special case of TLFNs called focused TLFNs.

10.8.1 Focused TLFNs

In the focused TLFN the memory PEs are restricted to the input layer (Figure 10-17). The overall input-output mapping created by a focused TLFN is decomposed in two stages: the memory PE layer that is a linear time-representation stage, and a nonlinear mapper between the representation layer and the decision (output) space, which is static but nonlinear.

Figure 10-17 Single input focused TLFN

Let us analyze the function of this network when all the elements are trained to minimize the output MSE (Figure 10-18).

Figure 10-18 Projection of the input u(n) and the nonlinear mapping produced by the MLP

The representation stage finds the best projection of the input signal onto an intermediate projection space. This search is directed by the backpropagated error which is equivalent to an implicit desired signal at the output of the representation stage. We can utilize the Kolmogorov interpretation to say that, with the optimal weights, the output of the representation stage is the orthogonal projection of the input signal onto the projection space. The representation stage is exploring in an optimal way (through linear filtering) the spectral differences embedded in the time series patterns.

In TLFN architectures the mapping stage is a feedforward neural network such as the MLP (or RBF). The MLP inputs are the outputs of the memory filters. Hence, the space where the nonlinear mapping takes place has bases created by the hidden PE outputs that can be interpreted as the outputs of the memory filters. The number of bases is given by the number of memory filters.

The MLP places its discriminant functions in the projection space to achieve the desired processing goal. Normally, the goal is the minimization of an error in the decision (output) space. Note, however, that the weights of both the representation and mapping layers are trained together, so the user has no direct control on the projection space.

In this topology the dimension of the projection space (the number of memory filters) and the number of input PEs of the MLP are normally coupled together. There is no reason to couple the size of the space to the number of hidden PEs as we saw in Chapter 3. We can decouple the number of axes from the number of hidden PEs by explicitly including in the topology the output of the memory filters as an extra layer of linear PEs, as we show in the following example.

NEUROSOLUTIONS EXAMPLE 10.11

Increasing Versatility for TLFN Architectures

We can separate the number of FIRs from the number of hidden PEs by explicitly including a layer of linear Axons and connecting them to the MLP with a Synapse (with fixed weights of 1).

With this arrangement we can show and control the effect of each part (representation or decision) in the mapping. The size of the decision space is controlled by the number of memory filters, while the number of discriminant functions is controlled by the number of PEs in the hidden layer.

Remember that the Scatter Plot shows us the pattern space. This is where the MLP is going to draw its decision surface. But the difficulty of the task is that the projections are also changing, since the weights of the memory filters are being adapted. Overall, the system tries to minimize the number of errors in the classification and move the projections so that they are easy to classify with linear discriminant functions. Notice, however, that all the "action" occurs around the origin. So the order of the filter counts as well as the number of PEs.

NeuroSolutions Example

10.8.2 Focused TLFNs as Function Approximators

In Chapter 5 we developed the idea that neural networks are function approximators. At that time we mentioned only the special case of the linear regressor and the classifier. In Chapter 9 we presented this same interpretation for the adaptive linear combiner. By separating the function of the delay line and the regressor, we showed that the linear combiner does function approximation in the signal space, and in this regard it is equivalent to the linear regressor.

Here we revisit the same topic for focused TLFNs. We submit that the previous perspective of dividing the function of focused TLFNs into signal representation and nonlinear mapping is very general and very productive. Applying this interpretation to the focused TDNN, we can immediately conclude that the focused TDNN is a universal function approximator in the signal space. In fact, the delay line produces a representation-preserving mapping onto the signal space (an embedding provided that the size of the delay line is large enough – see Takens Embedding), and the MLP is able to approximate arbitrarily complex functions in the newly created signal space. Sandberg proved this statement mathematically. The demonstration basically states that the focused TDNN can approximate arbitrary time-invariant mappings that have a finite region of support. Other TLFNs share the same property of the TDNN, as we discuss below.

The arbitrary mapping capability of the focused TDNN is very important because it makes this neural topology highly suited for function approximation involving time signals, extending to time the universal approximation results of the MLP and RBFs in vector spaces. As we discussed in Chapter 9, function approximation in time is what engineers call system identification. Therefore we can understand the reason why the focused TDNN outperformed the linear combiner for system identification and time series prediction. nonlinear system identification with neural networks

10.8.3 Focused TLFNs Built from Context PEs

When the memory of focused TLFNs is a delay line, we obtain the focused TDNN. We already discussed the focused TDNN, and we have seen the power of such networks. But we saw that the tap-delay line was not the only type of memory PE. Let us examine the topology of Figure 10-19.

Figure 10-19 MLP with an input layer of context PEs (focused architecture)

The topology has an input layer of context PEs followed by an MLP. The first layer of input PEs can be also thought of as a representational layer for time information. The context PEs remember the past values of the input indirectly because their output is a weighted average of the present and past inputs. The hidden-layer PEs nonlinearly combine this information to produce the desired output. This topology is also a focused topology and implements a static nonlinear function of the present and past of the input.

The focused topology of Figure 10-19 is a recurrent neural network, but notice that the recurrency is local to the PE. One of the advantages of locally recurrent neural networks is that we can judge the stability of the system by constraining the value of the local feedback parameters so that the local PE is stable. If local stability is enforced, the global system will be stable.

The goal of the following examples is to solve the same three classes of problems that were presented for the focused TDNN with the focused TLFN built from context PEs. We show that the TLFN produces acceptable results, although the context memory structure is less versatile than the linear combiner.

NEUROSOLUTIONS EXAMPLE 10.12

Phoneme Classification with Focused TLFNs (Context PEs)

This example solves the phoneme classification problem of Example 10.3 with a focused TLFN built from context PEs. Run the network and verify the performance. Change the context PE parameter and observe its effect on performance.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 10.13

Nonlinear System Identification with Focused TLFNs (Context PEs)

This example is the system identification problem solved in Example 10.4 with the TDNN. Here we apply the focused TLFN to solve the problem. If we try the straight architecture with the default time constant of 0.9, we find that it fails. The reason is that for this system identification case we have a single input, and so the context PE is limited to lowpass filtering this input with a single time constant. The MLP is unable to extract the necessary information to find the mapping.

We can help the system if we provide several views of the input with different time constants, which can be easily done with a layer of context PEs with different feedback parameters. To implement this modification go to the Matrix Editor on top of the context PE and enter values between 0 and 1 for each PE. Observe that, in fact, the focused TLFN now works as well as the TDNN does. The lesson is that creating a space where signals differ in their frequency content is very useful for function approximation. We saw in Chapter 5 that having more bases helps the reconstruction. The same is true here. A single context PE gives you a single view of the trajectory. Many context PEs provide a multispectral view of the same trajectory, which captures much more information. This reasoning will be expanded later for designing the memory space.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 10.14

Nonlinear prediction with focused TLFNs (context PEs)

This example is the prediction of the Mackey-Glass signal using a focused TLFN built from context PEs. As in the previous example, we have a single input, so we need to create a projection space to represent signals in time. We can do this by creating a layer of context PEs with different time constants.

Run the example and verify that for this problem the TDNN works slightly better. The context PEs are lowpass filters, so they smooth out the sharp discontinuities of the input signal. The approximation suffers near the sharp transitions (peaks and troughs). We can expect this from the discussion on memory depth and memory resolution. The context PE increases one at the expense of the other. This compromise is nevertheless better than the delay line PE, where there is no compromise (the delay line always uses the highest resolution).

NeuroSolutions Example

10.8.4 Focused TLFNs with Gamma Memories

The gamma memory can be used as the first layer of the focused architecture, yielding another type of focused TLFN that has been called the focused gamma neural network. Sandberg also showed that the focused gamma network is a universal function approximatorJust as the gamma filter has advantages over the linear combiner, the gamma network has several advantages over the focused TDNN. Experience has shown the gamma network is able to provide a smaller output error than a focused TDNN with the same number of parameters in identification of plants with long region of support (as is the case for most real plants). Another interesting characteristic of the gamma neural network is that it has the capability to produce a uniform warping of the time axis.

The following examples yield a one-to-one comparison of the TDNN with the gamma TLFN. Since the gamma memory for [image] [image] =1 defaults to a tap delay line, we can immediately see in the same breadboard the change in performance between the two types of memory.

NEUROSOLUTIONS EXAMPLE 10.15

Phoneme Classification with a Focused TLFN (Gamma Memory)

This example deals with the phoneme classification. We have replaced the tap delay line with a gamma memory. Remember that [image] [image] = 1 defaults to the tap delay line, so we can obtain the performance of the TDNN. Run the network which is set with [image] [image] = 0.5 for the value of the gamma memory. Notice that it performs very well. This is a great example to see how the network is assigning internal resources, and when one output is correct, the others modify themselves to "grab" their own desired response.

Modify the gamma parameter to 1, and see that now the system has more difficulty solving the task and trains to a larger error. This tells us that longer memory depth is important for classifying phonemes, more important than high resolution (which is provided by [image] [image] = 1).

Just to see how well the network can solve this problem when optimized, let us substitute the output PE by a softmax nonlinearity. You have to decrease the step size to 0.05 and the momentum to 0.5. Run the network and see how perfect the classification is (remember that this is the training set, so this fact tells little about the overall accuracy in the task). The Softmax cleans the outputs because it guarantees that the sum of outputs has to be 1(competition).

In classification applications, the Softmax should be the component of choice.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 10.16

Nonlinear System Identification with a Focused TLFN (Gamma Memory)

The system identification example is also a good test of the gamma TLFN. Most real-world plants have long impulse responses since they are typically IIR. We know that the TDNN has a memory that is FIR (the tap delay line), so for adequate modeling long memories are necessary. But long tap delay lines mean many parameters, which make the TDNN overparameterized and reduces generalization. The answer is to use generalized feedforward memories such as the gamma. The gamma memory, because it is globally FIR but has an IIR kernel, can choose the appropriate memory depth with the selection of a single parameter. For the gamma D = K/ [image], so for a fixed number of taps (K) we can increase D, the memory depth by decreasing [image]. Let us see how this helps for the nonlinear plant.

Start with [image] [image] =1 and write down the final MSE. Then enter [image] [image] =0.7 and let the system adapt. Note that the error now is smaller and the waveform is smoother. Experiment with more taps and find that the improvement as a function of [image] is not as apparent.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 10.17

Nonlinear Prediction with a Focused TLFN (Gamma Memory)

Finally, we show an example where the gamma memory does not perform better than the TDNN. Let us do nonlinear prediction on the Mackey-Glass time series. The waveform, if you recall, has a great deal of detail (high frequencies) mainly at the extrema (peaks and troughs). The gamma basis with small [image] emphasizes depth at the expense of resolution. The MG30 does not require a long depth, but it needs the detail to follow the high frequencies in the waveform.

Let us start with a tap delay line of 5 and [image] [image] =0.7. Write down the error, and repeat the experiment with [image] [image] =1. You will see that the error is smaller for the TDNN.

This does not necessarily mean that the gamma memory is worse for predicting the MG30. Suppose that we reduce the memory to K= 3 taps. Now with [image] [image] =1 the prediction is poor, since there is not enough depth. Now if we select [image] [image] =0.5, the error improves. Even for this application, depth is important for discovering the correct mapping.

One of the issues that you may have noticed is that we have to manually enter the value of [image], and there is no guarantee that it is the best value for minimizing the MSE. In the next chapter we present a method to adapt [image] based on gradient descent using the output MSE. In such cases the gamma memory always provides the best compromise depth resolution.

NeuroSolutions Example

There are several other delay operators that can be defined to create TLFNs, each exploring differently the information contained in the input signal spectrum. Effectively each delay operator establishes a different basis to project the time signal. more versatile memories

10.8.5 How to Adapt the Focused TLFN

How can focused TLFNs be trained? We saw that focused TDNNs can be trained with static backpropagation provided a desired signal exists at each time step. But notice that the TDNN does not have adjustable parameters in the delay line, while other focused TLFNs, such as the one in Figure 10-14 have one parameter in the memory structures. If we preselect the value of the feedback parameter [image], static backpropagation can still be used, because the adaptive part of the network is static. Even if the feedback parameter is changed manually or automatically during operation, this effect is equivalent to a change at the input, so the gradient computation for the feedforward network weights is still static. This is one of the advantages of the focused topologies and the reason for the name.

A word of caution is in order at this moment. The feedback parameter of these memories cannot be adapted correctly by the backpropagation algorithm discussed in Chapter 3. Moreover, if the recurrent connections occurred in the hidden layers, even if the recurrent parameter was kept constant, static backpropagation could not be used to train the weights between the input and the location of the recurrent PEs. The gradients would become time dependent, and the backpropagation algorithm covered in Chapter 3 is not prepared to handle them. We will address this issue in the next chapter. For all these reasons you see the special niche occupied by the focused TLFN.

Go to next section


 10.9 Focused TLFNs Built from RBFs

Radial basis functions (RBFs) are the other alternative for creating universal mapping networks. RBFs were explained in Chapter 5, and they can substitute for the MLPs in focused TLFNs. This type of focused TLFN is built from a memory-representation layer followed by a RBF, as shown in Figure 10-20.

Figure 10-20 A focused TLFN built from a memory layer and an RBF network

The idea of the topology is the same as that of the TLFNs built with MLPs. The representation layer produces the embedding from the time series to the signal space. In the signal space the RBFs find the mapping to the desired response. Note, however, that the RBFs work directly in the signal space when the tap delay line is used as the memory layer (there are no weights between the input and the RBF PEs). The gamma memory has here an advantage because it still provides a projection on the gamma basis. The RBF centers and variances should be adapted as discussed in Chapter 5.

One potential advantage of the RBFs is that once the centers and variances are established the adaptation is linear in the parameters, so RBF normally train much faster than MLPs. However, one potential disadvantage is that we may need many RBFs when the signal space is large. The gamma memory is particularly well suited as the memory for RBFs because it tends to decrease the size of the reconstruction space. For system identification many researchers do not adapt the centers and variances. Since the signal trajectory in general visits the full space (if the embedding dimension is correctly specified), placing the centers randomly or in a grid is commonly preferred. We are going to exemplify the performance of TLFNs built from RBFs in the same three problems we encountered before.

NEUROSOLUTIONS EXAMPLE 10.18

Phoneme Recognition with a Focused RBF-TLFN (Gamma Memory)

The first problem is temporal pattern recognition. We now have the memory structure feeding a RBF layer. Notice that the weights of this layer are used solely to establish the centers of the RBF network. Hence they are going to be trained in the unsupervised mode using competitive learning. Once the centers are established and the variances estimated, the second phase of training is to find the weights that best meet the class specifications.

Train the network and see that it does a good job of classifying the three phonemes. Modify the number of RBFs and the number of delays to see the impact on accuracy. RBFs must cover the representation space well, otherwise performance will be poor. Unlike the sigmoids, the RBF Gaussians are local, so they just "see" a small portion of the signal space. Hence we need many centers (here 100), and it is important to place them in areas that have data. Notice that with a delay line of 5 taps for 11 LPC coefficients we have a 55 dimensional space which is very large. This is why the gamma memory is very useful for these networks; it can cover a given memory depth with many fewer taps, decreasing the size of the signal space. Include a gamma memory instead of the delay line and see that the performance is normally better.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 10.19

Nonlinear System Identification with a Focused RBF-TLFN (Gamma Memory)

In this example we solve the system identification problem for the nonlinear system of the Example 10.4. We start with the gamma memory instead of the tap delay line to decrease the size of the signal space.

The network is basically the same as before, except that now it is a single-input, single-output system. We again have the two phases of learning, first unsupervised to place the centers of the RBFs and to estimate the variances, followed by the adaptation of the output weights. Run the network and observe that it works as well as or better than the TDNN case, with much faster training. Now you understand why this topology is attractive for real-time applications. You can compare the performance of the memory kernel (gamma versus tap delay line) by selecting [image] [image] = 1 (tap delay line).

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 10.20

Nonlinear Prediction with a Focused RBF-TLFN (Gamma Memory)

This example implements a nonlinear predictor for the MG30 chaotic time series. The network is basically the same, here with 25 RBFs. Notice that the time series is well modeled, even with a few RBFs, but notice that the extremes of the waves lack detail. This is the price we pay for using the gamma memory, since we are trading detail (resolution) for depth. In this case the fine detail of the time series requires more resolution than depth.

NeuroSolutions Example

Go to next section


1.10 The Linear Regression Model

  1. 10 The Linear Regression Model

We started this chapter by pointing out the advantages of building models from experimental data. In the previous sections we developed a set of techniques that adapt the parameters of a linear system (the Adaline) to fit the relationship between the input (x) and the desired data (d) as well as possible. This is our first model and it explains the relationship f(x, d) as a hyperplane that minimizes the square distance of the residuals. We will have the opportunity to study other (nonlinear) models in later chapters.

It is instructive to stop and ask the question: How can we use the newly developed regression model? One interesting aspect of model building that we mentioned previously is the ability to predict the behavior of the experimental system. Basically, what this means is that once the Adaline is trained, we can forecast the value of d when x is available. We do this by computing the system output y and assume that the error [image] is small (Eq.1.3). You can now understand why we want to minimize the square of the error, since if the square of the error is small, d is going to be close to y in the training data. Figure 1-19 shows a productive way of looking at the input-output pairs that we used to train the Adaline.

Figure 1-19 A view of the desired response as the output of an unknown model

We assume that the experimental system produces the desired response d for each input x according to a rule that we do not know. The purpose of building the model is to approximate as well as possible this hidden relationship.

We expect also that even for x values that the system did not use for training, y is going to be close to the corresponding unknown value d. Our intuition tells us that if:

  • the data used for training covered all the possible cases well,

    • we had enough training data, and

  • the correlation coefficient is close to 1,

then in fact y should be close to the unknown value d. However, this is an inductive principle, which has no guarantee of being true. The ability to extrapolate the good performance from the training set to the test set is called generalization. Generalization is a central issue in the adaptive systems approach since it is the only guarantee that the model will perform well with the future data that will be presented to the system while in operation.

Remember that in the test mode the system parameters must be kept constant, that is, the learning algorithm MUST be disabled. In the next section we familiarize ourselves with training and using the linear model.

  1. 10.1 Regression Project

Getting Real-World Data

We will end Chapter 1 by giving you a flavor of the power of linear regression to solve real-life problems. We will go to the World Wide Web and seek real data sets, import them into NeuroSolutions, and solve regression problems. We will use the breadboard from Example 1.7.

The first thing is to decide on the data with which we will work. There are many interesting Web sites to visit in the search for data. We suggest the following sites:

http://ferret.wrc.noaa.gov/fbin/climate_server

http://seamonkey.ed.asu.edu/~behrens/teach/WWW_data.html

These sites have plenty of data (some duplicated). We assume that you know how to connect to the Web and how to download data. You should get the data in ASCII and store it in column format with one of the variables (the independent variable) in the input file and the dependent variable in the output file. Alternatively, we have provided sample data on the CD-ROM under the Data directory. Read the readme file to choose the data sets that interest you.

NeuroSolutions Project

The fundamental question is to find out how well a linear relation explains the dependence between the input data and the desired data. We will exemplify the project with a one-dimensional set of input data, but the multidimensional case is similar.

The first thing to do is to modify the NeuroSolutions breadboard so that it will be able to work with the data you downloaded. The data should be stored in an ASCII file and formatted in columns. Right-click on the input file icon and select properties. The Inspector will appear on the screen. Remove the present file (click the remove button) and click on the add button. The Windows file Inspector will appear; open the file that contains the input data, that is, the input to your linear model.

In NeuroSolutions, the associate panel appears, which you can close (we assume that the input file has ASCII data in column format). The next panel that pops open is the customize panel. Here you select the columns that you want to use (for those columns that you do not want, select the column label and click the skip button), and then click the close button. The input file is now open and ready to be used by NeuroSolutions. You should repeat the procedure for the desired data file. Make sure that the number of samples in the input and desired data files are the same.

Another thing that we should do is to normalize the data. Sometimes the input and desired variables have very different ranges so one should always normalize both the desired and input data files between 0 (or -1) and 1. To do this, go to the stream page (click on the stream tab) of the Inspector to access the normalization panel. Click the normalize check box and set the normalization.

We always recommend that you visually check the data - either with a plotting program or the Scatter plot in NeuroSolutions to ensure that there aren't any outliers present in the data. When outliers exist, they may distort any possible linear relationship that may exist.

Once the data sets are open, we can effectively start the adaptation of the linear regressor. The first important consideration is the largest step size that can be used for convergence. When the data is normalized we can always guess an initial value of 0.1. By plotting the learning curve or the weight tracks (if the problem has few input channels), we can judge how appropriate this value might be. Alternatively, we can compute the eigenvalues and find the exact largest possible step size, but this is rarely done. The trial-and-error method is OK for small problems.

If the problem takes a long time to converge and increasing the step size creates instability, then the eigenvalue spread is large, and there is little we can do short of using Newton's method.

After the algorithm converges (the error stabilizes), we should use the correlation coefficient DLL to estimate the correlation coefficient. Note that it is always possible to pass an hyperplane through some data points, but the real issue is whether the hyperplane provides a good model. To answer this question one needs to estimate the correlation coefficient.

For the multiple-variable regression case the relative weight magnitude tells us about the relative importance of the each variable in the regression equation (for normalized inputs). It is therefore interesting to look at the values of the regression weights, including the bias. Remember that if the data is normalized, the displayed weight values must be "unnormalized" in order to compare them with the original data. You can find the values that NeuroSolutions used to normalize the data by going to the dataset page of the Inspector and opening the normalization file. NeuroSolutions multiplies the data set by the first value in the normalization file (range) and adds the second value in the normalization file (offset). To reverse this process, you must subtract the second value and then divide by the first value.

Remember that the parameters of the regression equation can be used to predict the system output when only the input is known. We can do this by testing the system with another data file for which we do not have a desired response. To do this in NeuroSolutions, you should go to the Controller Inspector (the yellow dial) and turn off the learning check box; this fixes the weights.

No problem is finalized without a critical assessment of the results obtained. You should start with a hypothesis about the data relationship and confirm your hypothesis with NeuroSolutions results. If there is a discrepancy between what you expect and the results, you must explain it. This is where the NeuroSolutions probes are very effective. You should verify that the data is being properly read, whether the input and output files are synchronized, whether the system is converging (weight tracks, learning curve), and so on. Computers are great tools, but they are very susceptible to the garbage in, garbage out syndrome, so it is the user's responsibility to check the input and the methodology of data analysis.

NEUROSOLUTIONS EXAMPLE 1.21

Linear Regression Project

We will illustrate the project with a regression between two time series; the sea temperature and atmospheric pressure downloaded from the NOAA site (climate data base). We will start with the breadboard from Example 1.7. We replace the input with the file containing the sea temperature and replace the desired response data with the file containing the pressure data. NeuroSolutions automatically sets the number of inputs from the file (verify this in the File Inspector), and the number of exemplars in an epoch (verify this in the Controller Inspector). We also have to decide how many iterations we need. In the Controller Inspector, enter 1,000 in the Epochs/Run field. This number may be too large, but when the weights or the MSE stabilize, we can always interrupt the simulation. Experiment with everything we have learned in this chapter.

NeuroSolutions Example

Go to next section


1.11 Conclusions

  1. 11 Conclusions

In this first chapter we introduced ideas that are very important for the rest of the book. Probably the most important was the concept of adaptive systems. Instead of designing the system through specifications, we let the system learn from the input data. To achieve this, the system has to be augmented with an external cost criterion that measures how well the model fits the data and with an algorithm that will adapt the system parameters so that the minimum of the cost can be reached. We will use this concept throughout this book.

We covered much more in this chapter. We described an extremely simple and elegant algorithm that is able to minimize the external cost function by using local information available to the system parameters. The principle is to search the performance surface in the opposite direction of the gradient. The name of the algorithm is least means squares (LMS) and in just two multiplications per weight and data sample it is able to move the system parameters towards the neighborhood of the optimal values. Gradient descent is a powerful concept that we will also use throughout this book.

When we apply the LMS to the linear network, we end up with a system called the linear regressor or Adaline that can fit hyperplanes to data. The solution is identical to least squares.

We quantified the properties of the LMS algorithm, and we showed the fundamental trade-off of adaptation: the compromise between speed of adaptation and precision in the final solution. We defined the learning curve, which we called the thermometer of learning, that we will also use over and over. Therefore this chapter covers the basic concepts for the intriguing adventure of designing systems that learn directly from data.

We have also provided a project to help you understand the power of adaptive systems. The applications of the Adaline are bounded by our imagination and the data we can find to train it. Thus, knowing how to get data from the Web and how to use it in NeuroSolutions is of great value.

Go to next section


 11.1 Introduction

In the previous chapter we create time lagged feedfoward networks (TLFN) that processed information over time and were easy to train. Basically, they could implement static (but arbitrary) mappings from the present input and its memory traces to the desired response. There is often a need to extend the network capabilities to time-dependent nonlinear mappings. This means that short-term memory mechanisms have to be brought inside the feedforward network topologies (TLFNs) or that the networks have to be made spatially recurrent, that is, recurrent connections must be created among some or all PEs. We call these spatially recurrent networks simply recurrent networks.

The complexity of these two solutions is very different. TLFNs have locally recurrent connections and can be made stable by enforcing the stability of the short-term memory mechanisms, while it is much more difficult to guarantee stability of recurrent networks. Moreover, TLFNs are easier to train than recurrent systems, so they are more practical. Last we can still interpret how a TLFN is processing the information by combining our knowledge of MLPs with adaptive filters, while the complex interconnectivity of the recurrent system usually inhibits our ability to study the system.

You should have noticed how carefully we picked the focused TLFN topology so that we could still use the static backpropagation algorithm. This is no longer possible for distributed TLFNs nor for recurrent networks. One of the central issues that we have to address in this chapter is how to train recurrent networks. We start by extending static backpropagation to adapt systems with delays, that is, systems where the ordered list does depend not only upon the topology as in Chapter 3, but also on the time order. This concept gives rise to the back-propagation-through-time (BPTT) algorithm, which trains recurrent networks with a segment of a time series. Learning a segment of a time series is called trajectory learning. This is the most general case for learning in time. BPTT is applied to train gamma networks, distributed TLFNs, and fully recurrent networks. We also study how recurrent systems are trained to memorize static patterns by extending static backpropagation to what has been called fixed point learning.

We also superficially study dynamic systems in terms of their fundamental definitions and topologies. An example of the insight gained with dynamics is Hopfield's interpretation of the computational energy of a recurrent neural system. We cover this view and see how it can be used to interpret a recurrent system with attractors as a pattern associator. We end the chapter (and the book) with a description of the Freeman's model, which is a new class of information processing system that is locally stable but globally chaotic.

Throughout the Chapter we provide applications of time-lagged feedforward networks ranging from nonlinear system identification, nonlinear prediction, temporal pattern recognition, sequence recognition, and controls.

Go to next section


1.12 Exercises

  1. 12 Exercises

    1. 1 (a) Compute by hand the linear regressor for the following data:

X= {-0.5, -0.2, -0.1, 0.3,0.4, 0.5, 0.7}

D= {-1, 1, 2, 3.2, 3.5, 5, 6}

(b) Compute the correlation coefficient.

(c) Estimate D for the values x=0 and x=1 . Which d can you trust the most? Justify your answer.

(d) Use NeuroSolutions to confirm your results.

  1. 2 For the data of Problem 1.1, compute the performance surface and plot it. Estimate the gradient at the point w =2.

  1. 3 (a) Compute the linear regression for the following data:

X1= {-0.5, -0.2, -0.1, 0.3,0.4, 0.5, 0.7}

X2= {3, 3, 2.5, 2, -1, -1, -4}

D= {-3, -1, 0, 1.2, 1.5, 3, 4}

(b) Use NeuroSolutions to confirm your results.

  1. 4 Find the eigenvalues and eigenvectors of the following matrix

  1. 5 Show that when you apply Eq. 1.5 to data that is Gaussian distributed, effectively you obtain a set of linear equations.

  1. 6 Rerun Example 1.8 with white noise that has a mean value of m =0.5 and explain the results.

  1. 7 Show mathematically that Eq. 1.14 will always give a value of |g|<1.

  1. 8 For the data set in Problem 1.3, estimate the largest step size, the critically damped step size, and the time constant of adaptation for each step size. Run NeuroSolutions and verify your results by plotting the weight tracks. Also verify the time constant of adaptation. For which value does the system adapt the fastest?

  1. 9 If you know the eigenvalues of the input autocorrelation function, would you schedule the step size for fast adaptation? Justify your answer. Run NeuroSolutions to confirm your conclusion.

  1. 10 Verify in the data of Problem 1.3 the statement of Section 1.6.5 that when the step size is very large in on-line learning the MSE in NeuroSolutions can decrease below the theoretical limit. Plot the weight tracks and comment on the validity of the model.

  1. 11 Compute the largest eigenvalue for the data of Problem 1.3, but now use a regressor with bias. Show all the work. What can you conclude about the largest step size for the bias?

  1. 12 Apply Newton's search to the data of Problem 1.3, and compare the speed of adaptation with LMS. You should also use NeuroSolutions.

  1. 13 Go to the Data directory and open the folder Auto MPG. Read the instructions and determine the regression equation to estimate the MPG of a car. Use NeuroSolutions. Make sure the system is converging. What is the estimated misadjustment for the solution? How would you obtain a better linear fit? Compute also the correlation coefficient and conclude about the applicability of your model.

  1. 14 Repeat Problem 1.13 with the data found in the folder Abalone.

  1. 15 Construct the Breadboard for Problem 1.13 from scratch, i.e. one component at a time.

Go to next section


11.2 Simple Recurrent Topolgies

 11.2 Simple Recurrent Topologies

All the focused TLFNs studied in Chapter 10 implement static nonlinear mappings. Although focused TLFNs have been shown to be universal mappers, there are cases where the desired function is beyond the power of a reasonably sized, focused TLFN. We can easily imagine a sequence of input values that gives rise to two different outputs depending on the context. Either we have enough memory to span the full context, or the network will be unable to discover the mapping. Jordan and Elman proposed simple networks based on context PEs and network recurrency that are easy to train (because the feedback parameters are fixed) and accomplish the mapping with small topologies (Figure 11-1).

Figure 11-1 Jordan and Elman networks

Note that both the Jordan and Elman nets have fixed feedback parameters and there is no recurrency in the input-output path. They can be approximately trained with straight backpropagation. Elman's context layer receives input from the hidden layer, while Jordan's context layer receives input from the output.

These systems are in principle more efficient than the focused architectures for encoding temporal information since the "memory" is created by recurrent connections that span several layers, that is, memory is inside the network, so the input-output nonlinear mapping is no longer static. However, the Jordan and Elman networks are still very special architectures that were derived to simplify their training. Notice that the outputs of the context layer can be thought of as external inputs (which are controlled by the network instead of by world events) so that there is no recurrency in the input-output path. Both systems have been utilized for sequence recognition and are sometimes called sequential networks. Jordan networks can even associate the same (fixed) input with several output sequences, depending on the context.

NEUROSOLUTIONS EXAMPLE 11.1

Jordan's Network

We are going to use a data set with time dependencies. Suppose that we have 9 items coded as amplitude levels 0.1, 0.2, …. 0.9, which appear randomly in a sequence. However, when each item appears, we know that it will appear for a predetermined number of time steps. For instance, 0.1 appears for 1 time step, 0.2 for 2 time steps, and 0.9 for 9 time steps. Elman associated the random values with consonants that are followed by a predetermined number of vowel sounds.

Can a network learn the time series? The answer must be no, because each subsequence appears randomly. However, it should not be uniformly high over time, since there is a predictable part in the time series structure, namely, the length associated with each level. A network that can capture the time structure should yield a low error once a given level appears at the network input.

Our first architecture is the Jordan network, which feeds back the output of the system to a layer of context PEs. The network has a single input and a single output. The desired response is the same as the input but advanced one time step (i.e., the network is trained as a predictor). We use backpropagation to train the system, although this is an approximation as we will see shortly. We start with 5 hidden PEs and 5 context PEs with fixed feedback, but feel free to experiment with these parameters. Running the network, we observe the expected behavior in the error. The error tends to be high at the transitions between levels.

In Chapter 10 we worked with a time series built from two sine waves of different frequencies (since this is also a problem of time structure) let us see whether the Jordan network can also learn it. The answer is negative, since the feedback is from the output: so if the output is always wrong, the feedback does not provide valuable information.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 11.2

Elman's Network

We will repeat here the previous problem but now with an architecture that feeds back the state to the context PEs, that is, the hidden-layer activations are providing the input to the context PEs.

The Elman network works as well as the Jordan network for the multi-level data set, and is able to solve the two-sine wave problem of Chapter 10. Working with the past system state seems more appealing than working with the past output. Again, notice that we are using static backpropagation to train the weights of the system.

NeuroSolutions Example

We saw that one of the difficult problems in the processing of time signals is to decide the length of the time window. Normally we do not know the length of time where the information relevant to processing the current signal sample resides. If the window is too short, only part of the information is available, and the learning system is working with only partial information. If we increase the size of the window too much, we may bring in information that is not relevant (i.e., noise), which negatively affects learning. We saw in Chapter 10 that the value of the feedback parameter controls the memory depth in the context PE, so in principle, its adaptation from the data may solve our problem.

In this chapter we lift the restriction of working with constant feedback coefficients and special architectures, so we have to first understand the problem created by feedback when training neural networks.

Go to next section


 11.3 Adapting the Feedback Parameter

Let us consider the simple context PE. An appealing idea for time processing is to let the system find the memory depth that it needs to represent the past of the input signal. If we utilize the information of the output error to adapt the feedback parameter 1- [image] (which we will call [image] 1), then the system will work with the memory depth that provides the smallest MSE. This is possible in principle since the feedback parameter [image] 1 is related in a continuous way (a decaying exponential) to the PE output. To adapt the feedback parameter we will need to compute the sensitivity of the output to a change in [image] 1.

Can static backpropagation be used to adapt [image] 1? The answer is a resounding NO. The reason can be found in the time-delay operator (z [image])and in the recurrent topology. The time-delay operator creates an intrinsic ordering in the computations, since the output at time n +1 becomes dependent on the output value at time n (Figure 11-2). When we derived the backpropagation algorithm in Chapter 3 we mentioned that the algorithm would compute gradients for any ordered topology, that is, any topologies that obeyed a dependency list (see Eq. 3.34). This dependency list was static; it addressed only the dependencies created by the network topology. However, the delay also imposes a time dependency on the variables, so it will interfere with the dependency list created by the topology, which does not consider time. In practical terms, the recurrent connection makes a tremendous difference.

Let us compute the sensitivity of the output with respect to the weights [image] 1 and [image] 2 for the network depicted in Figure 11-2.

Figure 11-2 A first order recurrent system

The input-output relationship is

(11.1)

The partial derivative of y(n) with respect to [image] 2 is simply

(11.2)

The derivative of y(n) with respect to the feedback parameter [image] 1 contains two terms (derivative of the product):

(11.3)

The first term in Eq. 11.3 is similar to Eq. 11.2, but notice that y(n-1) also depends on [image] 1 because of the recursive nature of Eq. 11.1. This is a major difference between the static case and the recurrent system. Notice that Eq.11.3 basically says that the effect of any change in the parameter [image] 1 (the recurrent parameter) lasts forever, while the effect of a change in the feedforward parameter [image] 2 only matters in the current sample.

As you may recall, the backpropagation algorithm covered in Chapter 3 did not include this effect, since there were no feedback connections. The algorithm in Chapter 3 was called static backpropagation exactly for this reason. This also means that for recurrent topologies the equations need to be rederived to cope with time dependencies.

11.3.1 Error Criteria to Train Dynamic Networks

The fundamental difference between the adaptation of the weights in static and recurrent networks is that, in the latter, the local gradients depend on the time index. Moreover, the type of optimization problems are also different, because we are generally interested in quantifying the performance of adaptation within a time interval instead of instantaneously.

The most common error criterion for dynamic neural networks is trajectory learning where the cost is summed over time from an initial time n = 0 until the final time n = T,

(11.4)

where Jn is the instantaneous error and m is the index over the output PEs (we omitted the summation on the patterns for simplicity). The time T is the length of the trajectory and is related to the length of the time pattern or of the interval of interest. The cost function is therefore obtained over a time interval, and the goal is to adapt the adaptive system weights to minimize Jn over the time interval. Eq. 10.4 resembles the batch mode cost function if we relate the index n with the batch index. But here n is a time index, so we are using different samples of the time series to compute the cost.

For dynamic systems trained with fixed-point learning, the static error criterion can still be used,

(11.5)

This cost measures the performance, assuming that the output and desired signals do not vary over time (after the system relaxes to a steady state). Since the states y(n) in a dynamic network are time dependent, the sensitivity of the cost with respect to the states also becomes time dependent. The only way to implement fixed-point learning is to let the system response stabilize (arrive at a steady state) and then apply the criterion.

Go to next section


1.13 NeuroSolutions Examples

  1. 13 NeuroSolutions Examples

  1. 1 The Linear Processing Element in NeuroSolutions

  1. 2 Computing the MSE for the Linear PE

  1. 3 Finding the Minimum Error by Trial and Error

  1. 4 Plotting the Performance Surface

  1. 5 Comparison of Performance Curves for Different Data Sets

  1. 6 Adapting the Linear PE with LMS

  1. 7 Batch versus On-line Adaptation

  1. 8 Robustness of LMS to Noise

  1. 9 Estimating the Correlation Coefficient During Learning

  1. 10 The Learning Curve

  1. 11 Weight Tracks

  1. 12 Linear Regression without Bias

  1. 13 Rattling

  1. 14 Scheduling of Step Sizes

  1. 15 Multivariable Regression

  1. 16 Checking the LMS Solution with the Optimal Weights

  1. 17 Visualizing the Weight Tracks and Speed of Adaptation

  1. 18 Visualizing Weight Tracks with On-Line Learning

  1. 19 Linear Regression Without Bias

  1. 20 Newton's Method

  1. 21 Linear Regression Project

Go to next section


1.14 Concept Map for Chapter 1

  1. 14 Concept Map for Chapter 1

Go to Next Chapter


 11.4 Unfolding Recurrent Networks in Time

To apply the backpropagation procedure for recurrent networks, we need to adapt the ordered list of dependencies for recurrent topologies. As we saw in Eq. 11.1 the present value of the activation y(n) (also called the state) depends on the previous value y(n-1). The sensitivity of the present state also depends on the previous senstivity Eq. 11.3. There is a procedure called unfolding in time that produces a time-to-space mapping, replacing a recurrent network by a much larger feedforward network with repeated coefficients. As long as the network is feedfoward, we can apply the ordered list. As an example, let us take the case of the network at the top of Figure 11-3, which shows a linear dynamic PE (PE #1) followed by a static, nonlinear PE (PE #2). The goal is to adapt [image] and w1.

Figure 11-3 Unfolding in time of a simple recurrent network

The forward equation for this system is

(11.6)

Each time tick n produces an output y2(n) that is a function of the previous value of y1(n-1) combined with the present input x(n). Notice the dependence on the initial condition, normally set to zero (y1(0) = 0). Each time tick creates a stage of the unfolded network. Each additional time tick will be a similar network cascaded on the previous one (Figure 11-3). Provided that the recurrent network is in operation for a finite time, we can replace it with a static, feedforward net with T + 2 inputs and T + 1 outputs. This operation is called unfolding the network in time. From this unfolded network we can generate an ordered dependency list for the recurrent network, as we did in Chapter 3

(11.7)

since the unfolded network has 2T + 2 state variables. However, in this particular case we know that there are only two state variables at different time steps, so we can rename the state variables invoking the implicit time order

(11.8)

Note that the weights appear first in the list, since all the state variables depend on them. Next the states are ordered by their index on the topology (as we did for the MLP), but the same state variables are repeated indexed by time from time zero to T. For the unfolded network of Figure 11-3 the even states correspond to the values that y1(n) takes over time.

A procedure to establish the dependency list is to first label, from left to right, all the state variables (output of adders in the topology) along with the output to create the dependency list at n = 0. Feedback loops are simply not considered. Next we replicate the variables in the same order for all other time ticks until n = T. Therefore, in simple cases, just by looking at the network structure we can infer the dependency list quite easily; that is, we can skip the network unfolding once we understand the procedure. Now we just need to apply the backpropagation algorithm to the dependency list of Eq. 11.8.

11.4.1 Extending Static Backpropagation to Time: BPTT

We can apply the order derivative procedure explained in Chapter 3 to the dependency list of Eq. 11.7 which yields Eq. 3.35. We can instead apply the ordered derivative procedure to Eq. 11.8 which is equivalent because we just count the states differently. In this case we have a double sum going over the original network state variables (in our case y1 and y2), and also the variables created by the time unfolding (indexed by time n = 0, …,T), i.e.

(11.9)

Note that we use time [image] > n to enforce the rule that we can compute dependencies only of variables that are to the right of the present variable in the list. With this new labeling of variables, time as well as topology information, appears in the ordered derivative equation. We could still use the numbering of states in the unfolded network, but we would lose the power of interpreting the solution.

Likewise, the sensitivity with respect to all the weights ( [image] and w are treated equally) can be written

(11.10)

where we use the fact that the weights are first in the ordered list (sum over all times and states), the weights are internal, and the cost is computed over time as in Eq. 11.4.

The first important observation is that the time index [image] in Eq. 11.9 is greater than the present time n, so the gradient computation in recurrent networks using the backpropagation procedure is anticipatory. We saw that there are no physically realizable systems that are anticipatory (i.e., that respond before the input is applied). However, in digital implementations we can implement anticipatory systems during finite periods by using memory. We simply wait until the last sample of the interval of interest and then clock the samples backward in time, starting from the final time.

The second observation relates to the number of terms necessary to compute gradients over time. It is known that a recurrent system propagates any present change in one of its state variables to all future time (due to the recurrency). This indicates that, in principle, the sensitivity of one of the state variables at the present time also needs all its future time sensitivities to be computed accurately. However, notice that in Eq. 11.9 the summation term blends only the explicit or direct dependence of yj( [image]) on yi(n) in time and on the topology, so we are computing dependencies over time with explicit dependencies on the topology. For example, if we closely examine the feedforward equation Eq. 11.6 (or Figure 11-3) the present value of y1(n) depends explicitly only on the previous value y1(n-1). However the topology produces a link to y2(n). In the ordered list (Eq. 11.8) only y1(n+1) and y2(n) will depend explicitly on y1(n). This shows the advantage of the ordered list in computing sensitivity information.

With this prologue we are ready to apply Eq. 11.9 to the two state variables y2(n) and y1(n) and to the weights [image] and w1. For y2(n) we see that all the terms in the sums are zero, since PE #2 is a nonrecurrent output PE. Therefore

(11.11)

where [image] (n) is the injected error at time n. For y1(n) we get

(11.12)

Let us analyze Eq. 11.12. We have no injected error, since PE #1 is internal. The double sum of Eq. 11.9 has two terms different from zero, since y1(n) has two direct dependencies, one from the time variable y1(n+1) and the other from the topology (PE #2), y2(n). The direct dependence over time gives the first nonzero term, while the dependence over the topology gives the second term.

Now for the weight gradients we get immediately from Eq. 11.6

(11.13)

and for the feedback parameter

(11.14)

There are many implications of this derivation. Probably the most important is that by unfolding the network in time, we are still able to apply the backpropagation procedure to train the recurrent network. This new form of backpropagation is called backpropagation through time (BPTT) because the ordered list is now reversed not only in the topology (from the output to the input), but also in time (from the terminal time n = T to n = 0).

The second important observation is that BPTT is NOT local in time. This may come as a surprise, but the beautiful locality of the backpropagation algorithm on the topology does not extend to time. We can see this easily by noting that both the state gradient (Eq. 11.12) and the weight gradient (Eq. 11.14) are a function of more than one time index.

The third implication is the interpretation of the sensitivity equations as activities flowing in the dual network. Let us see what is the form of the dual PE for the linear context PE of Eq. 11.1. We re-write Eq.11.12 with our previous convention of error signals, e(n) and [image] (n) (since the PE is an internal PE in the topology, the error that reaches it is internal):

(11.15)

If we interpret the error that is backpropagated from the output as the input to the dual network, the dual of the linear recurrent PE is as shown in Figure 11-4.

Figure 11-4 Dual of the linear recurrent PE

Notice that the dual network is constructed from the original network with the same rules presented in Chapter 3, with the addition that the delay has been substituted by an advance of one sample, that is, the dual of the delay is the advance in time. If we start by running the dual system from n = T, the sign of the advance operator is reversed, so the transfer functions of the original and dual systems can be implemented with the delay operator.

In Chapter 3, Section 3.6.3, we showed how to implement static backpropagation as a data flow machine, that is, without explicitly deriving the equations for adaptation, but simply constructing the dual network and specifying the forward and the backward maps of each component. The maps for the static components are shown in section 3.6.4. We are now in position to extend the data-flow procedure to dynamic neural networks, since we discovered the dual of the delay operator, the only operator we have not encountered before. We will be using the same notation, so the equations to include the ith linear context PE of Eq. 10.3 (no bias) in our data-flow machine are

(11.116)

and the weight update is given by

You should compare these equations with Eq. 3.40 and Eq. 3.41, respectively, to see the effect of the dynamics. Note that the sum over the trajectory (n) can be computed sample by sample from the maps in Eq. 11.16. The sums over j are available at the PE since they represent the input activation and the injected error, respectively.

The data-flow algorithm of static backpropagation must be adapted to cope with the time-varying gradients. But the modifications are minor.

  • First, the data for the full trajectory (T samples) is sent through the network (one sample at a time), and the activations are stored locally at each PE.

  • Next, the set of outputs is compared with the desired trajectory, using our criterion of choice to build the error sequence.

  • This error sequence is reversed in time according to the BPTT procedure to compensate for the required advances in time, which are the dual of the delay.

  • The reordered error sequence is sent through the dual network, and the local errors are stored locally. NeuroSolutions implements this procedure using the Dynamic Controller.

  • Once the sequence of local errors and local activations exist at each PE, the weight update can take place using any of the first-order search methods described in Chapter 4.

Due to the fact that BPTT is not local in time, we have to compute the gradients of all the weights and states over a time interval, even if only part of the system is recurrent, as in the TLFN topologies found in Chapter 10. We have to prespecify the initial time and the final time. This affects the storage and computational requirements of the algorithm. Even for the case of the focused TLFN topology that has a single recurrent connection at the input, if BPTT is selected to adapt [image], ALL gradients need to be computed over time. backpropagation versus BPTT

NEUROSOLUTIONS EXAMPLE 11.3

Training the Feedback Parameter of Elman's Network

This is our first example using BPTT. The first thing to notice is that the Controllers are different: now they have three dials.

Their functionality is also different, since now they have to orchestrate the firing of the data through a trajectory and command the PEs (and the duals) to store their activities (and errors). The errors also have to be reversed in time (starting from the error at the last time step toward the beginning).

The user has one extra level of variables for BPTT, which is related to the size of the trajectory and the initial conditions. Open the Dynamic Controller Inspector and look at the dynamic page. Notice that you have three choices: static (the same as the Static Controller), fixed point and trajectory learning. We are interested in trajectory for this example. The size of the trajectory is called the number of exemplars, so when we set the number of samples per exemplar we are specifying the length of the trajectory. In the same panel we can further specify the initial conditions. If we click the box "zero state between exemplars," for each new trajectory the state of the network (i.e., the values of activations and errors) will be reset. Normally we do NOT want this, since it creates discontinuities between adjacent trajectories. Once we specify the parameters in this panel, the simulations work in exactly the same way as before.

We are now ready to use BPTT to train the feedback parameters of the Elman network for Example 11.2. We choose the size of trajectory as 15 samples. This means the Controller will fire 15 samples at a time from the input file, each PE will store 15 activations, the network output will be compared with 15 samples of the desired response, and a 15-long error vector will be created. This error vector will be reversed in time and fired through the dual network, one error at a time. At the end of the trajectory each PE and its dual will have 15-sample-long activation and error vectors, and momentum learning can then be used to update the weights. Notice that the weights are updated with the cumulative error along the trajectory. The process is then repeated for the next 15 samples of the input file, and so on.

Running the network, we see that the it learns very fast. The initial values of the feedback parameters are set to 1 (no memory), and they slowly change to the appropriate value that minimizes the output MSE. This is the beauty of the context PE. It automatically configures its memory to minimize the output MSE. Notice that the time constants spread out to cover all the time scales well. The final error is normally smaller than when the network is trained with static backpropagation using an arbitrarily chosen feedback parameter.

NeuroSolutions Example

11.4.2 Computing the Gradient Directly

As we saw in Chapter 3 for the MLP, the gradients can also be computed by applying the chain rule directly to the cost (Eq. 11.4) to obtain

(11.17)

We denote the gradient variables (sensitivity of the states) with respect to the weights as

(11.18)

To compute the state sensitivities, we use the forward equation for the neural network to yield

(11.19)

where f' is the derivative of the nonlinearity and [image] i,m is the Kronecker delta function that is 1 only when m = i. It is important to compare this equation with the one given in Chapter 3 (Eq. 3.39) to see the differences brought in by the recurrency (the sum in the parentheses).

If the weights change slowly compared with the forward dynamics, we can compute these equations for each time sample instead of waiting for the end of the trajectory:

(11.20)

This is why this procedure is called real time recurrent learning (RTRL).

It is interesting that the RTRL algorithm is local in time but not local in space, so in some sense it is the dual of BPTT. In fact, Eq. 11.17 shows that we are computing all the sensitivities with respect to every weight wij, so the computation is not local in the topology (in space), but it also clearly shows that the method is local in time. The time locality makes it appealing for on-line applications and VLSI implementations. The direct method was computationally unattractive for static networks when compared with backpropagation. It is even more so for dynamic networks and small trajectories, as we will discuss later.

Let us apply RTRL to the topology of Figure 11-3. Since there are only two weights, it is easy to see that the procedure gives

(11.21)

and

(11.22)

Notice that the formulas to compute [image] and w with RTRL are different from the ones obtained for BPTT (Eq. 11.13 and Eq. 11.14 ). However, we can show that for the same initial conditions at the end of the interval the two sets of equations provide the same updates, so they are equivalent (see Haykin). For small networks RTRL provides very compact equations, which may be beneficial for understanding dynamic training.

For instance, it is clear from Eq. 11.21 and 11.22 that the local gradients are a function of f'(.) so, to ensure stability the largest slope of the recurrent PE nonlinearity should be less than one: otherwise, we may be amplifiying the errors. If we use an [image] of 1 in the PE nonlinearity, as is normally the case, we are in the marginally stable case. This is often forgotten, and produces instability in training over time. training the gamma filter

NEUROSOLUTIONS EXAMPLE 11.4

The Effect of the Slope of the Nonlinearity in Recurrent Networks

Let us go back to the Elman network and set the slope of the hidden PEs to a large value. The training becomes very difficult, producing unusable results most of the time. Try a large value and decrease the step size to see whether you can control the learning. You will see that it is very difficult.

Another important thing to notice is that the system output seems quite normal (just flat), but the internal PEs are all pegged, that is, stuck at one extreme value or the other. Here the system is unstable, but it does not behave like an unstable linear system (the output goes to infinity) due to the nonlinearity. Recurrent systems behave very differently from static systems, so you have to be much more careful while training them.

We recommend that the slope of the nonlinearity be set at 0.8. We also recommend that you place a Megascope over the hidden PEs to see whether they are pegged all the time. If they are the PE is simply not responding, and the network effectively has fewer degrees of freedom.

NeuroSolutions Example

11.4.3 Comparing RTRL and BPTT

Computational Complexity of RTRL

RTRL computes the sensitivity of every system state with respect to all the weights. However, for trajectory learning it has the important characteristic of being local in time. When the weights change slowly with respect to the forward dynamics, the quantities in Eq. 11.17 can be computed on-line, that is, for every new time sample. This means that the state gradients [image] are computed forward in time with every sample. If the desired signal is available for every sample of the trajectory, the errors [image] m(n) are obtained, and the weight gradients [image] can also be computed for every new sample. If the desired signal is available only at the end of the trajectory, the state gradients have to be stored for the length of the trajectory, and then at the final time the weight gradients can be computed.

With this explanation we can compute the number of computational operations required and the storage requirements of the algorithm. For a fully connected N-PE system there are N² weights, N³ gradients and O(N³) operations required for the gradient computation per sample. If the length of the trajectory is T, this means O(N [image] T) overall computations. However, the number of items that need to be stored (assuming that the desired signal is available) is independent of time and is O(N³) which is the number of instantaneous state gradients.

The disproportionately large number of computations means that the method is realistic only for small networks. An interesting aspect is that RTRL can be applied in conjunction with static backpropagation in networks where the feedforward path is static, but the input is recurrent (as in the focused architectures). The recursive memory is trained dynamically and the feedforward network is trained statically. When implemented, this combination is the most efficient way to train focused TLFNs, since all the computations become local in time. training focused TLFNs

Computational Complexity of BPTT

The basic observation for applying the backpropagation formalism in the training of recurrent networks is to modify the activations of a network evolving through time into activations of an equivalent static network in which each time step corresponds to a new layer (unfolding the network in time). Static backpropagation can then be applied to this unfolded network, as we have seen (Eq. 10.9). These equations can be put in a one-to-one correspondence with the data-flow algorithm. As in the static case, there is also an intrinsic order in the computations:

  1. An input is presented and an output computed. The process is repeated for the trajectory length T, and the local activations are stored locally.

  2. Once the network output is known for the full trajectory, a sample-by-sample difference with the desired signal is computed for the full trajectory such that the error signal [image] (n) is generated. This error must be backpropagated from the end of the trajectory (n = T) to its beginning (n = 0) through the dual topology. Local errors are also stored.

  3. Once the local errors and local activations for each time sample are available, the weight update for the trajectory can be computed using the specified search methodology.

  4. The process is repeated for the next trajectory. Normally, trajectories are stacked to better teach the time relationships.

As we can see, the computational complexity of BPTT (number of operations and storage) is much larger than that of static backpropagation. The number of operations to compute one time step of BPTT in an N-PE fully connected network is O(N²), so for the trajectory of size T it becomes O(N²T). This is much better than for RTRL. The problem is the storage requirements. As we can see, the activations need to be saved forward which gives O(NT). This means that for long trajectories compared with the size of the net, the BPTT algorithm uses more storage than RTRL. The following table shows a comparison of the two algorithms

It is interesting that BPTT is local in space but not in time, while RTRL is local in time but not local across the network. NeuroSolutions implements BPTT to train recurrent neural networks for trajectory learning. The data-flow algorithm explained in Chapter 3 for static backpropagation can also implement BPTT and fixed-point learning with the appropriate control of the data flow. The local maps for the neural components are associated with the PE and are independent of the type of training strategy. This attests to the importance of the data-flow implementation of backpropagation as mentioned in Chapter 3.

Go to next section


 11.5 The Distributed TLFN Topology

The learning machines created as a feedforward connection of memory PEs and nonlinear sigmoidal PEs are called time-lagged feedforward networks (TLFNs). Figure 11-5 shows the general topology of TLFNs.

Figure 11-5 A time-lagged feedforward network

The focused architectures can produce only a static nonlinear mapping of the projected representation, but there is no need to restrict ourselves to this simple case. As shown in Figure 11-5, we can also populate the hidden layers with memory structures and then produce time-varying nonlinear maps from the projected space to the output. These TLFNs with memory throughout the topology are called distributed TLFNs. Each memory PE in the TLFN is effectively processing information over time by working with the projections of the PE activations of the previous layer on its local linear memory space. The size of each memory space (i.e., the number of bases) is determined by the number of memory taps.

TLFNs use memory PEs that are ideal delays or locally recurrent, such as the tap delay line PE, the gamma memory PE, or the Laguerre memory PE. When the bases are created by local IIR filters, the span of the memory space (i.e., the amount of past information that is retained in the memory space bases) is not uniquely determined by the number of delays, but also becomes a function of the feedback parameters that the learning system can control through adaptation.

When the memory filters have feedback, distributed TLFNs are recurrent networks. However, the global signal flow is feedforward: the networks are always stable, provided the locally recurrent memory PEs are stable, and train better than fully recurrent networks. The most important characteristic of distributed TLFN topologies is that when they are recurrent, the feedback is restricted to be local. Moreover, the recursive part is linear, so we can utilize the well developed linear adaptive filter theory to partially study these networks. vector space interpretation of TLFNs

NEUROSOLUTIONS EXAMPLE 11.5

The Issue of Memory Depth in the Bus Driver Problem

This is a benchmark problem to find the memory depth of TLFNs. It is called the bus driver problem, because it can be enunciated as the decision made by a bus driver, who is going down a route with many bus stops. He will stop at the next bus stop if the bell rings before the stop. However, he does not have control of how far in advance the passenger rings the bell.

A simple flip-flop in the bell line will solve this problem using an AND function (if there is a bus stop and the bell flip-flop is high, then stop the bus). However, here we would like to use a TLFN so the issue is one of learning and memory. The network has two inputs, one for the bus stops along the way and the other for the bell. The system is trained with a signal that contains the information about where the bus should stop.

From the point of view of complexity, this is a trivial problem as long as the TLFN has sufficient memory (the AND can solve it). But learning the relation in time is not easy since the TLFN will be bombarded with time signals and has to find the relationship between the two inputs that makes the output follow the desired response. The input file is constructed from 1 and 0 (1 means a possible stop or a ring) asynchronously spread in time. The system is trained with static backpropagation to clearly show the effect of the memory depth.

The purpose of the example is to show that a TLFN with memory in the hidden layer can complement the memory at the input layer, eventually saving weights (the number of weights in the first layer tends to be always the largest).

We use a gamma memory in the first layer and a tap delay line in the second layer. First we set the hidden layer memory to 1 (just the current sample, i.e., no memory), and the number of taps in the gamma to 5. The largest time distance between a ring and a stop is 10 and occurs in the second and the fifth stops. Run the network and see that it does not learn these two stops.

Now change [image] [image] to 0.5. As shown in the memory depth formula, this should provide enough depth to learn all the stops. This is indeed the case.

An alternative is to increase the memory in the hidden layer. Let us divide the number of taps between the input and the hidden layer (5 at the input and 7 at the hidden layer) and set [image] [image] = 1 for the gamma. This solves the problem. We can even use 2 taps and 10 taps and still get a good solution. So this means that putting memory in the hidden layer "adds" to the memory in the input layer (in fact, in a nonlinear way due to the nonlinearity of the PE).

NeuroSolutions Example

11.5.1 How to Train Distributed TLFNs

The supervised training of TLFNs follows the gradient-descent procedure on a cost function defined by the output mean square error (MSE) over time (Eq. 11.4). Since distributed TLFNs are systems with integray or recursive memory, either real-time recurrent learning (RTRL) or backpropagation though time (BPTT) must be utilized.

As we discussed extensively in Chapter 3, to train neural networks using the data-flow framework, two pieces of information are required:

  • The implementation of the data flow algorithm

  • The local maps for the PEs.

We saw in section 11.4.1 how the data-flow algorithm of backpropagation is extended to BPTT and how it is implemented in the dynamic controller of NeuroSolutions. The local maps depend exclusively on the form of the PE. The local maps for the sigmoid PEs, the softmax, the linear PE, and the Synapse are exactly the same as covered in Chapter 1 and 3. The local maps for the context PE are given in Eq. 11.16, hence just need to address the maps for the new memory PEs (gamma, Laguerre, gamma II) to complete the training of distributed TFLNs.

11.5.2 Training Gamma TLFNs with BPTT

The data-flow implementation is intrinsic to BPTT and it is implemented by the dynamic controller, but the local maps depend on the PEs chosen. In section 11.4.1 we presented the local maps for the context PE, which are going to be the basis to develop the equations written below. But notice that the gamma memory is a single-input, multiple-output component, so its dual is a multiple-imput, single-output component. The equations will be written for the kth tap of the ith gamma PE. For simplicity we will omit the ith index with no confusion arises.

(11.23)

The activation map is used by the gamma memory component while the error map is utilized by the dual gamma memory PE. We should remember that for k = 1 (the first tap) the activation map has to take into consideration the spatial connections to the ith gamma PE, that is

The second equation in Eq. 11.23 is just the application of Eq. 11.9 to the gamma memory. In fact, the gamma memory is a cascade of first-order recursive elements in which the stage is indexed by k. The last sum comes from the fact that each tap of the gamma memory dual also receives inputs from the spatial topology. It representes the errors being fed to the kth tap of the ith dual PE. To update the feedback parameter using straight gradient descent, the weight update is

(11.24)

These equations are all that is necessary to update the recursive coefficient with BPTT in systems using the gamma memory, regardless of the topology. training alternative memories

Note that in Eq. 11.23 the activation is equivalent to Eq. 11B.1, but the error in Eq. 11.23 is different from the formula obtained in Eq. 11B.3 since they have been developed under two different methods (RTRL for Eq. 11B.3 versus BPTT for Eq. 11.23). However, as was said previously, the gradient obtained over the same interval will be identical.

Next we present examples of two different topologies that use the gamma memory. The simplest is the gamma filter, which is a linear system. The second topology is a nonlinear TLFN with gamma memory.

NEUROSOLUTIONS EXAMPLE 11.6

Output MSE and the Gamma Memory Depth

This example illustrates the importance of the memory depth for system identification. We use the nonlinear system first presented in Chapter 10 and the topology is a gamma filter with 5 taps. There are two parts to this example. First, we would like to show that the MSE changes with the value of [image], so we sill step by hand the values of [image] from 1 to 0.1 and record the final MSE. As you can see, the curve is bowl shaped: there is an intermediate value of [image] that produces the smallest output MSE (in fact, if we have enough resolution we see that the curve is NOT convex, with 5 minima). Notice that the Wiener filter provides the MSE of [image] [image] = 1, which is not the smallest for this example (and hardly ever is).

The second step is to find out whether we can train the [image] parameter with BPTT to reach the minimum of the performance curve. A good rule of thumb is to use a step size for the feedback parameter at least 10 times smaller than for the feedforward weights. Notice that, the system quickly found the best value of the [image] parameter in this case.

NeuroSolutions Example

NEUROSOLUTIONS EXAMPLE 11.7

Frequency Doubling with a Focused TLFN

To get a better intuition of how different a recurrent memory is from a tap delay line, we present the following problem. We wish to construct a dynamic neural network that will double the frequency of an input sinusoid. Anyone familiar with digital signal processing knows that a nonlinear system is required for this task.

First let us build a one-hidden-layer focused gamma net with a two PEs (tanh nonlinearity), a linear output PE, and an input layer built from a gamma memory with 5 taps. The period of the input sine wave is set at 40 samples/period and the output at 20 samples/period. BPTT with 80 samples trajectory is utilized to adapt all the weights, including the recursive parameter [image] of the gamma memory.

Notice that the [image] parameter starts at 1 (the default value, which corresponds to the tap delay line). The value decreases to 0.6 in 150 iterations, yielding a memory depth of 8.3 samples . This means that with 5 taps the system is actually processing information corresponding to 8 samples. This memory depth was found through adaptation, so it should be the best value to solve this problem.

Now let us reduce the size of the gamma memory from 5 taps to 3 taps, keeping the MLP architecture and the task the same. This time [image] converges to 0.3, giving an equivalent depth of 10 samples (K/ [image]). This makes sense, since the memory depth to solve the problem is determined by the input-output map (frequency doubling), and it is the same in both cases. The second system had fewer taps, so the parameter [image] that controls the memory depth adapted to a lower value, as expected. The dynamic neural net was thus able to compensate for the fewer number of taps by decreasing the value of the recursive parameter, and achieving the same overall memory depth. The memory resolution in the latter case is worse than in the previous case. The tap delay line with 3 taps will never solve this problem.

It is interesting to place a Megascope at the hidden layer to visualize how the network is able to solve the frequency doubling. The solution is very natural. The PE activations are 90 degrees apart. One is saturated to the most positive value, and the other to the most negative value, yielding half sinusoids. Then the top layer can add them up easily.

With the trained system, let us fix the weights and see how it generalizes. Modify the input frequency and find the range for which the output is still a reasonable sinusoid of twice the frequency of the input.

NeuroSolutions Example

We conclude that when the data-flow implementation of BPTT is used to train TLFNs only the topology of the network and the local maps need to be specified, hence there is no need to rewrite the learning equations for each new topology. This brings flexibility and great savings to simulations because TLFNs have highly complex topologies. The data-flow approach to train TLFNs was implemented in NeuroSolutions in 1992, but the methodology was only published by Wan in 1996.

Go to next section


11.6. Dynamic Systems

 11.6 Dynamic Systems

TLFNs encapsulate recurrencies into the processing elements, (i.e. in the memory PE), and all the other connections are feedforward and instantaneous. There is no reason why a PE output cannot be fed to some other PE that is already connected to the original one, producing a feedback loop across the topology, as we did in a restricted fashion in the Elman and Jordan networks. In spatially recurrent systems, assuming an instantaneous map at the PE, an instantaneous connection between the PEs (a product by a weight) leads to infinite looping which is unrealistic and can not be modeled. Delays have therefore to be incorporated either at the PE or at the interconnection level to create time dependencies. We need a rigorous way to treat this type of system, and the best strategy is again to use the ideas from dynamics and mathematical system theory, where these issues were addressed a long time ago.

Figure 11-6 A dynamical system producing a time series

In Figure 11-6, y(n) is the system output and it is a time function. Such a system is called a dynamic system, a system whose state changes with time. A first order dynamical system can be described by a relationship (a function) that links the next value y(n+1) to the present y(n),

(11.25)

This is one of the simplest dynamic systems we can think of, since it is governed by first-order dynamics, that is, the (next) value at time n+1 depends only on the (previous) value at time n. As we saw in Chapter 8, Eq. 11.25 is called a first-order difference equation. If the function f(.) is a constant [image], we obtain a first-order linear dynamical system. We already studied such a system in Chapter 10 in the form of the context PE. But f(.) may also be a nonlinear function, and in this case the system is a first order nonlinear dynamic system.

In neural networks we are interested in first-order nonlinear dynamic systems. In fact, each PE of all the networks studied so far can be modeled in a very general way by a first-order nonlinear first-order difference equation:

(11.26)

where x(n) is the input to the PE, f is one of the nonlinearities studied, and w is a constant.

11.6.1 The State Space Model

A productive (and rigorous) way to think jointly about static and dynamic PEs is to introduce the concept of state-space model [Haykin]. Let us assume that there is an internal variable in the PE that describes its state at sample n. We call this variable the state variable net(n). We can rewrite Eq. 11.26 using the state variable as a set of equations

(11.27)

where w is a constant that may depend on the iteration. Having a set of two equations enhances our understanding. The first equation is dynamic and shows the evolution of the state through time: in neural networks it is a linear first-order difference equation. x(n) is the combined input to the PE. The second equation is static (both y(.) and net(.) depend on the same instant of time), and it shows the nonlinear relationship between the output of the PE and the state. A dynamic neural network will be a distributed interconnection of these PEs.

11.6.2 Static versus Dynamic PEs

Now we can better understand the relation between the static PE covered in Chapters 1 through 7 and the dynamic PEs we have discussed for temporal processing. Basically, the static PE represents only the second equation in Eq. 11.27. The function f(.) is the nonlinear map that we found in the McCulloch- Pitts PE or the sigmoid PE. For all practical purposes the static PE receives the input through the state.

The dynamic PE is different. The next state is a function of the previous state (dynamics) and of the combined input to the PE. The state is then nonlinearly modified to provide an output.

The memory PE that we studied conforms with this principle, except that f(.) is the identity, that is, we use the states as the outputs. It is instructive to analyze the equation of a nonlinear context PE, which is obtained by including the nonlinearity in Eq.10.3:

(11.28)

where f is one of the sigmoid nonlinearities found in Chapter 3. The recurrent nonlinear PE is depicted in Figure 11-7. In Eq. 11.28, the quantity in parenthesis is the state equation that represents a dynamical system, since it depends on two different time indices.

Figure 11-7 Nonlinear context PE

Eq. 11.28 should be contrasted with the equation that defines the McCulloch-Pitts PE indexed by sample n, which we copy here for convenience,

(11.29)

Note that the M-P PE is static, because the output depends only on its current input (or equivalently, on the current outputs of the other PEs when connected in a network), while in Eq. 11.28 the output at time n depends on the previous output at time n-1. This is the big difference between a static and a context PE.

Go to next section


 11.7 Recurrent Neural Networks

How can we construct recurrent neural networks? There are probably many different ways, but here we describe one that allows us to construct recurrent neural networks directly from the previous static components. Let us implement the static map of Eq.11.27 with the PE to achieve total compatibility with all the PEs studied so far. Moreover, let us assign the dynamics to the connectivity among PEs in the following way: If the connection is feedforward, we use the present value of the activation, but if the connection is a feedback connection, we include a delay of one time step in the activation. This can be written as

(11.30)

where I(n) is an external input connected to the PE. Note that we divided the computation of the state into two sums: The first is the feedforward connections, for which the first index of the weight is always larger than the second index with our notation. The second sum is the feedback connections, for which the first index is always smaller than the second. Such a system is called the (fully) recurrent neural network (Figure 11-8).

Once again we have to remember that there are no instantaneous loops in this network, that is, all the feedback loops include a delay of one sample. Unfortunately, this is never represented in the diagrams of recurrent networks. When a dynamic system is built, it is a good practice to include a z [image] symbol in the feedback connections to avoid any confusion.

The recurrent networks include the feedforward systems (MLPs and linear combiners) as well as the TLFNs as special cases. In fact, let us write the interconnection matrix for a fully recurrent system with N PEs (Figure 11-8). In such cases the weight matrix is fully populated with nonzero values.

Figure 11-8 Fully recurrent network

A fully recurrent network with N PEs has N² weights. Note that the diagonal terms in W are the self-recurrent loops to each PE. The upper triangular weights represent feedback connections, while the lower triangular weights represent feedforward connections.

In order for the system to be feedforward, the main diagonal weights, as well as the upper triangular weights (with the numbering as in the Figure 11-8) are zero. The feedforward network is depicted in Figure 11-9. Note that this system is static. A layered system has even fewer connections different from zero (Figure 11-10).

Figure 11-9 The corresponding feedforward network

Figure 10 The corresponding layered network (perceptron)

In feedforward systems the first weight index is always larger than the second (with our notation). For the layered system of Figure 11-10, which includes the MLP, we added another input and another output; otherwise there would be PEs that would not have input or for which the output would not be connected to the external world.

The TLFNs can also be derived from the fully recurrent system when some of the PEs are made linear and their connectivity restricted to cascades as in the memory PE. This arrangement indicates a specialization (prewiring) for temporal processing. TLFN architectures

Go to next section


 11.8 Learning Paradigms for Recurrent Systems

11.8.1 Fixed-Point Learning

How can we train recurrent systems to associate a static input with a static desired response? Here we present an extension of backpropagation to train recurrent systems, which has been called fixed-point learning. The cost function is given by Eq. 11.5. Our goal is to use the paradigm of static learning, that is, to present a static input pattern, clamp it, compare the (steady-state) response with the desired static response and backpropagate an error. With a local activation and a local error, any gradient descent update can be used to adapt the weights. Can backpropagation still be used for this task? Notice that now the system is not instantaneous, as the MLP is. When an input is presented to a stable recurrent system, the output "relaxes" (i.e., slowly evolves) to a final value. We say that the recurrent system has an attractor, or a fixed point. After relaxing to a fixed point the system response does not change, hence the system becomes static.

We are not going to provide a demonstration here (see Almeida), but backpropagation can be extended to train recurrent system with fixed points if the following procedure is followed:

  1. Present the static input pattern for a number of samples (clamp it) until the output stabilizes.

  2. Compare the output with the static desired pattern, form the error, and backpropagate the error through the dual system, just as was done in the static case.

  3. Apply the error in the dual system (clamp it) until the propagated error stabilizes.

  4. Then utilize the search procedure of your choice to update the weights. Repeat the procedure for the next pattern.

There are three important remarks to be made. The input pattern and the error have to be clamped (held at a constant value) until the response stabilizes. This process converges as long as the forward system is stable, that is, the system outputs must stabilize. If the forward system is stable, the dual system is also stable, so the local errors will also stabilize. However, the relaxation time constants of the two systems can be different from each other and change from iteration to iteration. In fact, experience shows that the relaxation time constants often decrease (i.e., activations and errors take longer to stabilize) when the system approaches the solution.

The other aspect is that the learning rates have to be slow. Unlike the static case, here we have two dynamic processes at play: the dynamics of learning that change the system weights and the dynamics of the recurrent system. The dynamics of learning have to be much slower than the dynamics of the system (adiabatic condition); otherwise we are not training one system but a family of systems, and no guarantee of convergence exists.

As a result of these four steps, we can see that standard backpropagation is a special case of the fixed-point learning procedure (no relaxation). To implement fixed-point learning, we just need to control appropriately the relaxation across iterations. Any of the recurrent topologies can thus be trained with fixed-point learning if the goal is to create a mapping between two static patterns (the input and the desired response). An important question that does not have a clear answer is the advantage of using a recurrent topology versus an MLP to learn static mappings. Due to the fine tuning required to successfully train a recurrent system with fixed-point learning, we suggest that the MLP be tried first.

NEUROSOLUTIONS EXAMPLE 11.8

Learning the XOR with a Recurrent Neural Network

In this example we will solve the old XOR problem, but now using a recurrent topology and fixed-point learning. The topology is a fully connected set of PEs that receive input from the file and produce an output through one of the PEs. We use an Arbitrary Synapse with a weight of 1 at the output. All the other weights will be adaptive. We can solve this problem with just 2 PEs, and this is how many we will use.

We use fixed-point learning to train this system. If we go to the dynamic level of the Dynamic Controller Inspector, we can select fixed-point learning. The difference between fixed-point and trajectory learning is that the same sample is always sent to the network during the specified number of samples per exemplar. The goal is to let the system relax to a stable output so that we can compute a meaningful error, send it to the dual network, let the dual stabilize and then compute the weight updates. The time the system takes to relax depends on the current weights, so this is a difficult variable to set, but at an initial guess we can use a long relaxation time of 100 samples (enter 100 in the exemplar per sample window). We should monitor the appropriateness of the relaxation time by putting a Megascope on the hidden-layer PEs.

We are ready to train the system. There is a local minima at 0.5 (input weights go to zero and the system stalls), so we may have to randomize the weights. The selection of the step size and momentum are very important for this problem. Notice that the system trains better for long relaxation times. We have two PEs but many weights, so this topology is not as efficient as the MLP. It may, however, be more resistant to noise than the MLP.

NeuroSolutions Example

11.8.2 Learning Trajectories

One of the unique applications of recurrent networks is to learn time trajectories. A trajectory is a sequence of samples over time. Trajectory learning is required when we want to specify the system output at every time sample, such as in temporal pattern recognition, some forms of prediction (called multistep prediction) and control applications. The cost function is given by Eq.11. 4. We dare to say that a recurrent system is naturally trained in time since it is a dynamic system, so trajectory learning is rather important in many dynamical applications of neural networks. Static networks cannot learn time trajectories, since they do not possess dynamics of their own.

Unlike the previous case of fixed-point learning, here we seek to train a network such that its output follows a pre-specified sequence of samples over time. In a sense, do we not only enforce the final position as in fixed point learning, but we are also interested in defining the intermediate system outputs (i.e., a trajectory). Another case of interest is when the desired response is known at each time step but sometimes later in the near future. The learning system has to be run forward until the desired response is available, and then the weights updated with all the information from the time interval.

There are two basic principles to implement trajectory learning: real-time recurrent learning (RTRL) and backpropagation through time (BPTT). We already covered these two training paradigms in Section 11.4. With the same boundary conditions they compute the same gradients: hence they are equivalent. However, the computation and memory requirements of the two procedures are very different as we discussed in Section 11.4.3.

NEUROSOLUTIONS EXAMPLE 11.9

Trajectory Learning: The Figure 8

This example shows that a recurrent system can follow trajectories in time. We start with a very simple case of learning a figure 8 trajectory in 2-D space using an MLP. We start by using prediction to train the system.

It is more interesting to actually see whether the trained system can be used to generate the trajectory autonomously. Toward this goal we feed the output back to the network input and disconnect the signal input. Can the neural network trained as a predictor still generate the figure 8? Try to verify that it is almost impossible.

We then train the recurrent system with the global feedback loop, but we have to face a problem. Which system input will we use: the waveform generators or the feedback? Since we want the neural network to ultimately create the trajectory without an external input (just the feedback), we should disconnect the input, but notice that the training becomes very difficult.

It will be very difficult to move the weights to the right values using the error computed over the trajectory. You may want to do this, but it takes a long time to train and the learning rates have to be small. Also, you have to avoid the local minima problem (e.g., if the trajectory length is equal to the length of the figure 8 zero is a local minimum).

An alternative is to use BOTH the input and the feedback, but in a scheduled way. In the beginning of training most of the information should come from the input to help put the weights at approximately the correct positions. But toward the end of training only the feedback should be active. We can do this either by segmenting the trajectory into a portion that comes from the signal generator and another portion that comes from the feedback or simply by creating a weighted average between the two signals. In the beginning of training a large segment should be given to the signal generator, and toward the end of training most of the data should come from the feedback.

Observe the system train (it takes a while). Once it is trained, you can fix the weights and you can see that it keeps producing the trajectory 8, that is, we have created an oscillator with a very strange form. This is a powerful methodology for dynamically modeling complex time series (e.g. we can replace the simple figure 8 by arbitrarily complex trajectories created by real world time series).

NeuroSolutions Example

A word of caution is in order here. In the literature we sometimes see training of recurrent networks with static backpropagation. This is NOT correct, since, as we have seen the gradients in recurrent systems are time dependent. When we use static backpropagation we are effectively truncating the gradient at the current sample, that is, we are basically saying that the dual system is static, which is at best an approximation and can lead to very poor results. On the other hand, this does not mean that we need the full length of the trajectory to effectively compute the gradient over time. An interesting approach is to find out a reasonable number of samples to propagate back the gradients, which leads to what has been called truncated backpropagation (see Feldkamp). Truncated backpropagation is more efficient, since we are no longer storing the activations and sensitivities during the full length of the trajectory.

11.8.3 Difficulties in Adapting Dynamic Neural Networks

It is fair to say that enough knowledge exists to train MLP topologies with backpropagation. However, it is not easy to adapt dynamic neural networks, whether TLFNs or fully recurrent topologies. The difficulty stems not only from the sheer computational complexity that produces slow training, but also from their complex performance surfaces, the possibility of instability (in fully recurrent systems), and the natural decay of gradients through the topology and through time. Training recurrent neural networks with BPTT (or with RTRL) is still today more of an art than a science, so extreme care should be exercised when training dynamic neural networks. We would like to add that TLFNs are easier to train than fully recurrent networks, and should be the starting point for any solution.

The performance surface of dynamic neural networks tends to have very narrow valleys, so the step size must be carefully controlled to train them. Adaptive step size algorithms are preferred here. In NeuroSolutions we can schedule the step size by hand when automatic algorithms do not perform well.

During training the recurrent network can become unstable. The nonlinearity in the PEs will not allow the system to blow up, but the PEs become pegged and may oscillate widely between the extreme values. Monitoring for this situation and resetting learning is normally the only way out. In effect, such PEs are not being used for processing functions, and the effective number of degrees of freedom of the system is much smaller than the number of available PEs would suggest. There are definitions of stability more appropriate than BIBO stability for nonlinear systems but they are difficult to apply.

Another problem is the decay of the gradient information through the nonlinearities, which has been called the long-term dependency problem (see Bengio ). A dynamic neural network in a classification task tends to have its hidden PEs saturated, so that it can operate like a finite-state machine. In such cases the gradients are attenuated greatly when the system is trained with BPTT, which makes training difficult and very slow. There is no known mathematical solution to this problem, but engineering solutions have been proposed. This characteristic of dynamic networks implies that using memory PEs inside the network topology as we did in the distributed TLFN network may simplify training and turn the networks more accurate. advantage of linear memory PEs

All these aspects raise the question of the applicability of gradient-descent learning to training dynamic neural networks. Recently, alternative training procedures using decoupled Kalman filter training (dKft) have been proposed with very promising results. [Feldkamp and Puskorios]

Go to next section


 11.9 Applications of Dynamic Networks to System Identification and Control

System identification and control is probably the area which benefits most from the power of dynamic neural networks and BPTT. We already presented a few applications of ANNs to system identification in Chapter 10. The advantage of ANNs is that they are adaptive universal mappers. However, in Chapter 10 we did not know how to adapt the feedback parameters, so we had to restrict ourselves to only a few topologies.

Now with the introduction of BPTT we can lift this restriction and apply ANY ANN topology (static or dynamic) to identify or control plants. There are many different ways to establish a successful strategy to do this. See Narendra for a complete discussion. Here we address some preliminary issues that are important for the understanding and application of the material covered in this book.

11.9.1 Adapting ANN Parameters in Larger Dynamical Systems

In Chapter 3 we showed how to optimally design a system built from an adaptive subsystem (a neural network) with other static subsystems that were differentiable with fixed parameters. Enhancing this view when the subsystems are dynamic is critical for system identification and control. In practical applications we may be faced with the task of including neural networks in larger systems built from unknown plants or other engineering systems (such as controllers), and of course we would still like to be able to optimally set the network parameters.

If you recall, the reason we could easily adapt an ANN inside a larger system is that backpropagation is local in space. This means that the ANN only cares about the signals that were transmitted from the input (through an unknown subsystem) and backpropagated from the output (also through another unknown subsystem). Since backpropagation was generalized to BPTT, we can still fully apply the same principle, but now extended to dynamic neural network. Since BPTT is not local in time, however, we have to work with gradients from a segment of data. To preserve the on-line learning ability, we must advance the trajectory one sample at a time. However, for some applications the use of trajectory chunking is still practical and leads to more efficient implementations than RTRL. The length of the trajectory is important, because it should be long enough to capture the dynamics of the overall system; however, this is not known a priori, so the trajectory length is subject to experimentation. Notice also that to avoid discontinuities at the end of the trajectories, the system state at the final trajectory time must be stored and used as the initial condition for the following trajectory.

Dynamic backpropagation is probably the most widely used method in the neural control literature [Narendra] Dynamic backpropagation combines RTRL and BP. It adapts the static neural networks using BP, but uses RTRL to propagate forward the sensitivities. Each representation of Figure 11-11 requires a different analysis. For example, dynamic backpropagation for the block diagram of represention 1 can be written as

(11.31)

where the sensitivity in the ANN is computed with backpropagation, while the propagation of the parameter change is done with RTRL on a copy of P(v, [image]). dynamic backpropagation

Hence learning can be implemented on a sample-by-sample basis (in real time). The problem is that the forward propagation of sensitivities is very expensive computationally, and the method must be adapted to the particular system interconnectivity (block diagram).

Figure 11-11 The four most common configurations involving ANNs and fixed dynamic systems

NEUROSOLUTIONS EXAMPLE 11.10

Training an Embedded Neural Network in a Dynamic System

To show that we can use BPTT in a system built from ANNs and other dynamic components, let us solve the following problem. Consider a feedback system (representation 3 in Figure 11-11) where the forward transfer function is

and the feedback P(v, [image]) is linear and given by

The input is x(n) = 2sin( [image] n/25). The goal is to identify this feedback system, assuming that we know the feedback transfer function; in other words, we want to identify the forward path only. We cannot disconnect the two components, so training must be done in the integrated system since we only know the combined output. We use BPTT for this task.

The breadboard has two basic networks: one called the modeled plant and the other called the ANN model. The modeled plant has an unknown subsystem implemented by the top Axon with a DLL and a feedback subsystem, which is known and also modeled by an Axon with a DLL. The feedback block, called the known feedback model, is copied to the ANN model at the bottom of the breadboard. The desired response for the ANN model comes from the modeled plant, also through the DLL on the L2 component, while the input is fed to both the plant and the ANN model. This arrangement is pure system identification.

The ANN model identifies only the feedforward part of the plant, which is easier than identifying the full plant with feedback. As a rule of thumb, our ANN models should include as much information as available from the external world. This breadboard shows how you can do it.

NeuroSolutions Example

11.9.2 Indirect Adaptive Control

Probably the most general control problem is one in which we do not have an equation for the plant, but we have access to the plant input and output. How can we derive a model to control such a plant? This question can be answered with the system identification concepts covered in Chapter 9, extended now to TLFNs. The block diagram we use is shown in Figure 11-12 and is called indirect adaptive control.

The main idea is to use two neural networks, one to identify the plant and the other to control the plant. The identification model is used to pass the errors back to adapt the controller. If the plant is dynamic, BPTT (or dynamic backpropagation) must be used to train the models. The figure also shows an extra input-output loop that includes the reference model. The reference model dictates the required plant behavior when the input is r(t). It allows us to compensate for known plant characteristics such as delays, thus simplifying the training process.

Figure 11-12 Indirect adaptive control

Let us explain the figure. Since we do not have an equation for the unknown plant it would be impossible to adapt the controller (ANN2), which requires sensitivities generated by comparing the reference input (desired behavior) to the plant output. Hence, we create a neural network identifier (ANN1), which will learn to produce an output that is similar to that of the plant. A typical topology for ANN1 is the NARX model (Chapter 10), in which the output is a nonlinear function of both the plant input (and its past values) and the plant output (and its past values). But other choices are possible.

Once the system identification (SID) neural network ANN1 is adapted, it behaves like the plant in terms of input-and-output, but most important, we created a known system description for which the dual network can be obtained. Therefore the controller ANN2 can be adapted by simply backpropagating the errors through the dual of ANN1.

This scheme requires to adapt ANN1 first (possibly off-line), fix its parameters, and then adapt the controller ANN2. When both the controller and the system identifier are trained at the same time, the step size for ANN 2 (the controller) has to be much smaller than that of ANN 1 (the system identifier). Once again, the rule of thumb of 10 times slower is a good starting point.

NEUROSOLUTIONS EXAMPLE 11.11

Training a Neural Controller

We solve the following problem with NeuroSolutions. Suppose that we want to control a plant with unknown characteristics using Figure 11- 12. We have the ability to only excite the plant and measure the output. A fundamental problem is how to adapt the weights of the controller, since we need the dual of the plant. We therefore have to use the black-box approach of indirect adaptive control. We use a reference input r(n) = sin(2 [image] n/25). The unknown nonlinear plant is described by the second-order differential equation

where

This function is implemented in the top Axon through a DLL and labeled "model plant".

There are two parts to the solution. The first step is to identify the plant; the second is to train the controller. In the first step we create ANN1, labeled "ANN for system ID", which is a one-hidden-layer focused TDNN. Filtered white noise is fed both to the ANN1 and the plant. The desired response is transmitted from the plant output through the DLL placed on the L2 Criterion. This is the pure system-identification configuration, and static backpropagation is used to adapt the plant model. In the second panel, we simply check the result of the system identification step. Notice that the backpropagation plane was discarded, since ANN1 (the plant model) will keep its weights fixed after the identification.

In the third panel we implement the ANN2 (controller) of Fig 11.12. We also use a focused TDNN with one-hidden-layer to create the controller and adapt it with BPTT. Even though we can train a focused TDNN with static backpropagation, we are backpropagating through ANN1 which has memory, thus we must use BPTT. Notice the interconnections. ANN2 receives as input the reference input (delayed by one sample), and its desired response is obtained from the difference of the plant output and the reference input. However, the error signal is propagated through the dual of the ANN1 (plant model). We have to assume that the system identification is sufficient accurate, since we are passing the forward signal through the unknown plant and the errors through the dual of the plant model. If the plant model and the plant are different, this backpropagation strategy has catastrophic consequences.

NeuroSolutions Example

Go to next section


 3.10 NeuroSolutions Examples

3.1 McCulloch-Pitts PE for Classification

3.2 Discriminant Probe to Visualize the Decision Surface

3.3 Behavior of the Sigmoid PEs

3.4 Classification as the Control of the Decision Surface

3.5 Perceptron Learning Rule

3.6 Delta Rule to Adapt the M-P PE

3.7 Comparing a Linear and Nonlinear PE for Classification

3.8 Decision Boundaries of the Perceptron

3.9 The Perceptron for Character Recognition

3.10 Adatron Algorithm

3.11 Perceptron and the XOR Problem

3.12 One-Hidden-Layer MLP in 2-D Space

3.13 Solving the XOR Problem by Hand with the-One-Hidden Layer MLP

3.14 Creating a Bump with the One-Hidden-Layer MLP

3.15 Solving the XOR with Backpropagation

3.16 Solving the Bump with Backpropagation

3.17 Effect of the Number of Hidden PEs

3.18 Fewer Hidden PEs than Required

3.19 Creating a Halloween Mask by Hand with a Two-Hidden-Layer MLP

3.20 Comparison of the Perceptron and MLPs

3.21 Solving the Bow Tie with MLPs

3.22 Data Flow Implementation of Backpropagation

3.23 Training Embedded Neural Networks with Backpropagation

3.24 Softmax and A Posteriori Probabilities

Go to next section


 3.11 Exercises

3.1 Create by hand the weights of a perceptron that will work as an AND function. Test the solution in NeuroSolutions. If you take out the bias of the PEs, can you still solve the AND problem? Confirm with NeuroSolutions.

3.2 For the data shown in the figure (dark circles are one class, white circles the other) solve the classification problem with a M-P PE by hand, that is, find the appropriate weights. Give the equation for the separation surface. If the perceptron weights are (2,-2) and the bias b=1, is the problem solved?

3.3 Explain why the classification regions created by a perceptron are always convex.

3.4 Specify the weights of a single layer MLP that will create discriminants as in the figure (the shaded area should provide a response of zero). Confirm your solution with NeuroSolutions. Then create a data file with two classes that will provide the same basic discriminant function. Train an MLP to show that you are correct.

3.5 Construct by hand the MLP weights that will be able to create the following figure (shaded is 0, white is 1).

3.6 Specify the dual network of the softmax PE.

3.7 Suppose that you use the following activation function for a new PE:

(a) Derive the learning equations

(b) Create the dual network

(c) Implement in NeuroSolutions a DLL that will implement your PE.

3.8 For the sigmoid nonlinearities of Section 2.1, derive the equations that will adapt the scalar [image]. What do you gain by adapting [image]? Can you achieve a similar result by adapting other learning parameters?

3.9 Compare experimentally a perceptron network trained with the perceptron learning rule, the delta rule and the Adatron in the data set prob3.9train (and test) in the folder Chapter 3, contained in the data directory (CDROM). Use the confusion matrix to rank the different solutions.

3.10 Is the following network an ordered network? Determine the ordered list if your answer is affirmative.

Go to next section


 3.12 Concept Map for Chapter 3

Go to Next Chapter


 4.10 NeuroSolutions Examples

4.1 Visualization of Learning

4.2 Learning as a Stochastic Process (XOR)

4.3 Learning Rate Scheduling

4.4 Flow of Errors Across MLP Layers

4.5 Effect of Initial Conditions on Adaptation

4.6 Momentum Learning

4.7 Adaptive Step Sizes

4.8 Adaptation with Noise in the Desired Signal

4.9 Stopping Based on MSE Value

4.10 Stopping with Cross-Validation

4.11 Sparse Connectivity in the Input Layer

4.12 Confusion Matrix for Classification Performance

4.13 Regression Performance as a Function of the Norm

4.14 Classification Performance as a Function of the Norm

4.15 Cross-Entropy Training

4.16 Weight Decay

4.17 Committees

4.18 Crab Data Classification

4.19 Iris Data Classification

4.20 Risk Decision in the Cancer Data

Go to next section


 4.11 Exercises

4.1 Explain why consecutive steps are orthogonal to each other in line search methods.

4.2 Experimentally analyze the convergence of the MLP as a function of the step size using NeuroSolutions Example 4.2. Use the basketball player data contained in the data directory (CD-ROM). You should plot the learning curve every iteration and log the number of iterations required to reach a fixed MSE. Then do this for a range of step sizes. The end result may amaze you, but you should try to explain it.

4.3 Repeat Problem 2 using momentum learning.

4.4 Repeat Problem 2 for the delta-bar-delta search and compare performances.

4.5 Repeat Problem 2 with noise in the desired response and compare performances.

4.6 Repeat Problem 2 varying the [image] [image] slope parameter of the PEs, but keep the step size constant. Compare performances.

4.7 For the female-male height-weight data set, compare the generalization obtained with cross validation versus the one based on training-set error. Use the confusion matrix to evaluate the test set performance.

4.8 Study the effect of the size of the cross-validation set on generalization. Use the data set in the folder chapter 3 contained in the data directory (CD-ROM). You should construct cross-validation sets of different sizes (30, 60, 90 samples) taking data from the training set, and plot the corresponding test-set accuracy.

4.9 Study the effect of different norms on classification. Choose the male-female data set. First, study the convergence time of different norms, then study the test-set accuracy, and draw conclusions. Can you extrapolate these results to other data sets?

4.10 Right or wrong: One should compare the performance of different networks using cross-validation instead of imposing a threshold in the MSE. Carefully explain your reasoning.

4.11 Construct the sign error norm for a two-class problem:

and apply Eq. 4.12 to train the network. Experimentally verify its performance on the data set of your choice. Explain this error in terms of Lp norms.

4.12 Construct an asymmetric error using the concepts of Eq. 4.11 in the following way: Errors for class 1 should be weighted by L8, while errors for class 2 should have a L1 norm. Explain what this does in terms of placement of decision surfaces (use the L2 norm as a comparison). Apply this mixed norm to the breast-cancer data and compare results with the risk analysis.

4.13 Experimentally compare the performance of the K-L criterion (Eq. 4.20) with the L2 for a data set of your choice.

4.14 Experimentally compare the performance of the MLP (use the confusion matrix) with and without weight decay in a data set of your choice. Work with large topologies to evaluate the differences.

4.15 Use the following two strategies to train an MLP for one of the data sets:

(a) (a) Use the minimum of the cross-validation error to find the best network architecture. This means you have to run different-sized networks over the data and choose the one that gives the smallest cross-validation error.

(b) (b) Use weight decay to find the best network architecture.

Compare the performances of these strategies and also the number of weights different from zero. You should find the two very similar.

4.16 Experimentally verify the performance improvement of committees, in one of the data sets. Start with two very different topologies (one and two-hidden layer MLPs). All should be large networks. Train each separately, and test the classification performance. Then combine all of them according to Eq. 4.21. Compare the testing results. Then train the combination weights and test the performance again.

Go to next section


 4.12 Concept Map for Chapter 4

Go to Next Chapter


5.10 Conclusion

 5.10 Conclusions

In this chapter we provided a view of neural networks as function approximators. This is the more general view of this family of systems and improves our understanding about their capabilities, establishes new links to alternative methods, and provides a better understanding of the problems faced in training and using adaptive systems.

One of the interesting things about neurocomputing is that it lies at the intersection of many diverse and complementary theories, so it is a very rich field. The price paid is that the reader was bombarded with many different concepts, and since our goal is to keep the text at an introductory level, the presentation addressed only the key concepts. Our hope is that the reader was motivated enough to pursue some of these topics.

MLPs are very important for function approximation because they are universal approximators, and their approximation properties have remarkably nice properties (the approximation error decays independent of the size of the input space). This may explain why MLPs have been shown to outperform other statistical approaches in classification.

In this chapter we also introduced another class of neural networks, called the radial basis function networks (RBFs). RBFs can be used in the same way as MLPs since they are also universal approximators, that is, they can be classifiers, regressors, or density estimators.

We also presented the basic concepts of the structural risk minimization principle and support vector machines (SVMs). This is a difficult theory, so we merely highlighted the many important implications, from the paradigm shift from the conventional compromise between generalization and network size, to the strict recipe of using the smallest-capacity machine for best generalization. The innovative ideas contained in the SVM principle will tremendously affect the evolution of the whole field of learning from examples and inductive inference.

SVMs are practical learning machines that minimize an upper bound to the Bayes error, so they are very useful in pattern recognition. SVMs are very easy to apply to practical problems, provided that the user has large computers. For small data sets the kernel Adatron algorithm allows a simple, "neural" like implementation of the quadratic programming required to find the support vectors and conquers one of the difficulties of the method (having access to easy to use quadratic programming software).

For the reader with a background in engineering this chapter provided a view of MLPs and RBFs as function approximation implementations with a new set of bases (the sigmoids, which are global, or the Gaussians, which are local). For the reader with a statistical background, the chapter provided a new view of generalization. Neural networks belong to the exciting class of nonparametric, nonlinear models, which learn directly from the data and thus can be used in experimental science.

Go to next section


 5.11 Exercises

5.1 (a) Compare the approximation capabilities of the linear regressor and the MLP for the following mapping: input ramp to output sine function. Use the Example 5.4, and change the input desired response files with the signal generators in NeuroSolutions. Choose the number of hidden PEs of the network, and correlate with the MSE. Observe the values of the weights, and conclude how the network is doing the mapping.

(b) Repeat 1a using for input the triangular function. What did you observe? Explain the differences.

(c) Now go to the Inspector of the input component and include a 90-degree phase shift. Explain the results.

(d) Now with this experience, predict what will happen when you use a square wave at the input. Run NeuroSolutions to see if you are right.

5.2 Repeat Problem 1a for the RBF network. Run the network. Can you explain what you see? You have to modify the setting of the centers, since they are chosen by hand in this example. Enter values between –1 and 1, and run the network again. Are you satisfied with the performance? You have also to control the width of the Gaussian. Go to the Inspector and change the width to 10. Repeat the simulation and compare performances.

5.3 Add a bias to the noise term of the Example 5.7 and explain the results.

5.4 Let us analyze in more detail the breadboard in Example 5.8. First, add an impulse of magnitude 0.5 to the desired response (by placing the function generator in the staked access). For visualization purposes, enter [image] 45 in the phase. Now retrain the system. You have to control learning a little better by decreasing the step size. Notice how the impulse changed the fit only marginally. Now repeat with the L2 norm and compare the fits.

5.5 Experiment with the number of neighbors for the adaptation of the RBFs in Example 5.9. Try 1 and 5. Place the matrix viewer on the width access point to see the center values. What do you conclude?

5.6 For the Example 5.13 the MLP provides fairly good performance. Now try to achieve a similar result with an RBF network. Modify the number of PEs and the training (both centers and variances), as well as the number of training iterations for the unsupervised part. Observe the end results on the scatter plot.

5.7 Solve the classification of Problem 6 with a SVM.

5.8 Use the sonar data to compare the performance of the SVM, RBF and MLPs. You should train the RBF and MLPs with crossvalidation before you do the comparisons.

Go to next section


 5.12 NeuroSolutions Examples

5.1 Sinc Interpolation

5.2 Fourier Decomposition

5.3 Linear Regression

5.4 Function Approximation with the MLP

5.5 MLP to Approximate a Square Wave (Classification)

5.6 Function Approximation with RBFs

5.7 Nonlinear Regressors

5.8 MLPs for Function Approximation with L1 Norm

5.9 Training RBFs for Classification

5.10 Overfitting

5.11 Akaike's Criterion for RBFs

5.12 Weight-Decay to Prune RBFs

5.13 MLPs and RBFs for Classification

5.14 Density Estimation with RBFs

5.15 The Kernel Adatron and the Spiral Problem

5.16 Prediction of SP 500

5.17 Estimating Prices in the Boston Housing Data

Go to next section


 5.13 Concept Map for Chapter 5

Go to Next Chapter


 6.10 Autoassociation

There are basically two types of associative memories: the heteroassociators and the autoassociators. As we have just seen, heteroassociation or simply association is the process of providing a link between two distinct sets of data (e.g., faces with names). Heteroassociation is the most widely used associative memory paradigm. Autoassociation links a data set with itself. You may wonder where autoassociation will be useful. It turns out that autoassociation can be used for input reconstruction, noise removal, and data reduction.

In autoassociation the output pattern is equal to the input (substitute x for d in Figure 6-10), and the system is trained either with forced Hebbian learning or with LMS. If we substitute x for d in Eq.6.24 we see that the cross-correlation function becomes the autocorrelation function, and so the weight matrix of Eq. 6.27 becomes the autocorrelation matrix of the input

(6.40)

Thus when a pattern is presented to the input and no crosstalk is present the autoassociator produces an output

(6.41)

since [image] is a constant equal to the length of the input vector. Recall that this is exactly the condition for a vector to be an eigenvector of a matrix, so we conclude that the autoassociator performs an eigendecomposition of the autocorrelation function, that is, the power of the outputs will be the eigenvalues of the autocorrelation function, and the weights are the associated eigenvectors. (Hebbian as the minimization of the reconstruction error).

The problem is that if we train the system with forced Hebbian learning and the inputs are not orthogonal, there will be crosstalk. However, if the learning rule is the LMS, the crosstalk will be decreased to virtually zero (one can show the solution exists unlike the case of hetero-association).

Notice that in the topology of Figure 6-10 there is no flexibility in the reconstruction. We can produce a more powerful network called an autoencoder (or autoassociator) if we include an extra layer of linear PEs, as in Figure 6-16. The network is normally trained with backpropagation (although the PEs are linear) since there is no desired signal at the hidden layer. We normally impose a constraint that the output matrix [image]. Under this constraint we can show that the network will operate in the same way as the PCA network studied in Section 6.4. The square of the signals zi are effectively the eigenvalues, and their number selects the size of the reconstruction space.

Figure 6-16 Autoassociator with W2 = W1

With this constraint we can alternatively train the network using LMS to determine the output-layer weights and then copy them to the transpose locations (reversal of the indices) in the input layer. For this case the weight update using the LMS rule for the output-layer weights is

(6.42)

which can be recognized as Oja's rule. The autoassociator is a linear system, so it has been analytically studied in depth (Baldi). We now know that the performance surface of the autoassociator is nonconvex with saddle points but does not have local minima. This means that the convergence to the global minimum can be ensured by approximately controlling the learning rates.

It is possible to lift the constraint of the transpose between the input and the output weight matrices and simply train the network with backpropagation (we cannot use straight LMS since we have a hidden layer). It can be shown that in this case the PCA solution is not always obtained, although the system still performs autoassociation, and the solution found by the hidden PEs exists always in the principal component space (but the outputs of the bottleneck layer are not necessarily eigenvalues Baldi). The interesting thing is that in some cases the autoassociator with no constraints on W2 is able to find projections that seem to better preserve the individuality of each input class, which makes it better for classification. However, no linear solution will be able to provide a better reconstruction error than the PCA.

NEUROSOLUTIONS EXAMPLE 6.15

Autoassociator and PCA

This problem is a duplication of the reconstruction of digits using PCA, but now we use an autoassociator trained with backpropagation. Notice the architecture with a hidden layer (called the bottleneck layer). This network effectively computes the PCA when the second weight matrix is restricted to be the transpose of the first weight matrix. For the system to train well, we have added a minor amount of noise to the input. In this example there is no constraint on the weight matrices.

Experiment with the number of the PEs in the bottleneck layer and compare the accuracy of the digits obtained with this autoassociator and with the PCA with the same subspace. Notice that the reconstruction error is higher than PCA, but the digits seem to be better discriminated. Use an MLP with the confusion matrix to quantify this observation.

NeuroSolutions Example

6.10.1 Pattern-Completion and Noise-Reduction Properties of the Autoassociator

Another interesting property of the autoassociator is the pattern-completion property, which is very useful for noise reduction and recovery of missing data. Suppose that a segment [image] of an input vector x is lost (for instance during transmission). Let us see whether we can recover the full vector after passing it through the autoassociator. The part of x that is lost is orthogonal to what was kept x [image], so this is equivalent to decomposing x into two orthogonal components x = x [image] = x [image], for instance,

Now if we write the weight matrix W as a function of the lost and kept parts,

(6.43)

the output becomes

(6.44)

One can show using the orthogonality of [image] and [image] that the output is

(6.45)

where [image] is a scalar ( [image] = (x [image]) [image] x [image]), hence, the true output x is obtained. The same argument can be used to show that the autoassociator filters out noise. These are very important properties for data transmission.

NEUROSOLUTIONS EXAMPLE 6.16

Autoassociator and Pattern Completion

In this example, we show how an autoassociator can be used for pattern completion. If the autoassociator is trained with noisy inputs, it will eventually learn the important parts of the input pattern. Then after training, if we input patterns that are noisy or incomplete (e.g., digits with missing segments), the autoassociator will reconstruct the correct image because it has enough information from the input pattern to correctly reconstruct the output pattern.

NeuroSolutions Example

6.10.2 Supervised versus Unsupervised Training

An interesting observation from the autoassociator discussion is that we reached the same solution with very different learning paradigms: for the PCA we used unsupervised learning, but for the autoassociator we used a supervised procedure (the LMS rule) on a linear architecture with a transpose constraint (W2 = W1 [image]) and a desired response d(n) = x(n). The conclusion is that supervised learning using the minimization of the L2 criterion defaults to unsupervised (Hebbian) learning when the desired signal is equal to the input.

We should ask what the real difference is between supervised and unsupervised learning. Until now we stated that the existence of the desired response made the difference, but the example of the autoassociator proved us wrong, so we have to further qualify the differences.

A learning system adapts its coefficients from the environment using one or several sources of information. In unsupervised learning the only source of information from the environment is the input. In supervised learning there is more than one source of information: the input and the desired response. For the learning to be qualified supervised, however, the information contained in the desired response must be different from the input source. Otherwise, as we just saw, supervised learning defaults to an unsupervised solution.

A further question is the efficiency of both learning strategies. It may be that even if we want to conduct unsupervised learning, a supervised training rule is preferable for more efficient extraction of information from the input signal (provided that we appropriately choose the desired response, e.g. d(n)=x(n)). We submit that in this context supervised learning is more efficient than unsupervised learning. This is reasonable, since the desired signal plays a specific goal in supervised learning, and we now know efficient algorithms to search the performance surface (the gradient-descent rule). We saw earlier that the autoassociator of Example 6.16 that was trained with backpropagation learned much faster than the PCA network of Example 6.6. The other practical condition for which supervised learning defaults to unsupervised is prediction, as we will encounter in Chapter 10. Others may exist.

Go to next section


 6.11 Nonlinear Associative Memories

Up to now we covered only linear associative memories, or LAMs. But there is no reason to limit ourselves to linear PEs. In fact, when the PEs are nonlinear, more robust performance is normally obtained. Some new designs are even able to automatically provide a normalized output when the input is normalized, which simplifies learning. The topology of a nonlinear associative memory is shown in Figure 6-17.

Figure 6-17 A nonlinear associative memory (NLAM)

Note that the nonlinear PE affects only the output of the memory, so the Hebbian learning of Eq. 6.1 has exactly the same form for nonlinear networks. One important advantage of bringing in the nonlinearity is to threshold the output of the LAM. For binary-encoded data the threshold can clean much of the crosstalk error from the output. We can see from Eq. 6.34 that mistakes occur only when the crosstalk term is larger in magnitude than the threshold used to make the binary assignment. The nonlinear LAM is more robust to noise. Equivalently, if the input is contaminated by noise, the output can be noise free, which is impossible with the LAM.

NEUROSOLUTIONS EXAMPLE 6.17

Nonlinear Associative Memories

Here we modify the breadboard of Example 6.12 with output nonlinear PEs. The big advantage of the nonlinearity is that it can threshold the errors (crosstalk) if it is below the level needed to make the decision (which is normally set at half of the dynamic range). This means that if the true output was a 0, but the crosstalk was 0.4 (between 0 and 1), the output is still 0, the correct response. Since this is done at the output, we either can consider the nonlinearity part of the network or simply an external read out.

When we implement this type of network in NeuroSolutions and train it with LMS, we have to make sure that the error is passed through a linear backpropagation component to mimic the effect of Hebbian learning; otherwise the final weights will differ from the linear solution. We can see that the system totally cleans up its outputs, so it provides a better memory.

NeuroSolutions Example

Example 6.17 shows some of the advantages of the nonlinearity. However, the vector space interpretation for the outputs is lost due to the nonlinearity. For instance, we can no longer talk about eigenfilters or PCA. However, the network may in fact perform better than the linear counterpart in some applications. In the autoassociator, when the bottleneck layer is built from nonlinear PEs, it has been shown that the result is still PCA, that is, the linear solution is obtained. However, if the network becomes multilayer, the nonlinear network may perform better. These are presently active areas of research.

Go to next section


 6.12 Project: Use of Hebbian Networks for Data Compression and Associative Memories

6.12.1 Data Compression

In data compression we have a source of data, a communication channel, and a receiver. Communication channels have a usable bandwidth, that is, for a given error rate the number of bits per second - the bit rate- has an upper bound. The goal is to be able to transmit as few bits per second as possible while preserving as much information as possible about the source. This means that we must squeeze into each bit as much information as possible from the source. We can immediately see the prominent role of PCA for data compression. Moreover, we can model data compression as a projection operation where the goal is to find a set of bases that produce a large concentration of signal power in only a few components.

Figure 6-18 Data transmission with compression

In practical applications data compression has been based on simpler schemes, where the projection vectors are fixed functions instead of being signal dependent as in the PCA. A good example is the cosine basis of JPEG, called the discrete cosine transform (DCT) (see Rao) But notice that there is no need for such constraints, since the determination of the optimal projection is an off-line operation, so we can strive for optimal decompositions as long as the reconstruction can be done efficiently.

In PCA compression the receiver must know the weight matrix containing the eigenvectors, since the estimation of the input from the eigenvalues is done by Eq.6.18. The weight matrix is obtained after training with exemplars from the data to be transmitted. It has been shown that for special applications this step can be completed efficiently and is done only once. But in general, a given set of coefficients for given signal types (e.g., in images a set of coefficients for people faces, outdoor natural scenes, buildings, etc.) will provide better results. Notice however, that new image coding schemes such as MPEG already provide this type of labeling, so the receiver can be constructed before hand. The reconstruction step requires M x N operations where N is the input vector dimension and M is the size of the subspace (number of features).

NEUROSOLUTIONS EXAMPLE 6.18

Data Compression with PCA

We have already shown data compression with PCA, but here we will treat the breadboard in more realistic terms. We have included one extra Synapse and an extra Axon between the output of the bottleneck layer and the reconstruction layer to clearly show the transmitter at left and the receiver. The extra Synapse depicts the communication channel.

With the PCA the compressor first has to be trained and its weights transmitted to the receiver (which we have done with a DLL), but this is needed only once after the weights converge. Run the network and experiment with the number of features.

Next let us include a noise source at the receiver to mimic the noise in the communication channel. Notice that the PCA encoding is very immune to white, zero-mean noise. Effectively, the eigenvectors work as low-pass filters, so the noise is averaged out.

NeuroSolutions Example

6.12.2 Associative Memory

Associative memories are one of the most widely used applications of Hebbian networks. In particular, in the cognitive sciences LAMs are used because of the analogies between associative memories and mammalian memory. In general, when the size of the input vectors are much larger than the number of patterns to be stored, this type of memory provides an effective way of associating input patterns with output patterns. The systems train fast, and there are no local minima, so they are practical.

Image processing is such an application due to the large input vector of a normal image. In fact, a N x N image is a point in a N² dimensional space, so we can store many image-to-image associations in a matrix of weights. In these cases we may not even need all the weights for perfect recall. This project explores the size of the weight matrix for association in image processing. Due to the size of the systems involved, you may need a fast computer for training.

NEUROSOLUTIONS EXAMPLE 6.19

OLAMs and Arbitrary Connections

In this example we will use a LAM trained with the LMS rule (an OLAM) to associate facial images of three people with images of their names. To reduce network complexity, we use the Arbitrary Synapse to reduce the number of weights in the system. A fully connected weight matrix would contain over 400,000 weights (48 x 48 pixel input and 7 x 30 pixel output). The system uses roughly 20,000 weights, which gives more than enough power to solve the problem. Remember that we have only three images, which is much less than the capacity of the network.

NeuroSolutions Example

Go to next section


 6.13 Conclusions

This chapter presented linear networks adapted with Hebbian learning and similar rules (Oja's and Sanger's) which are in principle unsupervised learning types. We showed that such networks can be used for data representation also called feature extraction, since they project high-dimensional data to smaller-dimensional output spaces. Hence PCA networks can be used as data preprocessors for other connectionist topologies such as the MLP.

There are analytic procedures to compute PCA, so we might think that this class of networks can be easily substituted by mathematical operations, which is true but does not address the implementation issues that are important in practical cases. Here all the learning rules were implemented sample by sample and eventually with local algorithms, so they are well suited for on-line distributed implementations. When the matrices are ill-conditioned the numerical solutions fail, while the adaptive solutions provide one of the many possible solutions.

Another application of linear networks trained with forced Hebbian learning is as associative memories. We saw that associative memories work with principles similar to those of human memory, since the memory is contained in the interconnection weights (pattern of activity). They are content addressable (it is enough to input the data to get the recall), unlike computer memories, which require an address to retrieve the data. They are also robust to noise and to failure in the components. On the other hand, they have limited storage.

We also presented other interesting views, such as linking supervised and unsupervised learning. We pointed out the fact that LMS can be thought of as a composition of forced Hebbian and anti-Hebbian learning, which shows that the learning rules studied so far explore only correlation about the input patterns (or second-order statistics about the data clusters).

Hebbian networks are very useful in many engineering applications, and they train rather quickly, so they are well suited to on-line applications.

Go to next section


 6.14 Exercises

6.1 Think of another way to limit the weight growth in Hebbian learning. Specify a network to implement your rule.

6.2 Formulate the Hamming network in terms of a Hebbian network (as implemented in Example 6.3).

6.3 For the Example 6.5 go to the function generator and change the period to 1,000 samples. Also make the input sine waves 90 degrees out of phase. Now run the network and explain what you observe. How can you make the weights follow the input all the time? What does this tell you about forgetting?

There are two stable positions for the weights. Can you describe them? Now go to the second panel of the example and see the difference when you schedule the step size.

6.4 Compare the performance of the PCA-MLP combination with that of the perceptron in Chapter 3 when you add noise to the input. Explain why the system is now much more robust to noise and show that the more PCA components the less robust the system is. (Note: You have to place the noise component directly over the input file for proper operation).

6.5 In the anti-Hebbian Example 6.9 change the Hebbian component to an Oja component and observe the difference. Can you explain it? See Eq. 6.13.

6.6 Compare the adaptation rule of the whitening transform with Sanger's rule.

6.7 Consider the following names:

HOPFIE

WIDROW

FREEMA

HAYKIN

VAPNIK

DGHEBB

WERBOS

MCPITT

RBLATT

Code each letter into the last 5 bits of the corresponding ASCII code and create an associative memory with the outputs 1 to 9. Evaluate the crosstalk experimentally in the following cases: Represent 1 by 1 and 0 by 0. Then represent 0 by –1. Compare the results.

What is the capacity of your LAM? Input the word WIENER and look at the result. Try also FISHER. What can you conclude?

6.8 Show in a vector diagram why Eq. 6.39 reduces crosstalk (compare also with Fig. 6.15).

6.9 Relate the power field with the performance surface of the Adaline and then explain the relation of LMS with Hebbian learning.

6.10 What is the difference between the autoassociator trained with backpropagation and the PCA network? Which one provides the smallest MSE? Using the breadbaord of Example 6.7 place the L2 criterion to help compute the error. Explain these results in view of the theory.

6.11 Compare the OLAM with a linear regressor with the same number of inputs and outputs.

6.12 Do you see any advantage of using a nonlinear associative memory?

Go to next section


 6.15 NeuroSolutions Examples

6.1 Training with the Hebbian Rule

6.2 Directions of the Hebbian Update

6.3 Signal Detection with Hamming Networks

6.4 Instability of Hebbian Rule

6.5 Oja's Rule

6.6 Sanger's Rule and PCA

6.7 PCA for Preprocessing

6.8 Anti-Hebbian Learning

6.9 Stability of Anti-Hebbian

6.10 Forced Hebbian Computes Cross-Correlation

6.11 Novelty Filter with Anti-Hebbian Learning

6.12 LAM Application

6.13 LAM and Crosstalk

6.14 Optimal LAMs

6.15 Autoassociator and PCA

6.16 Autoassociator and Pattern Completion

6.17 Nonlinear Associative Memories

6.18 Data Compression with PCA

6.19 OLAMs and Arbitrary Connections

Go to next section


 6.16 Concept Map for Chapter 6

Go to Next Chapter


 7.10 Modular Networks

In earlier chapters we described multiple PE network topologies that were trained in cooperation, that is, all the PEs were trained at the same time for the common goal of solving the problem. Competition and its winner-take-all function changed this view, because now only part of the network is responsible for mapping a portion of the input data space. So far the competition has been limited to the learning rule, not to the network topology.

This view of competitive learning can be brought also to the level of network topology to create what has been called modular networks. Modular networks are built from expert networks, all of which work with the same input and are coordinated by a competitive gating network (Figure 7-12).

Figure 7-12 A modular network

The output of the modular network is

(7.21)

where gi is the gate output. The gate can either receive information from the input or from the performance of each expert network. In the case shown in Figure 7-12, the role of the gate is to find out which weighting should be given to each expert contribution, in other words, what is the conditional probability that the present sample is produced by the model of expert k. Statistical reasoning can therefore be used to train such networks as is done in the mixture of experts.

Alternatively, we can use a hard partitioning (0/1) for the gate, and we are in the realm of competitive learning. In this case it is easier to have the gate monitor the output of each network and adapt the one that has the best performance.

NEUROSOLUTIONS EXAMPLE 7.14

Modular Network

This NeuroSolutions example implements hard competition in the output space by modifying the learning criterion. The problem is to create a network that will learn how to approximate a V shaped function in the input space. But instead of using a single network, as we demonstrated in Chapter 5, here we use two linear networks (called expert networks, since they specialize in a piece of the input space). We know that an MLP can solve this problem, but now we are using multiple linear networks to do a piecewise linear approximation to the functional mapping.

At the input we have a ramp generator, and at the output we use the triangular wave as the desired response. We use two linear regressors, which can only scale and displace the ramp. However, we train the two networks in tandem by using a very clever rule. If network 1 provides a smaller error for the present sample, only the weights of network 1 will be adapted. Likewise, when network 2 produces the smallest error, only its weights get adapted. We have created this simple rule in the DLL of the error criterion. Train the network and see how quickly the network learns to assign each piece of the V to a single expert network.

NeuroSolutions Example

The use of competitive learning to create expert networks should be contrasted with committees of networks. In committees we use several networks to cooperatively solve a problem, but here we are creating competition among the networks to force them to learn something different. This idea is very powerful and leads to practical solutions of many problems.

Go to next section


 7.11 Conclusions

Both the Hebbian learning networks from Chapter 6 and the competitive networks from this chapter can be used for data representation. The major difference between these two unsupervised methods is that Hebbian learning tends to extract information globally from the input space, whereas competitive networks are used to cluster similar inputs. Hebbian networks are associators, while competitive networks compete for resources. The competition in these networks allows them to "divide and conquer" the problem without the need for global control. Each PE responds to the inputs that are nearest to it. With hard competition, only the closest PE is active (and thus has its weights updated). With soft competition the closest PE is active, and its neighbors are also active, although somewhat less than the winning PE. This allows for a topological mapping of the input where neighbors in the input space are mapped to neighbors in the output space.

Go to next section


 7.12 Exercises

7.1 Extend the breadboard of the LAM in Example 6.14 with a winner-take-all network. First train the network and then test the combined network in a single-step mode (to make sure that the inputs are stable). This works well but limits how fast you send new inputs in, that is, it is not a static operation. Change the constants in the winner-take-all network to make it work as fast as possible.

7.2 Provide details to implement competition with an L1 norm (see Eq. 7.6). Work with Example 7.3 (choose the boxcar metric in the competitive synapse).

7.3 What input space characteristics do you lose when you do clustering? For instance, can you determine distances among clusters precisely? What do you gain with clustering?

7.4 Draw three different data clouds to illustrate the following cases: (1) clustering is equivalent to classification; (2) clustering is harmful for classification. Elaborate on the second case. How would you minimize the chances of this happening in practice?

7.5 In your opinion, what is the worst limitation of the clustering algorithms described in sections 7.4 and 7.5. Illustrate your point with the Voronoi diagram of Example 7.7.

7.6 Do you see any problem with the conscience term explained in Eq. 7.10?

7.7 Explain how you would use a Kohonen network in high-dimensional data visualization.

7.8 Example 7.11 uses too high a learning rate for demonstration purposes. Set these parameters without this constraint so that they always provide reliable topographic maps.

7.9 During Kohonen learning of the data in Example 7.9 the weights always converge toward the origin of the space in the initial stages of learning. Explain why.

7.10 Experiment with the breadboard of Example 7.14 using other data sets.

Go to next section


 7.13 NeuroSolutions Examples

7.1 Winner-Take-All Networks

7.2 The Instar Network

7.3 Competitive Learning

7.4 Winner Metric and Performance

7.5 Clustering and Voronoi Tessellation

7.6 Competition with Conscience

7.7 Performance and the Number of Clusters

7.8 Effect of the Mexican Hat

7.9 Kohonen Linear Neighborhood

7.10 Kohonen 2-D Neighborhood

7.11 SOM Topological Ordering

7.12 Learning Vector Quantization

7.13 Counterpropagation Networks

7.14 Modular Network

Go to next section


 7.14 Concept Map for Chapter 7

Go to Next Chapter


 8.10 Types of Linear Filters

We can design [image] to perform a desired operation on the input signal instead of simply finding out what the linear system did to the input. This is the basic idea of filtering. For instance, let us assume that our signal is contaminated by high frequency noise. High-frequency noise can be recognized as a small, fast changing, ripple on top of the waveform. One way to remove the noise is to design a linear system that will let our signal go through undisturbed yet attenuate the noise. Linear systems with prespecified frequency responses are called filters. filter shape factors

As an example, let us use the input and filter as shown in Figure 8-18. A filter that attenuates high-frequency noise is called a lowpass filter (because it multiplies low frequencies by numbers close to 1 and high frequency by very small numbers), and the magnitude of its frequency response should ideally be like the bottom middle graph of Figure 8-18.

Figure 8-18 The effect of a lowpass filter on a noisy time series

Observing the magnitude spectrum of X(k) we see that the energy falls off with k (toward the high frequencies) but that there is an increase of energy close to fs/2, which is produced by high-frequency noise.

The lowpass filter frequency response is flat (magnitude 1) until the sample kc, which corresponds to the cutoff frequency. This is an ideal filter that multiplies frequencies above kc by zero. The spectrum of the output signal, |Y(k)| is easy to construct since it is a product of the input spectrum |X(k)| and the frequency response |H(k)|. Notice that for frequencies below kc, |Y(k)| is identical to |X(k)|, but for frequencies above kc it is zero. The corresponding time signal y(n) is smoother but identical to x(n). Unfortunately, this ideal lowpass filter (called the boxcar filter for obvious reasons) is not realizable in practice, but we can approximate its response fairly well.

We can also conclude from this discussion that linear filtering works only if the signal spectrum and the noise spectrum do not overlap, otherwise, to attenuate the noise, we also attenuate the signal. This is where other, more sophisticated filtering procedures or nonlinear systems are necessary.

NEUROSOLUTIONS EXAMPLE 8.16

Lowpass FIR Filter

Let us implement a lowpass FIR filter and show that the signal in the passband is transmitted without attenuation, while in the stopband is attenuated. Add high frequency noise to show that it can be separated, whereas if the noise is white, there will be always noise at the output.

NeuroSolutions Example

The traditional types of filters (lowpass, highpass, bandpass, and bandstop) are shown in Figure 8-19. As the names indicate, lowpass and highpass filters accentuate the low and high frequencies respectively; a bandpass filter lets a narrow band of frequencies pass and attenuates all the other frequencies; and the bandstop filter (also called a notch filter) attenuates only a narrow band of frequencies.

These transfer functions can be achieved by appropriately choosing the order and the filter coefficients in Eq. 8.9. There are well accepted procedures to design digital filters to meet attenuation and frequency requirements (any book on digital signal processing addresses filter synthesis), but we will not pursue these issues here. We will design filters through optimization, that is, by providing the desired response and letting the linear combiner find the best set of weights to meet the requirements.

Figure 8-19 The four most common filter types

Go to next section


8.11. Project: Design of Digital Filters

 8.11 Project: Design of Digital Filters

Linear filters are very useful as a preconditioning stage. Data collected from the real world is almost always corrupted by noise of various kinds, so the ability to disregard parts of the signal spectrum is very important for good results. We present a very simple design of FIR filters that can be easily accomplished by hand, without the need for filter design programs. Filter design packages provide excellent filter characteristics through optimization, but they are recipes that do not provide the insight. By going through the frequency-sampling design you can develop a better understanding of filters (see Principe and Smith).

The idea is to work with the equation

(8.25)

which can be written in the Z domain as

(8.26)

The zeros of this equation can be recognized as the Nth roots of unity, which means that the zeros of the linear system are equally spaced along the unit circle. The frequency response is obtained as

(8.27)

i.e. the modulus of a cosine function (also called a comb filter).

This is very interesting, since with just one addition and the use of memory we created N zeros. Potentially, any of the lobes of the cosine function can be used as a passband for a bandpass filter. The first lobe (which peaks at w = 0) can be used for a lowpass filter.

NEUROSOLUTIONS EXAMPLE 8.17

Creating the Base Function for Filtering

The first step in this filter design is to create the function G(z), which provides many possible passbands. In this example we experiment with the function G(z)=z¯³+1. We always provide the impulse response and the corresponding magnitude of the frequency response. Two things to look for are the frequency where the function has the peak, and what is the –3dB bandwidth. The peak is always obtained midway between the zeros, so the trick is to find the zeros of the equation G(z). The –3dB points are roughly half the distance between zeros. Verify these guidelines in the breadboard. You should change the number of taps in the memory element. You can also change the values of the weights. In particular implement Eq. 8.26 with a subtraction instead of an addition.

NeuroSolutions Example

This transfer function is still not a usable filter, except as a periodic notch filter, because all the frequencies [image] = k [image] s/2N are eliminated.

To create low, high or bandpass filters, we have to select a region of frequency as the passband, and all the other frequencies should be the stopband, so we have to eliminate some of the multiple lobes created by Eq. 8.25.

This can be done if we place zeros at exactly the locations where the unwanted peaks of Eq. 8.27 exist, since this will make the overall gain at these frequencies equal to zero. We can accomplish this simple idea by cascading other FIRs to the network that creates Eq. 8.27. The simplest case is to cascade a single zero at z = 1:

(8.28)

The frequency response of (1 [image] z [image]) is as shown in NeuroSolutions, so we can expect that a bandpass filter is obtained around [image] s/3.

NEUROSOLUTIONS EXAMPLE 8.18

Cascading Stages to Create Different Filters

Here we implement the new transfer function G1(z) = (1-z [image]) and then cascade it with the original G(z). G1(z) peaks at fs/2 (or [image]), since it has a zero at the zero frequency. It is therefore a highpass filter. When we cascade the two functions together, we are multiplying their frequency responses, so G1(z) selects the peaks of G(z) that are going to have a larger value.

NeuroSolutions Example

This example provides the basic concept for the design of the frequency-sampling class of filters. First, we create a base function with many lobes, one at the frequency where we want to place the passband of our filter. Then we eliminate the unwanted lobes by placing zeros at the locations where the unwanted lobes have maximum amplitude. These zeros are implemented by networks that will be cascaded over the original filter. It is simpler to work with second or first-order FIRs, which create a pair of complex zeros or simply a real zero. They knockout pairs of peaks or a peaks at dc or [image] s/2. We are ready now to learn how to choose N and the coefficients of cancellation networks.

8.11.1 Bandpass Filter Design

It is easy to show that the peak frequencies of Eq. 8.27 occur at

(8.29)

The bandwidth of a lobe is

(8.30)

Notice that the center frequency and the bandwidth are interrelated by these two equations ( [image] p = 2kB), so narrowband filters require high orders.

Knowing [image] p and the bandwidth B of the desired filter, we have to select k, [image] s (above the Nyquist rate to avoid aliasing), and N. Since these filters have smooth transition bands, we should not try to find an exact fit. First find a good compromise of an integer ratio between the bandwidth and center frequency, and then the design is easy.

The bandwidth is the ratio of [image] s/N, so we select the smallest [image] s above the Nyquist rate that satisfies [image] s = 2NB. From this we can easily choose k in Eq. 8.29.

Now we have to eliminate the lobes that do not correspond to the passband of the filter. To do this, we have to design second-order (and eventually first-order) FIR filters to eliminate the remaining lobes of Eq. 8.26.

The equation for a system with a pair of zeros on the unit circle is given by

(8.31)

which is called an anti-resonator. The angle should coincide with the unwanted peak lobes,

(8.32)

The total filter order is 2N-2. The bandpass filters generated by Eq. 8.26 require a first order system with a zero at z = 1, and, for N even, a pair of zeros at –1 and 1.

There is a function very similar to Eq. 8.26 that also generates bandpass filters: just substitute the plus sign by a minus sign.

(8.33)

This effectively rotates the zeros in the unit circle by [image] /N, so everything we said before is applicable with minor modifications to the formulas. The formula for the bandwidth is the same, but the center frequency is now given by

(8.34)

and the anti-resonators should be placed at

(8.35)

For N odd, an extra first-order FIR with a zero at z = -1 must be added. To choose which of these two families we need to use, just substitute the center frequency and bandwidth in the equation

If [image] [image] = 0 choose Eq 8.26, otherwise choose Eq. 8.33.

Note that this design is linear phase; that is, the phase response continuously increases (after unwrapping). We can continue the addition of zeros midway between the stopband zeros to improve the filter-shape factor.

As a design example, suppose that we want to design a bandpass filter with a center frequency of 150 Hz, bandwidth of 120 Hz, when the minimal sampling frequency (Nyquist rate) is 500 Hz.

First lets ask the question, if we can live with a bandwidth of 150 Hz for this application. If the response is affirmative, [image] s = B and N = 4 which gives a sampling rate of 600 Hz. Since K = 0 and [image] [image] = 1

(8.36)

The anti-resonator has to cancel the second lobe of the frequency response, so [image] =135, 2cos (135)= -1.414, which can be approximated by

(8.37)

The final filter is

(8.38)

NEUROSOLUTIONS EXAMPLE 8.19

Bandpass Filter Design

In this example we perform the calculations outlined in the text. Note that each of the individual filters is very simple. We can easily enter by hand the coefficients of the filters in the linear combiner. First we create the second-order anti-resonator of Eq. 8.37 just to visualize its frequency response. Then we create the base function of Eq. 8.36 and cascade the two together. Notice that the combined frequency response is the product of the individual frequency responses. We recommend thinking in terms of the zero plot.

We have created a more complex filter by creating more individual filters. Alternatively, we could have cascaded more filters (the preferred method). You should develop the zero plot and figure out why the frequency response looks the way it does. Note also that the stopband attenuation is improved but that the passband is still very much like a lobe of a sine wave, so flatness in the passband was not improved.

NeuroSolutions Example

8.11.2 Design of Lopwass and Highpass Filters

A lowpass filter is a bandpass filter centered at zero frequency, so only Eq. 8.26 can be used. In the case of the lowpass the only design parameter is the bandwidth, so N and [image] s should be chosen to meet Eq. 8.29. The other lobes of G(z) can be canceled by anti-resonators as done before.

A highpass filter in the digital domain is a filter centered at [image] s/2 (since this is the largest frequency that can be represented in the digital domain), so only Eq. 8.33 can be used to create G(z).

We can design this type of FIR filter using very simple concepts. We can even avoid multiplications alltogether if we approximate the scalar in the anti-resonator by descending powers of two. This will affect the ripple in the stopband because we will not exactly cancel the peak of G(z).

One other aspect that we would like to demonstrate is the opposite forces of poles and zeros to shape the frequency response. We saw that poles provide gain, while zeros provide attenuation. For instance if in the filter G(z) = 1 [image] z¯4 we cascade a first-order IIR filter with a pole at z = 1, what frequency response are we going to get?

The pole at z = 1 will cancel the zero at z = 1, so basically the filter will end up with a larger gain at [image] = 0, creating effectively a lowpass filter. The problem is that the pole-zero cancellation is dangerous, since if the cancellation is not perfect, the system may become unstable.

NEUROSOLUTIONS EXAMPLE 8.20

Pole Zero Cancellation

This example exemplifies pole-zero cancellation. Instead of creating FIR filters by adding many first and second-order filter sections, we may want to exploit pole-zero cancellation. For instance, if we create the base filter section with many zeros (80), we end up with a 80 zeros uniformly spaced along the unit circle. If we want to create a lowpass filter, we need to place second-order anti-resonators in all the 39 peaks of the function. This is hard work. Instead, we may decide to cancel the zero at z = 1 with a pole at z = 1. Such a function has a transfer function H(z) = 1/(z-1). The combined system has a very large gain at z = 1, that is, a lowpass filter. We demonstrate this simple design on a piece of music.

NeuroSolutions Example

Go to next section


 8.12 Conclusions

In this Chapter we covered many new concepts that are the foundations of digital signal processing. As we said in the beginning, this chapter is not intended to substitute an in-depth study of DSP, but simply to present some of the most fundamental concepts of the field to appreciate the differences between static and time signals. More important, the discussion presents the tools of the trade. We hope that the reader is able to understand why DSP uses the tools that it does. Unfortunately, to become an expert in them the reader will have to consult the many excellent books written on the subject Oppenheim or Haykin and Van Veen.

We started with the important observation that discrete signals of finite length can be put into a one-to-one relationship with vectors. This is rather important because it allows us to think of filters as projection operators. Although interesting, this perspective does not offer an easy quantitative analysis of what happens to the signal characteristic when it passes through filters. But time domain or frequency domain analysis permits a precise quantification of the effects of filtering. We developed both approaches. We showed that frequency domain analysis has advantages when dealing with filters. It provides easy interpretation (pointwise operations) of the effect of filtering, but it requires the introduction of more concepts such as the spectrum. Once we get over the daunting name, however, we find that the spectrum is just another decomposition of a time signal, now using phasors. A phasor is a rotating unit vector of a given frequency. The spectrum shows how the energy in the signal is divided in frequency bins.

With these concepts we can design filters that achieve a predefined distortion of the input signal. One might wonder why we want to distort a signal, but when we think of additive noise, we can see the need for filtering as a spectral shaping tool. We showed how to design FIR filters with a very simple approach.

Go to next section


 8.13 Exercises

8.1 Explain in your own words the differences between analog and digital signals.

8.2 If a continuous sine wave signal has frequency f0 = 100 Hz, what is the minimum sampling frequency to avoid aliasing? If you sample at fs = 170 HZ can you tell how the digital signal looks like and what is its analog frequency after sampling?

8. 3 If you have the signal

and you pass it through the system [image] (n-3), what do you obtain at the output?

8.4 Construct the delay line with 3-taps followed by a PCA network suggested in Example 8.1. Put a scope at the output of the first PE output to see what happens to the noise. Explain the result.

8.5 In Example 8.3 find other weight vectors that attenuate either the noise or the sine wave at the output. You should present a geometric explanation for your choices.

8.6 For Example 8.4 with the two sine waves, why can we not attenuate the high frequencies better? Go to a higher-order filter (N = 10) and design a lowpass filter. Compare the two results, and explain why.

8.7 In Example 8.8 convolve the square wave with the triangular wave. Do the calculations by hand and compare with the NeuroSolutions results. Repeat with waveforms of your choice.

8.8 Example 8.9 is very useful. You should study it thoroughly. Here we study the performance in terms of the threshold. Start with a low threshold and create a plot of the number of correct detects (hits) versus the number of wrong detections (false positives). Then raise the threshold until no event is missed. This curve is the Receiver Operating Characteristics (ROC) curve. Plot the ROC for the matched filter.

8.9 Compute the impulse response of the second-order system

Is the system stable? Go to Example 8.11 and check your answer.

8.10 Show that the phasors [image] form an orthonormal set. Use the definition of the inner product.

8.11 Compute the N point DFS of the following signals

8.12 For the signals of Problem 11, set a value for N and use Example 8.12 to check your answers.

8.13 Using Example 8.12 show that the highest frequency you can represent is N/2, which is placed right in the middle of the display. Do the same for the lowest frequency that you can reliably determine from a DFS with N samples.

8.14 This problem explores spectral representations. Go to the first panel of the breadboard of Example 8.12 and do the following: In the Signal Generator Inspector, keep the period at 32 and change the Signal Generator to a sine wave, a triangular wave, and square wave. Then observe the spectra. How is the smoothness of the time signal related to spectral components?

8.15 Repeat Problem 14 but now use T = 128 samples, and compare with the previous spectra. How does the number of points per period affect the DFS?

8.16 The impulse response of an FIR is h(n) = 1, 1, 1. What is the transfer function? For the FIR h(n) = 1,-2,1 what is the transfer function? We found both of these systems early in Chapter 8. Can you now quantify better their filtering properties?

8.17 Evaluate the frequency response of the two filters in Problem 16. Go to Example 8.13 and verify your answer.

8.18 Determine the poles and zeros for the two systems of Problem 8.16.

8.19 Go to Example 8.15 and run the filter of order 3 with weights [0.3, 0.3, 0.3]. Now change the order of the filter to 10 with weights [0.1,…..,0.1]. Explain the results in terms of filtering and frequency response.

8.20 Filter the music file with a 16th order highpass filter implemented with the sampling method described in the project of Example 8.20.

Go to next section


 8.14 NeuroSolutions Examples

8.1 Signal Space

8.2 Signal Space and the Delay Line

8.3 Linear Combiner in Signal Space

8.4 Filtering in Signal Space: The Two Sine-wave Case

8.5 A Triangular Wave in 3-D Space

8.6 Square Wave in 3-D Space

8.7 Impulse Response of an FIR Filter

8.8 More on the Impulse Response of an FIR Filter

8.9 Matched Filtering

8.10 IIR Filters

8.11 Second Order Recurrent System

8.12 The Fast Fourier Transform (FFT)

8.13 Analysis of the Output in the Frequency Domain

8.14 Minimum Phase Systems

8.15 Frequency Response of the Second-Order System

8.16 Lowpass FIR Filter

8.17 Creating the Base for Filtering

8.18 Cascading Stages to Create Different Filters

8.19 Bandpass Filter Design

8.20 Pole-Zero Cancellation

Go to next section


 8.15 Concept Map for Chapter 8

Go to Next Chapter


 9.10 NeuroSolutions Examples

9.1 Design of Filters Using Adaptation

9.2 Adaptive Filters for Prediction

9.3 Choosing the Step Size

9.4 Speed of Convergence

9.5 Misadjustment

9.6 Tracking Ability

9.7 Prediction of Time Series

9.8 Prediction for Modeling

9.9 Prediction for Spectral Analysis

9.10 System Identification

9.11 Inverse Modeling

9.12 Inverse Modeling of Nonminimum Phase Plants

9.13 Interference Canceling

9.14 Adaptive DC Removal

9.15 Real Example of Noise Canceling

9.16 Finding Periodic Signals in Noise

9.17 Echo Cancellation

9.18 Adaptive Inverse Control

9.19 Maximum Eigenfilter

9.20 Optimal Filter Bank

9.21 Noise Removal by PCA

9.22 Spectral Analysis by Eigendecomposition

9.23 Blind Source Separation with Anti-Hebbian Learning

Go to next section


 9.11 Concept Map for Chapter 9

Go to Next Chapter


 10.10 Project: Iterative Prediction of Chaotic Time Series

Neural networks are well suited for the prediction of time series produced by nonlinear models, which we call here nonlinear time series. The idea is to extend the prediction model discussed in Chapter 9. The block diagram of the linear combiner is replaced with a focused TLFN trained with backpropagation in a single step prediction framework. One important consideration is to choose the size of the projection space appropriately. For predicting nonlinear time series produced by deterministic systems Takens embedding theorem can be used to help in this definition. Basically, the theorem states that we have to find the size of a reconstruction space where the trajectories do not cross. We can estimate the dimension of this space, but the algorithms are difficult to work with; see Abarbanel.

As an alternative, we can experiment with the size of the delay line and the value of the delay (i.e. multiple delays) using the simulator, as we experimented with the size of the MLP topology in Chapter 3. The results of nonlinear predictors frequently outperform the prediction with linear models, but both yield reasonable approximations.

Prediction can also be thought as the first step toward dynamic modeling, where the overall goal is to identify the dynamic system that created the time series. Once the predictor is trained, a dynamic model can be obtained by feeding back the output into the input (delayed by one sample). This creates an autonomous system as Figure 10-21 shows. Can the preceding prediction methodology create a good dynamical model? Our model is considered good if it can create in autonomous mode a time series with the same characteristics as the original time series (see Haykin and Principe).

Figure 10-21 Iterative prediction for dynamic modeling

When we attempt to use the linear system in autonomous mode, the results are disappointing, since the output quickly approaches zero (or oscillates). With the focused TDNN the response does not normally decay to zero, but hardly ever resembles the original time series. So how can we design improved dynamical models?

An important modification to the overall methodology of signal prediction is to introduce the concept of iterative prediction. Notice that until now we always used the time series as the input to the neural network. However, when we want to generate the time series from our trained model, we are using the most current output of the model as input (Figure 10-21). This arrangement is called iterative prediction. We first seed the predictor with one or more point in state space, compute the output and feed it back to the input for a number of samples. Try this in NeuroSolutions, and experiment with the parameters to see the effect in the training MSE and in the quality of the generated time series.

NEUROSOLUTIONS EXAMPLE 10.21

Iterative Prediction with Focused TLFNs with Feedback

In this example we provide a little more detail tothe problem of modeling a chaotic time series. This is important because neural networks are posed to play an important role in this area. We are going to demonstrate that not only can TLFNs predict the chaotic signal, as we saw in the previous examples, they can also learn the dynamics from the time series and produce a system that can generate the SAME signal. In a sense, they have identified the parameters of the system that produced the time series, and they can be used in place of the original dynamic system.

Signal generators are important, and those most widely used in engineering are the oscillators. But a linear system with constant parameters cannot autonomously generate waveforms more complex than a sinusoid. This is because the transient dynamics of a linear system either go to a point attractor or a limit cycle. The limit cycle produces the sinusoid.

Let us now consider the MG30. This is a very complex waveform. How can we build a system that produces such a waveform? There is a trivial way: Input white noise into a linear system and keep on adapting the parameters. This will provide a waveform very similar to the MG30, but notice that the source of complexity is in the WHITE NOISE input. White noise potentially contains all wavforms, so by filtering, we can obtain the one we want.

The second (more difficult) way to generate the MG30 is to use a system WITHOUT an external input. Such a system is called autonomous. Feeding the output of the system to itself creates a signal generator. Can we design an autonomous system that produces the MG30? We can immediately rule out linear systems. Hence we must have a nonlinear dynamic system. It turns out that dynamic neural networks can be taught to generate complex (even chaotic) time series such as the MG30. This is what we demonstrate in this example.

We start with the linear combiner. We can train it to a very small prediction error, but when we fix the weights and feed back the output to the input, the output decays to zero (or to a limit cycle). The autonomous dynamic regimes of a linear system are not very exciting.

Now let us do the same thing with the TDNN. We train it to a very small prediction error. Then we fix the weights and feed the output back to the input. Now, the output resembles the MG30. In fact the error (compared with the signal it has been trained with) is very small up to sample 50, and then the error increases (and depending upon the training may go back down). But the two waveforms are very similar visually. Let us take an FFT of each to see that their spectra match almost exactly. We have created a system that behaves just like the MG30.

The appeal of this method is that we never use the information about the equation. We just picked a time series and used it to train the TDNN. In principle this technique can therefore be used to create synthetic models from real-world signals. This application is called dynamic modeling.

Experiment with the size of the TDNN and see when it fails to produce a waveform similar to the MG30.

NeuroSolutions Example

Go to next section


 10.11 Conclusions

This chapter brought together the two fundamental concepts of nonlinear topologies and time, which were independently developed in Chapter 3 and Chapter 9 respectively. The MLP is a very powerful mapper but it works only with static patterns. On the other hand, the linear combiner can work with time signals but its mapping ability is reduced since it can only perform linear approximations. Combining a memory structure with the MLP gave rise to the class of dynamic networks known as time lagged feedforward network (TLFN).

TLFNs are powerful mappers and have the advantage that they can be trained with static backpropagation, provided that the memory layer is placed at the input, a topology called focused. However, static backpropagation can NOT train the feedback connections that may be present in TLFNs, so they have to be manually selected. The next chapter will deal with the training of TLFNs and fully recurrent systems.

This chapter introduced the concept of the memory PE as a special PE configured to process time signals. In a sense, the memory PE brings the representation of time to the inside of the neural topology, with the advantage of internal (and hopefully better) control of time representations. We quantified the properties of memory PEs and gave the interpretation of the memory PE and its connection weights to the other network PEs as a filter.

We also presented the two fundamental connectionist memory mechanisms, memory by delay and memory by feedback, and showed that the gamma memory is an unifying principle in this perspective. However, the gamma memory can still be extended to other memory mechanisms that extract other information from the time series (the gamma II memories). This important chapter opens up the possibility of using nonlinear systems for time processing. The major hurdle that we still need to conquer is how to train arbitrary topologies built from memory PEs and nonlinear PEs.

Go to next section


 10.12 Exercises

10.1 Can you see any advantage in bringing memory into the neural network topology?

10.2 Is there any difference between using the focused TDNN and applying a time window of the same size to the input data?

10.3 Experiment with Example 10.4. Increase the size of the linear system and see how it affects performance. You can also implement one of the preprocessors of Chapter 6 or 7 to see if it helps the performance of the neural network.

10.4 Use the Example 10.5 with the Lorenz time series, available in the file prob4 of the chapter10 folder in the data directory (CDROM). Experiment with the size of the delay line and the size of the MLP for better results. Present the learning curve. How can you decide if the MSE is acceptable or not? Does the MSE normalized by the power of the input make sense to you?

10.5 Analyze from the DSP point of view the advantage of using a memory structure at the input of the M-P PE in Figure 10-5.

10.6 Compute the memory depth and resolution for a D = 20 tap delay line PE and for a context PE with [image] = 0.01. Provide two examples where you would prefer one over the other.

10.7 What do you gain over the tap delay line when you use a gamma memory? Make your point clear by analyzing a time series where the information is mixed over 25 samples, but only four parameters are sufficient to represent the system that created the time series.

10.8 State at least one disadvantage of the gamma memory by analyzing Figure 10-18. How could you compensate for it?

10.9 You should study Example 10.9 very well because it provides an inside look of how a TLFN works. In particular, you should try to answer this question: In principle the memory filters that separate the two regimes should be lowpass and an highpass filters, but the solution found through adaptation is neither one. Can you explain why?

10.10 Provide an example for which the TLFNs are not a good mapping model. Explain your reasons.

10.11 State one advantage of the combination gamma-RBF over tap delay line- RBF. Indicate also an advantage of the combination tap-delay line- RBF over the tap-delay line-MLP. Also provide a disadvantage for each combination.

10.12 Explain in your own words the difference between static networks and dynamic networks. Write a small essay about the difference for speech recognition.

Go to next section


 10.13 NeuroSolutions Examples

10.1 Time Disambiguates Data Clusters

10.2 Disambiguating Data with the Tap-Delay Line

10.3 Temporal Pattern Recognition with the Focused TDNN

10.4 Nonlinear System Identification with the Focused TDNN

10.5 Nonlinear Prediction with the Focused TDNN

10.6 Context PEs Revisited

10.7 Memory Depth of the Tap-Delay Line

10.8 Memory Depth of the Context PE

10.9 Interpretation of Focused TDNNs as Nonlinear Memory Filters

10.10 Gamma Filter for System Identification

10.11 Increasing Versatility for TLFN Architectures

10.12 Phoneme Classification with Focused TLFNs (Context PEs)

10.13 Nonlinear System Identification with Focused TLFNs (Context PEs)

10.14 Nonlinear Prediction with Focused TLFNs (Context PEs)

10.15 Phoneme Classification with a Focused TLFN (Gamma Memory)

10.16 Nonlinear System Identification with a Focused TLFN (Gamma Memory)

10.17 Nonlinear Prediction with a Focused TLFN (Gamma Memory)

10.18 Phoneme Recognition with a Focused RBF-TLFN (Gamma Memory)

10.19 Nonlinear System Identification with a Focused RBF-TLFN (Gamma Memory)

10.20 Nonlinear Prediction with a Focused RBF-TLFN (Gamma Memory)

10.21 Iterative Prediction with Focused TLFNs with Feedback

Go to next section


 10.14 Concept Map for Chapter 10

Go to Next Chapter


 11.10 Hopfield Networks

Hopfield networks are a special case of recurrent systems that use threshold PEs, do not have hidden PEs, and where the interconnection matrix is symmetric. The original Hopfield model did not have self-recurrent connections (Figure11.13).

Figure 11-13 The Hopfield network

We can describe the discrete Hopfield net in discrete time as

(11.32)

where sgn represents the threshold nonlinearity (-1,1) and b is a bias. We assume that the update is done sequentially by PE number. The input is a binary pattern x = [x1, x2, … xN] [image] and works as an initial condition; it is presented to the network and then taken away to let the network relax so it disappears from Eq. 11.32. Due to the sign function we can see that the points visited during relaxation are the vertices of a hypercube in N dimensions. For simplicity the bias terms are set to zero in the following discussion.

Let us analyze the function of such a network for pattern association. Suppose that we have a binary input pattern xp, and we want this input pattern to be stable at the output, that is, we want the system to be an autoassociator and produce a stable output pattern y = xp. The condition for a fixed-point implies

(11.33)

because at this point the forward equation (Eq. 11.32) produces no change, i.e. the system is at a fixed point, also called the attractor. It can be shown (Hopfield) that to meet this condition, the weight matrix becomes

(11.34)

or in words, the outer product of the patterns automatically computes the weights of the Hopfield network without the need for any learning laws. Alternatively, we can use the Hebbian learning to create the weight matrix, just as with the static associative memory.

The weight matrix W is a symmetric matrix. But notice that this system is dynamic: when we present the input x to the network as an initial condition, the system dynamics produce an output sequence y(n) that takes some time to stabilize but will approximate the stored pattern.

NEUROSOLUTIONS EXAMPLE 11.12

Hopfield Network Dynamics

This first example with Hopfield networks shows the dynamics of the system. We have created a 3-input, 3 output Hopfield network with two point attractors at the vertices of the unit cube (1, -1, 1) and (-1, 1, -1). Just examine the weight matrix to see that they comply with Eq. 11.34.

Each of these attractors creates a basin of attraction. The input sets the initial condition for the system state. Hence the system state evolves toward the closest attractor under the "force" of the computational energy. This force is rather strong, as we can see by the "acceleration" of the system state.

Try inputs that will put the system state close to the boundary of the two bases (here the boundary is the plane x- y + z = 0). At exactly these locations the final value is zero. However, all the other values will produce a convergence towards one of the two point attractors.

NeuroSolutions Example

The attractive feature of Hopfield networks is that even when the input is partially deleted or corrupted by noise, the system dynamics still take the output sequence y(n) to xp. Since the system dynamics are converging to xp, we call this solution a point attractor for the dynamics. This system can also store multiple patterns P with a weight matrix given by

(11.35)

Just as we saw in the feedforward associative memory, there is crosstalk between the memory patterns, and the recall is possible only if the number of (random) input patterns is smaller than 0.15N, where N is the number of inputs. For the case of very large N the probability of recall (99 percent of the memories recalled with an asymptotic probability of 1) of m patterns is guaranteed only if

(11.36)

which is rather disappointing since it approaches zero even for large N. The intriguing property of the Hopfield network is pattern completion and noise removal, which is obtained by the convergence of the system state to the attractor.

NEUROSOLUTIONS EXAMPLE 1.13

Pattern Completion in Hopfield Networks

This example shows the convergence of the Hopfield network in a more realistic situation. We have created a 64-input Hopfield network to present the characters developed in Chapter 6 where we also treated the case of associative memories. We would like to show that the same behavior of association is also present in the Hopfield case.

We created a weight matrix by hand using Eq. 11.34. We can present the pattern at the input (just once) and watch the output pattern appear. This Hopfield network converges very quickly.

A more interesting observation is to show that even when the patterns are noisy or fragmented, the end result is still the full pattern. We can understand this in terms of basins of attraction. If the input places the system state within the appropriate basin of attraction, the final output will be the stored pattern, which is the complete (or noise free) character. Try adding more noise (larger noise variance) to the characters to see how much the system can handle.

NeuroSolutions Example

11.10.1 The Energy Function

We presented the practical aspects of Hopfield networks, but we assumed that the dynamics do in fact converge to a point attractor. Under what conditions can this convergence be guaranteed? This is a nontrivial question due to the recurrent and nonlinear nature of the network.

The importance of the Hopfield network comes from a very inspiring interpretation of its function provided by Hopfield. Due to the extensive interconnectivity, it may seem hopeless to try to understand what the network is doing when an input is applied. We can write the dynamic equations for each PE as Eq. 11.32, but due to the fact that these different equations are highly coupled and nonlinear, their solution seems to be beyond our reach. In practice this is not so. When the weight matrix is symmetric, the PEs are threshold nonlinearities and the biases are zero, we can show that the network accepts an energy function H (see derivation of energy function)

(11.37)

The energy function is a function of the configuration of the states {yi} that is nonincreasing when the network responds to ANY input. We can show this by proving that every time one PE changes state, H decreases. When the PE does not change, H remains the same. This means that the global network dynamics pulls the system state to a minimum (along the gradient of H), that corresponds to one of the stored patterns. The location of the minimum in the input space is specified by the weights chosen for the network. Once the system reaches the minimum (the memory) it will stay there, so this minimum is a fixed point, or an attractor.

When the Hopfield network receives an input, the system state is placed somewhere in weight space. The system dynamics then relax to the memory that is closest to the input pattern. Around each fixed point there is thus a basin of attraction that leads the dynamics to the minimum. This explains why the Hopfield network is so robust to imprecisions (added noise or partial input) of the input patterns. Once the system state is in the basin of attraction for a stored pattern, the system relaxes to the undistorted pattern. There are some practical problems, however, since when we load the system with patterns, spurious memories are created which may attract the system to unknown positions.

An energy surface with minima and a basin of attraction creates the mental picture of a computational energy landscape, which is similar to a particle in a gravitational field. This metaphor is very powerful, because suddenly we are talking about global properties of a tremendously complex network in very simple terms. We have ways to specify each connection locally, but we also have this powerful picture of the computational energy of the system. This is pictorially depicted in Figure 11-14.

Figure 11-14 Computational energy over the configuration space

Hopfield provided one of the strongest links between information processing and dynamics. The existence of the computational energy function makes the convergence of the state of the Hopfield network to the stored pattern the same dynamic problem as a ball rolling down a hill in a gravitational field (which also accepts a potential function). We can then say that a dynamic system with point attractors implements an associative memory, whether the "hardware" is VLSI, a computer running these algorithms, biological neurons, or a solar system. Hopfield's view was crucial for the revival of neural networks in 1987.

Note that the Hopfield network, although a dynamic system, becomes "static" after convergence. Therefore other paradigms must be examined to understand the temporal processing that constantly occurs in our brains. Notice that dynamic systems may have singularities higher than the zero order (point attractor), such as limit cycles and chaotic attractors.

The Hopfield network is normally used with prespecified weights. They can be computed easily for the case of associative memories. We can also apply the fixed-point learning algorithm to train a single-layer network as an associative memory. However, there is no guarantee that the network is an Hopfield network, unless the weight matrix symmetry is externally enforced.

NEUROSOLUTIONS EXAMPLE 11.14

Training a Recurrent Network as an Associative Memory

Up to now we have created the weights of the Hopfield network by hand, by loading them without training. However, we can also use fixed-point learning as the training rule to find the weights.

In general, we can no longer say that we have a Hopfield network since the weights may not be symmetric, but this should not discourage us from using such a training procedure. On rare occasions the system may become unstable, but we have workable solutions that are obtained without actually entering the weights by hand.

In this example (from Chapter 6) we use the association between long distance phone calls and their price to show that the Hopfield-like network can be used as a heteroassociative memory.

NeuroSolutions Example

Hopfield networks have also been used in optimization, because we can link the energy surface to problem constraints. The optimal solution is found by relaxing the system with the present input to find the closest solution (the attractor). Mapping the problem solution to the network is the difficult part and has to be done on a case-by-case basis (see Hopfield and Tank).

NEUROSOLUTIONS EXAMPLE 11.15

Hopfield Networks for Optimization

The convergence of the Hopfield network to point attractors can also be explored for optimization problems. Just imagine that the solution of an optimization problem is coded in the energy function of the network. When an input is given to the network, the system state evolves, providing the optimal answer to that particular instance, since the Hopfield network minimizes the energy. The difficulty is in encoding the optimization problem in the energy function, but once this is done, the simple application of an example to the network provides an output that is the desired answer.

Here we treat a simple problem: analog-to-digital conversion. Hopfield and Tank showed that we can see the problem of converting an analog signal into bits as an optimization, which can then be solved by a Hopfield network.

We restrict our example to two-digit conversions: that is, we approximate the integer input X by

X~ x1 + 2x2

where x1 and x2 will be 0 or 1. Hence the range for X is the set of integers [0, 3]. Hopfield proposed the performance index

where the first term computes the error and the second is a constraint that is minimized when the values of xi are 0 or 1. This leads to a Lyapunov function of the type of Eq. 11.35 (but extended with the bias term), which can be implemented with

The user-supplied value X is applied directly to the bias of the two PEs instead of the conventional input (as in the associative memory example). We have created a very simple breadboard and two input fields in NeuroSolutions. The user should enter 0, 1, 2, 3 in the box labeled X and the double of the first digit in the box 2X. Try several inputs and watch the network provide the result.

NeuroSolutions Example

Although very important from a conceptual point of view, Hopfield networks have met with limited success in real-world applications, mainly due to the spurious memories that limit their capacity to store patterns.

11.10.2 Brain-State-in-a-Box Model

Another interesting dynamic system that can be used as a clustering algorithm is Anderson's Brain-State-in-a-Box (BSB) model. BSB is a discrete-time neural network with continuous state (as the Hopfield with sigmoid nonlinearities). However, the equation of motion is

(11.38)

The nonlinearity is a threshold with a linear transition region. We assume here that there is no external bias and that [image] is a parameter that controls the convergence to the fixed points. The model is called "brain in a box" because such a network has stable attractors at the vertices of the hypercube [-1,+1] [image], provided that W is symmetric and positive semidefinite or [image]. We can show that the BSB behaves as a gradient system that minimizes the energy of Eq. 11.38. The weight matrix to create the memories at the vertices of the hypercube follows the Hebbian rule of Eq.11.35 where the values of xi are restricted to be ±1. we can show that the asymptotic capacity of the BSB model is identical to that of the Hopfield network.

The BSB model, however, is normally used as a clustering algorithm instead of an associative memory. Both systems create basin of attractions and have point attractors, but the BSB has faster convergence dynamics, and the basins of attraction are more regular than in the Hopfield network. Hence it is possible to divide the input space into regions that are attracted to the corners of the hypercube, creating the clustering function.

NEUROSOLUTIONS EXAMPLE 11.16

Brain-State-in-a-Box Model

The BSB model is basically a positive-feedback nonlinear system. Due to the shape of the nonlinearities, the system has to converge to one of the corners of the hypercube; that is, the system amplifies the present input until all of the PEs saturate. Hence the BSB is very sensitive to the initial pattern position in the input space, which has yielded its application in decision feedback. For the corners of the hypercube to behave as point attractors, all that is required is that the diagonal elements of the weight matrix be larger than the off-diagonal elements.

Here we have a simple 2-D example of the BSB. Enter a value in the boxes, and you will see the system relax to the closest corner. The system is similar in function to the Hopfield network, but here the location of the attractors is predefined at the vertices of the hypercube. Hence it can be used for clustering.

NeuroSolutions Example

Go to next section


 11.11 Grossberg's Additive Model

If we abstract what we know about the interconnectivity of PEs, we realize that all the neural networks presented so far have much in common. Each PE has simple first-order dynamics, it receives activations from other PEs multiplied by the network weights, adds them, feeds the result to a static nonlinearity, and possibly adds a weighted version of the previous value. This sequence of operations can be encapsulated in continuous time for each PE by a first-order differential equation as follows:

(11.39)

where I(t) represents the forcing function, and yi(t) represents the state of the dynamic system, and N is the number of PEs in the system. This is called Grossberg's additive model, and it is one of the most widely used neural models. In the additive model the weights are not a function of the state nor of time (after adaptation). Eq. 11.39 implements a computationally interesting model if the three dynamical variables, I(t), y(t), and w, span three different time scales. The parameters w contain the long-term system memory, and y(t) the short-term system memory. When I(t) is presented to the neural system, y(t) reflects the degree of matching with the long-term memory of the system contained in the weights. Figure 11-15 shows the block diagram of Grossberg's model.

Figure 11-15 Grossberg's additive model

Notice that all the connections are vectorized (multidimensional pieces), and they represent the computation, not the actual topology. A neural model is an abstraction of the neural network because it characterizes the dynamics and does not consider the interconnection matrix (i.e., the topology that we normally call the neural network). In the additive model the interaction among the PEs is additive, and the PEs have first-order dynamics. It is interesting to note that the additive model without the nonlinearity f(.) defaults to the ARMA model, the most general linear system model.

In discrete time the equation becomes a difference equation of the form

(11.40)

Grossberg's additive model gives rise to all the neural networks studied in this book. A dynamic neural network is obtained when the external patterns are attached to I(n), and the initial states of the dynamic system are held constant (normally zero) in Eq.11.40. Fully recurrent neural networks are the most general implementation of dynamic neural networks, but some other special cases are also possible. For instance, when the first term of Eq. 11.40 is zero, it means that there are no self-connections. When the first term is zero and the sum index j is kept smaller than i (i>j) the topology is feedforward but dynamic. The gamma model is also a special case of this topology where some of the PEs are linear and pre-wired for time processing.

The static mappers (MLPs) are obtained when the connections are restricted to be feedforward, the external inputs are applied to the initial states, and the forcing functions I(n) are zero in Eq. 11.40. When the initial states of the dynamic system are clamped by the inputs, there is no time evolution of states that characterize the relaxation, and y(n + 1) can be computed in zero steps.

Recurrent neural networks are regarded as more powerful and versatile than feedforward systems. They are able to create dynamic states. They have a wealth of dynamic regimes (fixed points, limit cycles, chaotic attractors) that can be changed by controlling the system parameters. They are thus very versatile and useful for modeling signals that vary in time, such as in time series analysis, control applications and neurobiological modeling. the gamma model

Go to next section


 11.12 Beyond First Order Dynamics: Freeman's Model

Neurocomputing has been centered on Grossberg's neural model for many years, but alternate models exist. Here we briefly describe a biologically realistic model of the cortex that was proposed by Walter Freeman following his studies of the rabbit olfactory system. The interesting thing about Freeman's model is that it is a computational model built from locally stable, coupled nonlinear oscillators, and it produces chaotic activity. Information is processed by the global chaotic dynamics, unlike any of the previous models, where information processing requires stable dynamic regimes (remember the fixed points of Hopfield's associative memory).

The simplest biological system we model is the cell assembly, an aggregate of thousands of neurons. Freeman models the cell assembly (K0 model) as a second-order nonlinear dynamic system, described by

(11.41)

where a and b are constants, x(t) represents the system state and Q(x(t)) is a forcing function. This forcing function brings the contributions of other PEs through a nonlinearity Q(x) where

(11.42)

The nonlinearity belongs to the sigmoid class but it is asymmetric. As we can see, this neuron model is divided into two parts: one is a linear, time-dependent operator defined as a second-order differential equation followed by a static nonlinearity. This division is in tune with the models used in neurocomputing. The nonlinearity models the synaptic transmission, while the dynamic equation models the transmission through axons and integration though dendrites.

To create a discrete network (a neural network) to approximate the continuous time behavior of Eq. 11.41, the impulse response of the linear subsystem is digitized and approximated by a three tap-gamma filter. This provides the following implementation for the K0 model shown in Figure 11-16.

Figure 11-16 Gamma implementation of the K0 model

The next level in the hierarchy is the modeling of interactions between cell assemblies. Freeman proposes that the individual cell assemblies modeled by the K0 are effectively connected by excitatory and inhibitory interactions with constant coupling coefficients, which represent mitral and granular cell interactions. He proposes the development of tetrads of K0 models, interconnected as shown in the Figure 11-17,

Figure 11-17 KII set, called here the Freeman's PE

where the negative and the positive signs mean inhibitory and excitatory connections, respectively. The tetrad of K0 models is called a KI model. Each group is described by a set of 8 differential equations with fixed connections. Since all the elements so far are fixed, they represent the building block for the model of the olfactory system and will be called Freeman's PE. Freeman's PE models the processing in the cortical column. Each column is therefore a neural oscillator. Table I shows the coefficients of Freeman's PE that have been derived from neurophysiologic measurements. The same table also shows the patterns stored in the model for the simulations.

Table 11.1 Freeman's PE Parameter Set

KII | [WMM_H; WMM_L; WGG]= [0.3; 1.5; 0.25] [WMM; WMG; WGM; WGG;]= [0.3; 5; 0.2, 0.25]
Patterns Stored |
KIII (Internal) | [WPP; WMM_H; WMM_L; WGG]= [0.2; 4; 0.5, 1.5] [WMP; WEM; WAM_L; WCB; WBC]= [0.5; 1.0; 1.0; 1.5; 1.0] [WMM; WMG; WGM; WGG;]= [0.25; 1.5; 1.5, 1.8] [WEE; WEI; WIE; WII]= [1.5; 1.5; 1.5, 1.8] [WAA; WAB; WBA; WBB]= [0.25; 1.4; 1.4, 1.8] [W1; W2; W3; W4]= [1.6; 0.5; 2, 1.5] [Te1; Te2; Te3; Te4]= [11; 15;12, 24] [Ts1; Ts2; Ts3; Ts4]= [20; 26; 25, 39]
Patterns Stored |

The KII model is a connected arrangement of Freeman's PEs (each K0 is connected to all other K0s in the same topological position) as shown in Figure 11-18.

Figure 11-18 The connections of several Freeman's PEs.

The KII system models the olfactory cortex as a dynamic system with two basins of attraction. When no forcing input is applied, the KII model has a fixed point at the origin, but when an input is applied, the system changes state to an oscillatory regime with large amplitude. The following simulation exemplifies the dynamics.

NEUROSOLUTIONS EXAMPLE 11.17

Simulating Freeman's KII Model

This example shows the NeuroSolutions implementation of the KII model. The KII model is an arrangement of KII sets which we called the Freeman's PE. Freeman's PE is an arrangement of 4 K0 sets, which are approximated by a gamma network of Fig 11.16. The connections to form the KII set (internal unit feedback) and KII model (global feedback among channels) have been separated for readability. Notice that a DLL implements the asymmetric sigmoidal nonlinearity.

We would like to show the response of the KII model to a step input. The model is a 20-PE system that receives 1 or 0 from the input. When we plot the response of one of the PEs that receive an input, we see that the amplitude of the PE oscillation increases. When the input is set to zero, the system state slowly decreases to zero. We depict time plots of the input and outputs of an active and inactive PEs. We also show the phase plot of one of the active Freeman's PEs (output of the excitatory input versus the inhibitory input). This example shows that the local PEs are stable oscillators with an amplitude modulated by the input. Hence the codification of information is contained as a spatio temporal pattern of amplitudes over the model.

NeuroSolutions Example

According to Freeman, the central olfactory system consists of two layers of coupled oscillators, the olfactory bulb (OB) and the prepiriform cortex (PC), mediated by lumped control of the anterior olfactory nucleus (AON). The receptor input connects to periglomerular (PG) and mitral cells (M). The mitral cells transmit to granular cells (G) and to the AON and PC. From the PC the final output is sent to other parts of the brain by deep pyramidal cells (P) as well as back to the OB and AON.

The central olfactory system is implemented by the KIII model, which is an hierarchical arrangement of KII models (Figure 11-19). The input is a fixed-weight, fully connected layer of excitatory K0 models (the periglomerular cells, PG). The second layer is the KII model described earlier (which models the olfactory bulb, OB). This is where learning takes place by changing the excitatory-to-excitatory connections using modified Hebbian learning. The next layer is a single Freeman's PE (modeling the anterior olfactory nucleus, AON), followed by another Freeman PE (modeling the prepiriform cortex , PC), which finally connects to a K0 set (modeling the pyramidal cells, P). There are extensive (but not fully) connections in the topology: the output of the OB layer is fed to the AON and the PC layers with diffuse connections (modeling the medial olfactory tract). Feedback transmission is implemented from the PG and PC layers through long dispersive delays (modeling the medial olfactory tract, MOT) which are represented in Figure 11-19 as f(.). The parameters of the Freeman's PE in layers AON and PC are different from the OB layer (in fact, creating incommensurate oscillating frequencies): see Table 11-1.

.

Figure 11-19 KIII model

The KIII model is a chaotic dynamic system. With no input the system at the OB layer produces chaotic time series with small amplitude and no spatial structure (the basal state). When a previously stored pattern is presented to the system, the OB layer still produces chaotic time series but "resonates" in a stable spatial amplitude distribution. This spatial distribution is what codifies the stored pattern. Hence the system works as an associative memory. However, the interesting thing is that the dynamics are no longer toward a point attractor, as in the Hopfield network, but to a spatio temporal chaotic attractor. Freeman states that this system is capable of very fast switching among patterns and is robust to noise in the patterns.

NEUROSOLUTIONS EXAMPLE 11.18

Simulating Freeman's KIII Model

This example demonstrates the oscillation produced by an 8 PE OB layer. The breadboard is rather large. We have labeled each block according to Figure 11-17. Notice that the breadboard is a replication of basically two pieces: the KII model and the gamma filter implementing the dispersive delay operator. Each has different parameters according to biological measurements.

The input is a rounded pulse. When there is no input, the system lies in a chaotic basal state in which the amplitude of each PE time series has the same basic amplitude but is very complex, as shown in the Megascope. When the pulse is applied, the system jumps to a new state, and the phase space plot (excitatory versus inhibitory PEs) shows a phase transition to one of the wings of the chaotic attractor. When the input disappears, the system goes back to its initial chaotic basal state. The system transitions are repeatable and fast. Notice that we do not need any extra mechanism to take the system from the pattern created by the excitation (as we did for the fixed point of the Hopfield network). The information is still coded in the amplitude of the chaotic oscillation across the PEs. Notice that all the parameters of the simulation were fixed (i.e., no learning is taking place).

NeuroSolutions Example

The exciting thing about Freeman's model is that it extends the known paradigms of information processing that have low-order dynamics (Hopfield networks have dynamics of order zero, i.e., fixed points) to higher-order spatio temporal dynamics. Moreover, the model was derived with neurophysiological realism. It is too early to predict its impact on artificial neural networks research and information-processing paradigms, but embodies an intriguing dynamic system approach for information processing.

Go to next section


 11.13 Conclusions

This chapter presented the development of learning rules for time processing with distributed TLFNs and fully recurrent networks. We showed the difficulty of training networks with delays, and we presented an extension to static backpropagation that was able to train topologies with delays. Backpropagation through time (BPTT) shares some of the nice properties of static backpropagation (such as locality in space and efficiency, since it uses the topology to compute the errors) but it is nonlocal in time. This creates difficulties that require the use of memory and backpropagating the error from the final time to the initial time (hence the name). With BPTT we harness the power to train arbitrary neural topologies to learn trajectories, which is necessary to solve problems in controls, system identification, prediction, and temporal pattern recognition. In a sense, this chapter closes the circle started with static nonlinear neural networks and time processing. Now we have the tools to train nonlinear dynamic systems.

In this chapter we also presented a view of dynamic neural networks as nonlinear dynamic systems. We tried to make the difference between static and dynamic neural networks more apparent. We could not discuss recurrent neural networks without presenting Hopfield networks, because of the perspective on computational energy. This analogy provides a "physical" view of computation in distributed systems, that gives us insight into the nature of computation and the tools to quantify and design computational distributed systems.

We also introduced the idea of neural models. Neural models sit at the top of the systematized hierarchy of neural computation. Although we covered many neural networks throughout this book (linear regressors, MLPs, Hebbian, RBFs, Kohonen, linear combiners, TLFNs, recurrent networks), they all belong to the same neural model. This clearly leaves the search for other neural models open.

We finished the chapter by providing a glimpse of other computational paradigms that may become important in the future due to their biological realism and information principles based on higher-order (chaotic) dynamic regimes. Freeman's model points the ultimate paradox: The quest for systematization (understanding) and organization that characterizes humans and their society emanates from brains that may be based on chaotic dynamics.

Go to next section


 11.14 Exercises

11.1 State the differences between a Jordan and an Elman network and present two cases where each would be preferable.

11.2 If you disturb one weight of a TDNN with a memory size of N=10, at t=0 for how many samples in the future will you be able to measure the perturbation? What about the recurrent system of Fig 11.2 when [image] is changed at t = 0?

11.3 Unfold the following network in time and present the ordered list in time. Compute the BPTT equation for a and b.

11.4 Present in a table the equation and its dual for the following components: sigmoid axon, synapse, delay, softmax.

11.5 Show how you would implement BPTT for the following network (T= 10 steps) using the duals and the data flow method. (Each arrow shows a weight.)

11.6 You have a fully connected network with 100 PEs to be trained by trajectory learning for T = 1,000 steps. Which method, BPTT or RTRL provides the best computational solution (number of computations) ? Which provides the best solution in terms of storage?

11.7 Create one state-space model for Problem 3. We recommend you define the state as the output of the adder.

11.8 Confirm the limit of Eq. 11.36 using the breadboard of Example 11.14.

11.9 Modify Eq. 11.40 (and Fig 11.15) to obtain

  • A one hidden layer MLP

  • A focused TLFN (2 taps)

  • A Hopfield network

  • An ARMA model

  11. 10 Study the effect of the relaxation on the training of the XOR problem of Example 11.8.

11.11 Fully train the system of Example 11.9 for the figure 8. Run the example several times and see how the system output follows the input signal. Now go to the test mode. In this mode the output of the net is fed back to the input, so there is no input.

Change the bottom signal generators at both input and desired signal to a triangular wave. Include a phase shift of 90-degreee at the input and 101.25 degrees at the output. This creates a harder problem to train.

11.12 Study the Examples 11.10 and 11.11. In particular, let the system train longer and change the inputs to observe differences in training.

Go to next section


 11.15 NeuroSolutions Examples

11.1 Jordan's Network

11.2 Elman's Network

11.3 Training the Feedback Parameter of Elman's Network

11.4 The Effect of the Slope of the Nonlinearity in Recurrent Networks

11.5 The Issue of Memory Depth in the Bus Driver Problem

11.6 Output MSE and the Gamma Memory Depth

11.7 Frequency Doubling with a Focused TLFN

11.8 Learning the XOR with a Recurrent Neural Network

11.9 Trajectory Learning: The Figure 8

11.10 Training an Embedded Neural Network in a Dynamic System

11.11 Training a Neural Controller

11.12 Hopfield Network Dynamics

11.13 Pattern Completion in Hopfield Networks

11.14 Training a Recurrent Network as an Associative Memory

11.15 Hopfield Networks for Optimization

11.16 Brain-State-in-a-Box Model

11.17 Simulating Freeman's KII Model

11.18 Simulating Freeman's KIII Model

Go to next section


 11.16 Concept Map for Chapter 11

Go to Appendix A


 A.1 Introduction

We are much more familiar with the typical operations we perform on real numbers than with the set theory concepts of the real numbers. However, the set properties are the ones that describe the structure of the space, providing the rules of the operations and ensuring the correctness of the results.

Here we extend our reasoning and computational skills to ordered lists of real numbers called vectors. The branch of mathematics that studies vector operations is called linear algebra. Once we understand the fundamental principles of vector operations, we can simply operate with vectors and disregard the principles underlying these operations, exactly as we do with the operations with real numbers.

Neural networks receive many inputs (usually real numbers) and produce multiple outputs at a time, so they operate with vectors. One goal is to quantify the operation of the learning machine in some mathematical sense. While the mechanics of vector operations may not require knowledge of the fundamentals of linear algebra, this knowledge becomes very important when we try to analyze and understand the function of neural networks, so a brief introduction to linear algebra is worthwhile.

In neural networks we need efficient methods to deal with quantities that are multivalued. For instance, to predict the stock market with a neural network we may use as inputs the present value of the S&P 500, its value a week ago, and two weeks ago, the value of the industrial productions, the short term interest rate, and so on.

Figure A-1 A neural network with multiple inputs

Each one of these variables can be treated separately as a number, but they can also be thought of as elements of a list that is sent in parallel to the neural net. The idea of treating several variables as a list can be mathematically formulated as a vector. A vector x is an ordered collection of numbers

This is called a column vector of N elements. We have to specify whether the elements are integers, real numbers or complex numbers. Normally we work with vectors of real numbers, that is, a vector will belong to R [image], the space of all the N-tuples of real numbers.

How do we operate with vectors? What are the properties of the operations? What is their meaning? The answer to these questions is the goal of this appendix.

Go to next section


 A.2 Vectors: Concepts and Definitions

We present here the most widely used operations with vectors and their meaning.

A.2.1 Definition of Linear Vector Space

The idea of a vector space is fundamental to operations with vectors, because it provides the structure of the space and the operations that are defined for vectors.

Definition: A vector space X is a set of vectors defined over a scalar field (here the real numbers) R, that satisfy the following properties:

  1. Vector addition: If arbitrary x1 and x2 belong to X, then x1+x2 also belongs to X.

  2. Vector addition is commutative, x1+x2 = x2+x1

  3. Vector addition is associative, (x1+x2) +x3= x1 + (x2+x3)

  4. There is a unique neutral element for addition, the zero vector, x1 + 0= x1 for all x1.

  5. For every x1 there is a unique vector -x1 such that, x1 + (-x1) = 0

  6. For all scalars a belonging to R and vectors x belonging to X, we can define the multiplication a x that also belongs to X.

  7. There is a unique neutral element for the multiplication, 1x1 = x1 for all x1.

  8. Multiplication by a scalar is commutative, a(bx1) = (ab)x1.

  9. Multiplication by a scalar is associative, (a+b)x1 = ax1 + bx1

  10. Multiplication by a scalar is distributive with respect to the addition of vectors, a(x1+x2) = ax1 + ax2

As we can see, these operations define a set of operation that can be done with vectors and show that the result is still a vector.

A.2.2 Notation and Construction

A vector is denoted by bold letters. The elements of vectors are subscripted to show the ordering of the element in the list: for example, xi is the ith entry in the list. Here a vector is a column of numbers. The size of a vector is equal to the number of its elements.

We can put vectors of size N into a one-to-one correspondence with points in R [image].

Figure A-2 Vectors and points in R

A vector x can be represented by a directed line segment that has the origin of the space as the initial point and the point of coordinate {x1,x2,…,xN} as its end point. Thus we assign the ith element of the vector to the projection of X onto to ith axis.

A.2.3 Transpose of a Column Vector

The transpose of a column vector is denoted by x [image], and it is a row vector with the same elements,

The transpose of a row vector produces a column vector.

A.2.4 Vector Addition and Subtraction

We can add or subtract only vectors of the same size, that is, with the same number of elements. To add two vectors of the same size, we add or subtract the elements of the same index zi = xi = yi,

(A. 1)

There is an intuitive, geometric picture for the vector addition operation. Vector addition corresponds to the geometric addition of the corresponding directed line segments, so it is easy to visualize.

Figure A-3 Addition and subtraction of vectors

The vector subtraction operation is similar, but now instead of adding the components we subtract them. Vector subtraction corresponds also to the geometric subtraction of the corresponding directed line segments (reverse the direction of the vector that is being subtracted; see Figure A-3).

A.2.5 Length (or Norm) of a Vector

The norm of a vector is a function that produces a scalar. The best-known norm is the L2 norm. The L2 norm of a vector is a scalar that is equal to its length. The norm is denoted by || || and is computed as

(A.2)

The length of a vector in an N-dimensional space is simply the extension of the Pythagorean theorem to N dimensions, so the L2 norm corresponds to the Euclidean distance.

A.2.6 Multiplication by a Scalar

A real constant a times a vector is the multiplication of a by every element xi:

(A.3).

This operation scales the vector length by the same constant a.

A.2.7 Inner or Dot Product

The inner product is the vector operation equivalent to the multiplication of real numbers. The inner product of two vectors

is the scalar

(A.4)

The inner product can be visualized as the projection of one vector onto the other (Figure A-4). The inner product is a very common operation with vectors. Note that the inner product of a vector with itself gives the square of its length. Another notation for the inner product [image] x,y [image] is x·y, so the inner product is also called the dot product.

Figure A-4 Inner product of two vectors

A.2.8 Cauchy-Schwartz Inequality

The absolute value of [image] x,y [image] is

(A.5)

A.2.9 Angle Between Vectors

The angle [image] between two vectors is defined by

(A. 6)

A.2.10 Orthogonal Vectors

Two vectors are orthogonal when the angle between them is 90 degrees, which implies that their inner product is zero. The concept of orthogonality can also be extended to spaces. A vector is orthogonal to a space when it is orthogonal to all vectors in the space; the vector must lie along the normal to the space.

Figure A-5 Vector normal to a space

The vectors that lie along the axis of R [image] are orthogonal. It is easy to write the coordinates of orthogonal vectors by thinking about the axis of the space

However, there are other, less obvious combinations that make vectors orthogonal (use the inner-product definition). When the norm of the set of orthogonal vectors is 1 the set is called orthonormal. Dividing each orthogonal vector by its length creates the orthonormal set. We can then conclude that in an N-dimensional space, we can have at most N orthogonal vectors.

A.2.11 Linear Independent Vectors

A set of vectors {xi} is called linearly independent if the equation

is true if and only if all the constants ai are zero. In a linearly dependent set of vectors at least one can be represented by a linear combination of the others, so some are superfluous because they can be composed from other elements in the set. A linearly independent set of vectors is minimal; we need them all since we can not represent any as a sum of the others.

A.2.12 Span of a Space and the Concept of a Basis

The span of a collection of vectors {xi} is the space of all their linear combinations. What this means is that if we have three linearly independent vectors, we can span the 3-D space, because we can create any vector as a linear combination of these three linearly independent ones. For this reason a set of linearly independent vectors is also called a basis. The dimension of a vector space is the size of the maximum number of linearly independent vectors.

Note that a basis does not mean that the vectors are orthogonal, but simply linearly independent. However, orthogonal basis have convenient properties, and we should try to work with them. One such property is that in an orthogonal basis the projection in each direction can be computed independently of the others.

A.2.13 Gram-Schimdt Orthogonalization Procedure

Gram-Schmidt orthgonalization is a procedure to obtain orthogonal vectors {oi} from any linearly independent set {xi}. Start with the first vector of the x basis and make o1 = x1. Then take the second vector x2, and subtract to it the part that lies along the direction x1, tha is, o2 = x2 - ao1, where a is the inner product of the two vectors o1 and x2,

(A.7)

Continue the process to obtain all the other vectors,

(A.8)

A.2.14 Vector Expansion

In a vector space we can always represent any vector as a sequence of numbers that are the projections of the arbitrary vector on each basis. Take an arbitrary vector x in a vector space with basis u = [u1,u2,…,uN] [image].

The vector can be written as

(A.9)

so we can always represent x by the column of numbers

(A.10)

This was our first definition of a vector, but now we relate it to a basis, as we should. We can appreciate the tremendous advantage of an orthogonal basis if we compute the components of vectors in a nonorthogonal basis. The component of the vector x over the jth basis is simply the inner product of the vector with that basis. From Eq. A.4 this gives

(A.11)

which will be a function of every xi. If the basis is orthogonal, we have a much simpler expression, since the inner product is zero for i different than j, yielding simply

(A.12)

A.2.15 Signals as Vectors

A discrete signal with N samples can be put in a one-to-one correspondence with vectors, by using the vector expansion explained earlier. First, we count the number of samples and choose a space of the same dimensionality, N. Then we map the signal onto a point in this space by the following procedure: Assign the first signal sample to the value of the first coordinate in the N-dimensional space, the second sample as the value of the second coordinate, and so on. A discrete signal of length N thus becomes a point in a N-dimensional space. We have seen that vectors can also be put into a one-to-one correspondence with points in R [image] , so we can represent the signal as a vector with elements given by the time samples.

Figure A.6 Signals as vectors

Remember however, that there is an implicit basis chosen when we go from a signal to a vector. If a different orthonormal basis to represent the vector is selected, the vector representation of the signal will probably be modified. But not everything changes. Since the length of a vector is independent of the orthonormal basis that is used to represent it, the length in the two representations will be thesame.

Go to next section


 A.3 Matrices: Concepts and Definitions

Below we present the most common definitions and operations with matrices.

A.3.1 Definition of a Matrix

Matrices are ordered arrangements of elements wij like vectors, but now there are two indexes, a row index j and a column index I, as shown in

(A.13)

We can think of a matrix as a column vector of elements that are row vectors or vice versa. Alternatively, we can think of a row vector x as a matrix with one row and a number of columns equal to the dimension of the vector. A matrix is square when the number of columns is equal to the number of rows (M = N).

A.3.2 Addition and Subtraction of Matrices

Just as in vectors, the addition (or subtraction) of two matrices of the same size produces a third matrix of the same size with elements that are the addition (or subtraction) of the elements with the same index.

A.3.3 Multiplication of a Matrix by a Constant

The multiplication of a matrix by a constant equals the multiplication of each element of the matrix by the constant.

A.3.4 Toeplitz Matrix

A matrix is a Toeplitz matrix if it is a square matrix with identical elements in the main diagonal and the rest of the elements are symmetric with respect to the main diagonal. Toeplitz matrices are highly symmetric so many simplified results are known for this class of matrices.

A.3.5 Vector-Matrix Multiplication

Vector-matrix multiplication is a very important operation. We define the multiplication of the row vector x by the matrix W as a new row vector y with a number of columns equal to the number of columns of W and elements given by

(A.14)

We cannot always multiply a vector by a matrix. To execute the operation, the number of columns of x has to be equal to the number of rows of W. Vector matrix multiplication is not commutative,

(A. 15)

As we will see, multiplying a vector by a matrix rotates the vector. If x is a column vector we can only post multiply W by x, that is. y = Wx is a column vector in which each element is the product of the corresponding row of W with x.

A.3.6 Range and Null Space of a Matrix

The range of an N x M matrix W is a subspace defined by all possible nonzero vectors y of size M (the column vectors of W) such that y = Wx, where x is an arbitrary N-dimensional vector. The null space is also a subspace where Ax = 0 for arbitrary N-dimensional x.

A.3.7 Definition of the Matrix Transpose

The transpose of a matrix is another matrix in which the rows are exchanged with the columns. We denote transpose by a superscript T. The transpose of a column vector of size N becomes a row vector of the same size N. When we transpose a matrix, the diagonal elements remain unchanged. A square matrix that is equal to its transpose is called symmetric.

When we multiply a row matrix by a column matrix, we get a single element. Hence the multiplication of a row vector by a column vector is simply the inner product of the two vectors

(A.16)

The transpose of a product is the product of the transposed matrices with the order reversed:

(A.17)

A.3.8 Definition of Outer Product

The outer product is the "mirror image operation" of the inner product. It is the multiplication of a column vector by a row vector, which produces a matrix of row size equal to the column vector size and column size equal to the size of the row vector.

(A.18)

A.3.9 Determinant of a Matrix

The determinant of a square matrix is a scalar given by

(A.19)

where W1j is a (N-1) by (N-1) matrix obtained by deleting the first row and the jth column of W. When the determinant is non-zero, the matrix is called nonsingular or full rank. When the determinant is zero the matrix is called singular. A diagonal matrix has a determinant that is the product of the diagonal elements.

A.3.10 Trace of a Square Matrix

The trace of a sqaure matrix is the sum of all the diagonal elements of W:

(A.20)

A.3.11 Inverse of a Matrix

The inverse of a square matrix W is another matrix [image] that obeys the relation

(A.21)

The matrix I is called an identity matrix. The inverse of a product of matrices is

(A.22)

Only nonsingular matrix have inverses. If the matrix is diagonal, its inverse is simply the inverse of the diagonal elements.

A.3.12 Computation of the Inverse

The inverse of a square matrix W is

(A.23)

where adjW is the adjoint of W and |W| is the determinant of W. The adjoint of W is

(A.24)

that is, the transpose of the matrix of cofactors of W.

To compute the cofactors, we have to first create the minors of W. The minor Mij of W is the determinant of W after the ith row and the jth column have been removed. The cofactor cij is the signed minor of W, i.e.

(A.25)

The matrix of cofactors C is the matrix formed by cij. As an example let us compute the inverse of

A.3.13 Rank of a Matrix

The rank of W is the dimension of the range of W, that is, the size of the number of linearly independent columns of W. A full-rank matrix has a nonzero determinant and accepts an inverse.

A.3.14 Linear Transformation

A linear transformation is simply the multiplication of a vector by a matrix. A linear transformation of x produces a vector y = Wx that is a rotation and a scaling of x.

Figure A-7 Multiplication of a matrix by a vector

If W is full rank then y has the same dimension as x, and it belongs to the range of W. This corresponds to a change of direction and length.

However, when W is singular (i.e., not full rank) the transformation is called a subspace projection, because the vector y exists in the null space of W.

A.3.15 Linear Transformations Revisited

The linear transformation described in the previous section can be described in more general terms. It effectively corresponds to a transformation or mapping between two finite dimensional vector spaces. Let X be a vector space called the domain; let Y be another vector space called the range. The transformation is a mapping relating each x [image] X to y [image] [image] Y. A mapping A is linear if these two conditions are met:

  1. For all x1 and x2 [image] X, A(x1+x2) = A(x1) + A(x2)

  2. For all x [image] X, and w [image] R, A(wx)=wA(x).

In this context it is important to relate an orthogonal basis of x to an orthogonal basis of y. We can show that this relation is given by

(A.26)

There is thus always a matrix representation for the linear transformation between x and y. However, the matrix representation is not unique. We can, for instance, change the orthogonal basis of y (there are many possible orthogonal bases of a vector space), which implies that A must be different.

A.3.16 Change of Basis

Suppose we have a basis of X called {x1}, a basis of Y called {y1}, and a linear transformation A between X and Y, Y = AX. Now let us change the basis of X to {x2} and the basis of Y to {y2}. For these new bases, the linear transformation will be different, that is, Y = BX. How can we relate A and B? The obvious answer is to relate the two bases sets.

Let U be a matrix that represents the new basis {x2} in the original basis {x1}, that is,

(A.27)

Likewise let S be another matrix that represents {y2} into the original span {y1}. Then

(A.28)

This is called a similarity transformation and represents the relationship between two matrices under a linear transformation. If we are careful in choosing the basis, the linear transformation becomes obvious.

A.3.17 Eigenvectors and Eigenvalues

This formulation applies only to square matrices. We saw that when we multiply a vector by a matrix, the vector generally changes direction. However, there are a special vectors e for a given matrix W that do not change directions when multiplied by W. These are called eigenvectors of W, and they obey the relationship

(A.29)

The scalar [image] are called the eigenvalues of W. The advantage of operating with the eigenvectors is that matrix multiplication becomes a set of scalar operations, since we only need to multiply each vector component by the corresponding eigenvalue. If W is NxN, the product of any vector by W requires N² multiplications. However, if y is in the eigenspace of W, we need only N multiplications to obtain the result.

When W is N x N, there are at most N distinct eigenvalues. When W is symmetric, then the eigenvectors are orthogonal, so they become an orthogonal basis of the space. This has important implications because we can rewrite W as

(A.30)

where [image] is a diagonal matrix of the eigenvalues of W, and [image] is the matrix with columns given by the eigenvectors of W. We can always write any symmetric matrix W in this way. Notice that we effectively found a way to diagonalize the symmetric matrix W.

In terms of a linear transformation of a vector x, if the linear transformation is the eigenvector matrix [image], [image] then we found a way to represent x in a new coordinate system formed by the eigenvectors of W. In this new coordinate system each projection can be computed independently of the others, that is,

(A.31)

In the new coordinate system we can write the matrix W as a diagonal matrix, so the transformation by [image] diagonalizes W and it is called an orthogonal transformation. Under orthogonal transformations, Euclidean distances are preserved.

These results can be extended to nonsquare matrices and are called singular value decomposition.

A.3.18 Whitening Transformation

After applying an orthogonal transformation as done in A.3.18, we can still make the diagonal matrix L an identity matrix, that is, a diagonal matrix with all entries equal to 1, by including a new transformation to [image] equal to

(A.32)

The transformation [image] is called a whitening transformation.

All the eigenvalues after a whitening transformation become identical. This is important for gradient descent learning since the eigenvalue spread is one.

A.3.19 Derivative of a Matrix

We define the derivative a matrix W with respect to a parameter [image], as

(A.33)

Go to next section


 A.4 Random Vectors

Up to now we have assumed that the input vector x = [x1,…,xN]T was a deterministic quantity, that is, a set of fixed real numbers obtained from the external world through measurements. In a lot of important, practical applications related to classification and signal processing, it is important to reinterpret these numbers as random quantities.

Definition: A random variable is a set of real numbers whose values are controlled by a probability law.

Suppose that we create a vector of 3 components for an experiment. The values of the components are 1 and 0. However, we are going to decide if the element xi is a 1 or a 0 by flipping a coin. If the coin lands heads up, it is a 1, otherwise it is a 0. We immediately see that there is an underlying structure to the values of these vectors, dictated by the probability of having heads or tails. A vector created in this way is a random vector.

In practice, there are a lot of unobservable variables affecting a given experiment. Although we are not sure whether the phenomenon is random or deterministic, sometimes it is more efficient to model it as random. A good example that is important for our study is the classification of patterns. Let us take handwritten digits. Each of us writes in a different way. If we want to develop a machine that automatically recognizes digits, a productive way is to model a given observation of a digit as a random vector. Suppose that we have 10 classes characterized statistically in pattern space (one for each digit). Then we ask the question: Which class is more probable, given our observation? The class that is more probable is the one to which we assign the observation. Thus we have to perform a statistical characterization of each class and make a statistical decision. The principles involved are briefly reviewed in the following sections.

Definition: Probability of an event A is the relative frequency nA with which the event appears in a set of N measurements, i.e.

Probability of an event is always between [0, 1]. The sum of the probabilities of all the outcomes of an experiment has to add up to 1, that is, if the possible outcomes are A, B, C, then

A.4.1 Distribution Function

A random vector is characterized by a probability distribution function, defined as

(A.34)

where Pr{x} means the probability of an event x, and p1,…pN are simply numbers between 0 and 1. A short hand notation for Eq. A.34 is

(A.35)

where x is a random vector. Basically, the distribution function measures the probability of an event is smaller than a given value p. The distribution function is a monotonically increasing function (always larger than or equal to zero).

A.4.2 Density Function

The probability density function (pdf) is the derivative of the probability distribution function:

(A.36)

Alternatively we also can define the probability density function as

(A.37)

where y is a vector with N components, so in fact this is an N-dimensional integral. The probability density function (pdf) is always positive and sums up to 1 for all x.

In practice we can visualize the shape of the pdf if we plot the histogram. The histogram is the relative frequency of events over the domain x. We start by dividing the range of x in equal intervals [image] x, and then we ask how many events occur in each interval. The pdf is the limit of the histogram for infinitely small [image] x and infinitely many events. If the variable is discrete, the PDF is called the probability mass function.

A.4.3 The Mixture Density

We often work with random vectors obtained from different classes, each one having a specific pdf. These class pdfs are called class conditional pdfs and are denoted [image] or simply [image].

We call these conditional pdfs likelihoods, since in practice we seek to model them as parametric forms and in such cases they work as likelihood functions for the observed values. The mixture density of x is defined as

(A.38)

where Pi is the a priori probability of class i. We assume that there are L classes.

A.4.4 A Posteriori Probability and Bayes Theorem

The probability of x belonging to one of the classes Ci is called the a posteriori probability of x, [image].

Given the a priori probability, we can obtain the a posteriori probability by using the famous Bayes theorem

(A.39)

where Pi are the prior probabilities, p(x|Ci) are the class conditionals also called the likelihoods, and p(x) is the probability of x.

A.4.5 The Central Problem of Pattern Recognition

When we have observations drawn from a mixture density, we want to find a rule that will associate the present observation with one of the classes. This is the central problem we face in real life. If you think a bit you see that the probability of x belonging to one of the classes (the probability of class I, given x) is exactly the a posteriori probability of the event x. However, we normally cannot compute the a posteriori probability directly. This is where Bayes' theorem becomes important, because if we know the a priori probability and the likelihood for each class we can compute the a posteriori probability by Eq. A.39. For most cases the denominator is simply a normalizing constant that can be omitted. Pattern recognition deals with the estimation of conditional probabilities, that is, with the estimation of class pdfs, the only unknown in Eq. A.39.

A.4.6 Decision Surfaces

The a posteriori probability P(Ci|x) is the probability of the class being i given the present observation vector x. Bayesian decision making evaluates the a posteriori probability for each class and assigns the observation vector to the class j that has the highest a posteriori probability:

(A.40)

This rule minimizes the number of errors. The pattern space is effectively divided into regions that are associated with each class. When a pattern vector falls into one of these decision regions, it is assigned (classified) to that class. This assignment rule may produce errors, but it is a strategy to minimize errors given the information existing in the training set and the assumption of Gaussian statistics.

We can say that the goal of pattern-recognition machines is to find these decision regions in pattern space to provide good classification accuracy.

A.4.7 Discriminant Functions

According to Bayes' decision making, we only need to compare the a posteriori probabilities to make decisions about class partnership. This greatly simplifies things and we can define functions yi(x), one for each class, that compute a partnership value over the full space. These functions are called discriminant functions. The classification is done by comparing discriminant functions, that is, we choose class j when

(A.41)

In Bayes' formalism the discriminant functions are the a posteriori probabilities. But other ways to construct discriminant functions exist that do not require the estimation of probability density functions.

A.4.8 Moments of Distributions

Most of the time it is impractical to find the pdfs for each class analytically. We resort to a description using moments. The most widely used moments are the first and second order moments, called respectively the mean and the variance. For Gaussian distributions they define completely the distribution.

A.4.9 Mean of a Random Vector

The mean of a random vector is defined as

(A.42)

where the integration (or summation for discrete spaces) is taken over the full space. The mean of a distribution gives the location of the center of mass of the distribution. E is the expected value operator that is widely used in statistics.

The ith component of M is

(A.43)

where p(xi) is the one-dimensional marginal density obtained by integrating p(x) over all other components. The conditional expected value is defined as

(A.44)

where pi(x) is the class likelihood.

A.4.10 Variance of a Random Variable

The variance of a random variable xi is defined as

(A.45)

and it measures the dispersion around the mean. Its square root is called the standard deviation. The variance measures the power in a zero-mean random variable.

A.4.11 Covariance Matrix

This is an important characterization of a distribution since it characterizes the dispersion around the mean. The covariance matrix of x is

(A.46)

The components of the covariance matrix diagonal are the individual random variable variances, and the off-diagonal terms are the covariances of pairs of random variables.

There are two important observations regarding covariance matrices. First, the elements of covariance matrices are all positive or zero, so the covariance matrix is semipositive definite. Second, the covariance matrix is square and symmetric, so it is a Toeplitz matrix. The covariance matrix can be written as

(A.47)

where R = E[xx [image] ] is the autocorrelation matrix of the random variable. The elements of the autocorrelation matrix are

(A.48)

that is, the expected values of the crossproducts of random variables with different indices. This matrix is a Toeplitz matrix.

A.4.12 Crosscorrelation Matrix

The cross-correlation matrix is very much like the autocorrelation matrix, except two different random variables x and y are involved:

(A.49)

This matrix is no longer a Toeplitz matrix.

A.4.13 Orthogonalization of the Covariance Matrix

Notice that the covariance matrix is a symmetric, semipositive definite matrix. Hence it can be orthogonalized, as discussed above as

(A.50)

where [image] is the matrix built from the eigenvectors of [image], and [image] is the eigenvalue matrix of [image] which is diagonal. We can then apply a linear transformation to the space of x and work on a new space y = [image] [image] x with an axis given by the eigenvectors of [image]. In this new space we can work with each eigenvector independently: the random vectors became uncorrelated. The eigenvalues become simply the variances of the transformed variables yi.

A.4.14 Normal Distribution

Gaussian distributions are pervasive in the analytical study of random phenomena for several reasons. First, because of the law of large numbers, they are a reasonable assumption for large collections of experimental data (the law of large numbers says that the distribution of a sum of arbitrary distributions approaches a Gaussian when the number of terms tends to infinity). Second, the Gaussian function is analytically simple to work with, because it is totally defined by the first and second-order moments (i.e., all the moments of higher order are zero). Third, if two Gaussian-distributed random variables are uncorrelated, they are also independent. The Gaussian distribution of a random variable xi is defined as

(A.51)

For the random vecto, we have to define a multivariate normal distribution

(A.52)

Where m is the mean vector and [image] is the covariance matrix. The exponent can be interpreted as a quadratic distance function of pairs of observations xiji to their respective means, normalized by the covariance. It is called the Mahalanobis distance from x to m.

These definitions give us some more interesting properties of the multivariate normal distribution: The marginal densities are also Gaussian, and a (nonsingular) linear transformation of a Gaussian is still Gaussian. It is easy to find a projection that makes multivariate Gaussian variables independent if we apply the diagonalization procedure mentioned earlier. In this case the pdf is simply

(A.53)

A.4.15 Discriminant Functions for Gaussian Distributions

If the class clusters can be modeled as Gaussians, we can construct discriminant functions easily. First, we do not work directly with the discriminants defined earlier, but with their natural logarithms. Nothing changes, since the log is a monotonically increasing function, but the new discriminants

(A.54)

became easier to apply. In fact, if we assume that the likelihoods are independent and Gaussian, the decision boundary becomes

(A.55)

which are quadratic functions. If the class covariances are all the same [image] I = [image] then the decision surface becomes the intersection of the linear discriminants

(A.56)

where

(A.57)

The decision boundaries correspond to hyperplanes perpendicular to the lines joining the class centers, and crossing at a point wi0.

A.4.16 Sample Estimates

Unfortunately, we still need an extra step to work with the data we collect from the external world. Even if we can make the assumption of Gaussianity for our data sets, we still do not know the mean vector and covariance matrix. We have to estimate these quantities from a finite number of observations. In general, when we cannot access or compute the true quantity, we have to settle for an approximation, called an estimator of the true quantity. Obviously, we would like to work with estimators that do a good job in the approximation. The biggest issue is that we are interested in estimating random quantities, so the estimators are themselves random variables, and it is not easy to compute upper bounds as we do when handling deterministic variables.

A.4.17 Maximum Likelihood

We explain here the maximum likelihood method. The idea behind maximum likelihood is very simple. It seeks to estimate the statistical quantity of the data that is more likely given the observations. Here the statistical quantities of interest are the mean vector and the covariance matrix.

Let us assume that the density function p(x) that created our data set is a function of an unknown, but fixed, set of parameters [image] = { [image] 1,…, [image] M}. The data set of N vectors x = {x1,…,xN} is assumed to be drawn independently from the distribution p(x). The joint probability density function of the data set x is

(A.58)

where L( [image]) is a function of [image] only for fixed x, and it is called the likelihood of [image] for the given x. Maximum likelihood picks the [image] that maximizes the likelihood or, equivalently, that minimizes the negative likelihood of the log of L( [image])

(A.59).

The optimization can be done analytically only for a handful of cases, including the case when p(x) is Gaussian. When the parameter to be sought is the mean and p(x) is assumed Gaussian, the maximum likelihood procedure gives

(A.60)

that is, the sample mean is the maximum likelihood estimate of the mean for Gaussian pdfs. If the parameter is the covariance matrix, under the Gaussian assumption the maximum likelihood estimate becomes

(A.61)

which is the sample average of the outerproduct of (xi–m)(xi–m) [image].

A.4.18 Bias and Variance of Estimators

The bias of an estimator [image] is defined as the difference between the mean of the estimator and the true quantity [image] :

(A.62).

When the mean of the estimator is equal to the true mean, the estimator is called unbiased. The variance of the estimator is defined as the expected value of the square difference between the estimator and its mean

(A.63)

When the variance of the estimator decreases to zero and the estimator is unbiased, the estimator is called consistent.

Hence the maximum likelihood estimate of the mean is consistent, since it is unbiased and the variance decreases to zero with the number of samples.

The mean square error of an estimator is defined as the expected value of the square difference between an estimator and the true quantity,

(A.64)

The mean square error is the most widely used criterion to compare estimators.

A.4.19 Unbiased Estimator of the Covariance Matrix

The maximum likelihood estimator of the covariance matrix is biased. To obtain an unbiased estimate of the covariance, we have to replace N with N-1 in Eq. A.61. This covariance matrix estimator is consistent.

Go to next section


 A.5 Conclusions

In this appendix we attempted to provide an overview of the most important definitions and operations in linear algebra and applied probability to pattern recognition. If you need further information please consult the following sources:

For linear algebra:

Golub G. and Van Loan C., Matrix Computation, John Hopkins University Press, 1989.

For probabilities

Papoulis A., Probability, Random Variables and Stochastic Processes, McGraw Hill, 1985.


 Advantage of Linear Memory PEs

One of the problems of fully recurrent systems is that they tend to work as finite-state machines, that is, their PEs tend to saturate on and off during operation. If we think of Markov models (Rabiner and Juang), the other technology for dealing with extracting information in time, this seems reasonable, in fact, essential for capturing the complex structure of time signals.

The problem is that recurrent systems take a long time to train, if they can be trained at all. The difficulty has been called the long-term dependency problem. It simply means that when we train a recurrent net with PEs that are saturated, the gradients are going to be heavily attenuated, so the relationships are either poorly learned or impossible to learn.

This is where linear memory structures come to our rescue. The advantage of linear memories is that the gradients are not attenuated when backpropagated through them. The disadvantage is that they are unable to create nonlinear relationships (states). However, a clever mix of memory PEs and nonlinear PEs, as done in TLFNs, has been shown to provide better results than fully recurrent topologies [Giles]

Return to text


 Algorithm Locality and Distributed Systems

One of the appealing issues of gradient-descent learning is the locality of the computation. Recall that any of the LMS, delta rule, and backpropagation algorithms use only local quantities available at the weight to perform their adaptation. This is crucial for the efficiency of the algorithms, in particular for distributed system implementations. If the algorithm was not local, the necessary variables would have to be fetched from intermediate storage or across the neural network. This means that there would be a great deal of overhead for algorithm implementation and, more important, that centralized control was necessary. Adaptation would lose the biological appeal that locality provides.

Complexity imposes stringent constraints on the system design. One of the luxuries that the designer must give up is centralized control, because otherwise most of the system resources will be eaten up by the control, rather than used to perform the required function. This is the beauty of distributed systems with local rules of interactions: They do not require centralized control. They may potentially be the only way to break the barrier of system complexity, as exemplified by our own brains.

Going back to the learning rules, let's appreciate for a minute the formidable task of moving a distributed system with thousands of parameters towards its optimal parameter set. There is no one orchestrating the change in weights. Each weight receives an error and an activation and, independently of all the other weights, changes its own value using only two multiplications. Overall the system approaches the optimal operating point, but centralized control is not necessary. This is a great model to construct complex systems.

Return to text


 Almeida's Adaptive Step Size

The idea is very similar to the adaptive step size algorithm previously described, but it is more stable due to the inclusion of nonlinearity. As before, there will be a different step size for each weight. The reasoning of the algorithm is as follows:

  1. If the gradient component has the same size in two consecutive iterations, the step size should be increased.

  2. If the gradient alternates sign, the step size should be decreased.

The weight update is

where

is each gradient component, and at each iteration

(4B.2)

where u and d are positive constants with values slightly above and below unity, respectively. The authors suggest d [image] 1/u. The initial value of each step size is set equal for all weights. Note that unlike the delta-bar-delta, here the update of the step size is geometric in both directions (decreasing or increasing).

There are four heuristics to help control the growth of the step size:

  1. The error obtained at each iteration should be compared with the previous error. If the new error does not exceed the old error by more than 2.5 percent, then the new weight vector is accepted.

  2. If the new error is higher than the previous by more than this margin, compute a new step size as in Eq. 4B.2, but do not change the weights.

  3. If step 2 does not decrease the error in 2 to 3 iterations, reduce all the step sizes by a constant factor [image], until the error starts to decrease again.

  4. At this point, restart with the normal adaptation of step sizes.

This method works well when the gradient is known with high precision, as in batch learning. The method can be also applied to on-line adaptation, but we cannot utilize the instantaneous estimates of the gradient since they are too noisy. Instead we maintain a running estimate of the gradient at each epoch (P is the number of training set exemplars)

to be used in the adaptive step size computation.

Return to text


 Approximation Properties of RBF

The formulation of function approximation using the projection theorem (Eq. 5.1) can be directly applied to study the approximation properties of the RBF network. Sandberg showed that the RBF network is, in fact, a general function approximator. This is an existence theorem, so it is up to the designer to choose the number, localize and, set the variance and the weighting of Gaussians to achieve an error as small as required.

Again using the Stone-Weierstrass theorem, they showed that the RBFs were dense in the mapping from a metric space V to the real number line. This is not difficult because the RBFs create an algebra, and they do not vanish in V.

Return to text


 Associative Memory

We are very familiar with the concept of memory in digital computers, where a set of bits (0 and 1) are stored exactly in a memory location in the address space of our computer. The computer memory is an organization of such locations that is accessed by the processor by an address and is therefore called location addressable. We can think of the computer memory as a filing cabinet, with each folder containing the data. The processor accesses the data by searching the tag of the folder. This is why computer memory is location addressable and local.

Associative memories are very different and in many ways resemble our own memory. They are content addressable and global. Content addressable means that the recall is not done through the address location, but through the content. During retrieval of information with an associative memory, no address is used, just the input data. When one of the inputs used in training is presented to the Hebbian PE, the output is the pattern created with the storage algorithm during training (we show this later). The memory is also global in the sense that all the weights contain the memory information in a distributed fashion, and the weights are shared by all the memories that eventually are stored in the system. This is unlike the computer memory, where the data is contained locally and independently in each location.

Associative memories are therefore more robust to destruction of information than computer memories. However, their capacity is limited by the number of inputs (as we will see in this chapter), unlike computer memories, where the size of the data path is independent of the number of memory locations.

Return to text


 B.1 Introduction to NeuroSolutions

NeuroSolutions is based on an object-oriented approach to adaptive system design. The networks are broken down into a fundamental set of components that are individually simple but can be combined to create powerful systems capable of solving very complex problems.

The NeuroSolutions user interface was inspired by the process of designing an electronic circuit. Different components are placed onto a "breadboard" and are then wired together to form a circuit. The circuit is then tested by injecting signals and "probing" responses. This user interface provides much more flexibility than a typical "black box" simulator. NeuroSolutions includes a comprehensive collection of probes that allows the user/designer to monitor every aspect of the neural network during training and testing.

The unique attributes of NeuroSolutions make it ideal for interactive training. This textbook ships with a restricted version of NeuroSolutions. Because of its simplicity and power, NeuroSolutions is also ideal for neural network development.

B.1.1 Overview of This Tutorial

Section B.2 of this tutorial provides an overview of how to use the demonstrations/examples in the textbook. It presents only the mechanics necessary to run the examples. The rest of the tutorial presents the methods to modify the breadboards used in the examples.

To reap the full benefits of this interactive teaching methodology, it is critical that you feel comfortable modifying and probing the examples – this is where the real learning begins. When you start modifying the parameters yourself and seeing how the modification affects the operation of the system, you will be learning important aspects of neural networks that are not easily taught in a textbook. We are firm believers in the "learning through experimentation" technique, and we feel that you will not only achieve a better understanding of the topics, but your retention will also be improved.

Go to next section


 B.2 Introduction to the Interactive Examples

Throughout the hypertext on CD-ROM you will come across interactive examples that consist of text describing the example and then a button that will execute the example. After reading the text, just click on the [image] icon, and the NeuroSolutions example will start. Below is a figure with an example.

The examples have a common feel. A breadboard is a NeuroSolutions "document window" that may have components, text, and buttons placed on it. The example breadboards consist of three subsections as follows. The top section contains the text describing the example, the middle section contains the components that make up the example neural network, and the bottom section contains the various displays or graphs that "probe" the network.

The examples typically occupy multiple pages. To move to the next page you simply press the [image] button in the top-right hand corner. To exit the example immediately, press the [image] button. The examples in this tutorial and in the textbook are created on-the-fly using interactive macros in NeuroSolutions. Because of this, there is no way to return to the previous page of an example. Most examples are short, however, so you can exit the example and use the textbook to restart the example.

All the examples in the book are "live." The displays are created while you watch, and you can modify the parameters of the network at any time. All the network controls are grouped on one toolbar called the "control toolbar." Like other toolbars, the control toolbar can be moved virtually anywhere, but it is typically found just under the menu bar.

NeuroSolutions Control Toolbar

A completed network can be "run" by pressing the green "start" button. Running the network consists of presenting the input data to the network, allowing the data to move through and be operated on by the network, and creating the displays that show the parameters and outputs of the network. At the end of each run, the start button is deactivated to indicate that the network has completed its operation. To rerun the network, press the reset button and then press the start button.

The following commands are most commonly used to control a simulation:

Control | Name | Action
 | Start | Starts the simulation
 | Pause | Pauses the simulation
 | Reset | Resets the system counters and randomizes the network's initial conditions
 | Zero Counters | Resets the system counters without randomizing the network's initial conditions
 | Step Epoch | Runs the simulation until the end of the next epoch of data
 | Step Exemplar | Runs the simulation for one sample of data
You have just learned the fundamentals necessary to run the interactive examples. Let's try out a very simple example to reinforce these concepts.

TUTORIAL EXAMPLE 1

The goal of this example is to allow you to get used to the "look" and "feel" of NeuroSolutions examples. Just click on the [image] icon and NeuroSolutions will launch the example. Once inside the example, follow the directions in the text. After the example has completed, you will return to this point.

NeuroSolutions Example

Go to next section


 B.3 Basic Operation of NeuroSolutions

NeuroSolutions' object-oriented user interface organizes its components into palettes. Each palette of components contains a family of components with similar function. For example, all the output displays or probes are contained in a single palette. All the palettes can be found under the palette menu item.

If a palette is open, it will have a checkmark next to it. An opened palette looks and acts like a toolbar with a set of icons. The palettes can be moved and docked just like a standard windows toolbar. Below is an example of the Axon palette.

The Axon family palette

Components can be selected from a palette and stamped onto the breadboard.

B.3.1 Stamping Components

If you put the cursor over the palette and wait a few seconds, you should see a small window pop up with the name of the component in it. This is called a "tool tip", and you can use it to determine which component you are selecting from the palette. When you click on a component, the cursor becomes a stamp [image] when you place it over the breadboard. If you click again anywhere in the breadboard, the component will be copied to that location on the breadboard. This operation is called component stamping.

If you select a component from a palette and place the cursor over an existing component from the same family on the breadboard, the cursor will switch to a [image] which indicates the current selection will replace the component on the breadboard.

TUTORIAL EXAMPLE 2

This simple example opens up a new breadboard and allows you to stamp components on it. You should practice opening, closing, and moving palettes. Use tool tips to determine the name of the different components and then stamp a few on the breadboard. Since some families cannot be stamped directly on the breadboard, we suggest you start with the axon palette.

The Axon family palette

If you select a component that cannot be stamped down, you may need to click the "selection cursor" button [image] to return to "selection mode."

NeuroSolutions Example

B.3.2 On-Line Help

A complete description of every component is contained within the on-line help. The easiest way to access the help for a given component is to click on that component with the Help cursor [image]. The Help cursor is located on the toolbar. Just click on it and then move the mouse over the component you want help for, clicking on the component's icon.

TUTORIAL EXAMPLE 3

This simple example allows you to experiment with using the help cursor to describe each component. This handy feature allows you to quickly and easily determine the function of each component. For now, don't worry too much about the contents of the help files, just practice using the Help cursor.

NeuroSolutions Example

B.3.3 Connecting Components

Notice that on the axon family components (as well as those of some other families), there is a double-diamond contact point on the left [image] (the FemaleConnector) and a single-diamond contact on the right [image] (the MaleConnector). The data flow in NeuroSolutions is designed to move from left to right on the breadboard. To connect two components, simply drag the MaleConnector on the right of a component to the FemaleConnector on the left of the other component. The connection is visually indicated by three lines drawn between the components.

You will notice that while you are dragging the connector, the mouse arrow changes to a move cursor [image], meaning that you can drop the MaleConnector [image] onto any unoccupied portion of the breadboard. If you place the cursor over a location that is an invalid connection, the cursor becomes a crossed circle [image], meaning that you cannot drop the MaleConnector in that place.

An alternative way to connect components is to select the first component (by clicking the left mouse button), then click the right mouse button on the second component, and select "Connect To" from the menu. The three connection lines will be automatically established.

TUTORIAL EXAMPLE 4

This example allows you practice connecting components.

NeuroSolutions Example

B.3.4 Axons and Synapses

The building-block approach can easily be applied to build adaptive or neural systems. These networks primarily consist of processing elements (PEs) tied together with "weighted" connections. The Axon family (much of the terminology in NeuroSolutions is based on neurobiology) implements the PEs in the network, and the Synapse family implements the weighted connections.

Axon family palette

Axon | Bias Axon | Sigmoid Axon
The Axon family has two functions. The components sum all of their inputs and then apply a function to that sum. The different components in the Axon family apply different functions to the summation of their inputs. Each of the different images on the Axon family icons represents the function that is applied. The Linear Axon (called simply an Axon) just passes the sum of the inputs directly to the output. The Bias Axon sums the input and adds an offset. The Sigmoid Axon applies a thresholding function to the data. A single component of the Axon family can represent any number of PEs.

Synapse family palette

Full Synapse | Arbitrary Synapse
The Synapse family is used to connect Axons together. In general, each connection in a Synapse is assigned a weight that scales the data passing through it. Adjusting these weights is how a neural network or adaptive system is trained to perform the desired task. The member of the Synapse family most commonly used is the Full Synapse and connects every PE in one Axon component to every PE in the other axon component. Thus, the Full Synapse component contains nm connections (and weights) if there are n input PEs and m output PEs. The Arbitrary Synapse allows you to select which of the nm possible connections to make between the axons.

B.3.5 Component Properties

Every NeuroSolutions component has a set of parameters that you can adjust. For instance, the Axon family components contain a parameter that sets the number of PEs represented by that component. You access a component's parameter set through a dialog box called the Inspector.

  • You invoke the Inspector for any component on the breadboard by right-clicking its icon and choosing "Properties":

Invoking the Inspector

  • You can also invoke the Inspector by left-click selecting an icon and then pressing "Alt+Enter." The Inspector below shows some of the properties of the Linear Axon. Notice that number of PEs in an Axon component can be defined via rows and/or columns, allowing you to create a matrix of PEs.

Sample Inspector

Once the Inspector is open, you can view or modify the parameters of other components by simply clicking on other icons on the breadboard.

A component's parameter set is organized according to property pages. The property pages are labeled on the tabs across the top of the Inspector. You access the various property pages by clicking on these tabs. The leftmost tab typically contains the most commonly used parameters for each component. Many of the other tabs are useful only to advanced users.

If multiple components have the same parameters, you can simultaneously edit the parameters of all the components using the following procedure:

  • Invoke the Inspector for the first component.

  • Click on the tab of the desired property page

  • Hold down the Shift key while left-clicking to select the remaining components.

  • Edit any parameter(s) within the Inspector. Any changes you make in the Inspector will be reflected in all selected components, so long as they all have the same parameter.

For example, you can simultaneously change the number of PEs for a group of Axons, even if they have different transfer functions (linear, sigmoid, etc.).

TUTORIAL EXAMPLE 5

This example starts with a small network of connected components and allows you to manipulate the properties of those components with the Inspector.

NeuroSolutions Example

B.3.6 The Data-Flow Controller and Running the Network

The last family we will discuss in this section is the Control family. The Control family is different from the Control toolbar, but they are related in operation. The Control family contains components that control how the data flows through the network. The simplest member of the Control family is the Static Controller [image]. Its properties include parameters such as the amount of data in the input file, how many times to run the data through the network, whether and how the network should learn, and so forth. Thus, pressing the Start button on the Control toolbar tells the Static Controller to begin sending data through the network. The Control toolbar and Controller component work together to control the network.

The Control toolbar

The Control palette

To explain the properties of the Static Controller, we must define a few terms. An exemplar of data is one complete sample of the input for one time-step. If the network has multiple inputs, then an exemplar corresponds to one piece of data from each input. An epoch is a complete set of data presented to the network. Normally an epoch is the number of data samples in the training set file. If there are 30 samples in a data file, then there are 30 exemplars in an epoch. Many times you may want to present the same data to the network multiple times – this is represented as the number of epochs per run.

If you double-click on the Static Controller, you will see a real-time view of the simulation's progress:

Simulation progress display

TUTORIAL EXAMPLE 6

This example starts with a small network of connected components and a Static Controller and allows you to run the network using the control toolbar.

NeuroSolutions Example

We have now covered the fundamental techniques used in NeuroSolutions. The next two sections discuss the two most important families of components for use in the interactive book. These two families are the Probe family, which allows you to view the parameters or data in the network, and the Input family, which allows you to modify the inputs to the examples.

Go to next section


 B.4 Probing the System

The Probe family is the most powerful feature in NeuroSolutions. Obtaining a good grasp of the use of the Probe family will greatly increase your ability to interact with the examples.

Probe family palette

Each Probe provides a unique way of visualizing the data available throughout the network. Since the probes need data to display, they cannot be placed directly on the breadboard; they must be placed on a network component with a data access point. For instance, the Axon and Synapse families all have multiple access points. The activity access point contains the output data from the component (after operation by the component), the weight access point contains the weights of the component, and so on.

Let's assume we want to place a Matrix Viewer probe on an Axon to demonstrate some of the key probing concepts. The Matrix Viewer probe simply shows the data in numeric format.

Matrix viewer

When you place a Matrix Viewer on an Axon, one of its page tabs will be the Access property page. This will list all the possible access points of the Axon.

Access property page

Selecting the access point in the Inspector determines what data the Matrix Viewer will show. The Matrix Viewer will move its position on the Axon slightly depending on which access point you select. The last thing we need to do is double-click on the Matrix Viewer icon to open its display window.

TUTORIAL EXAMPLE 7

This example loads a small network of connected components and a Static Controller, and allows you to place Matrix Viewers on the network and view the results.

NeuroSolutions Example

B.4.1 Static Probes

There are two major types of probes in NeuroSolutions: static probes and temporal probes. Static probes like the Matrix Viewer simply display the data at a specific instance in time. Temporal probes display data over time, like an oscilloscope or a graph of an output over time. Some of the more common static probes are shown below:

Icon | Name | Description
 | Matrix Viewer | Displays instantaneous data as a numerical matrix
 | Matrix Editor | Similar to the Matrix Viewer except that it also allows you to edit the data you are probing
 | Bar Chart | Displays the data in a bar chart form
 | Image Viewer | Displays the data as a gray-scale image

There are times where you might want to display a specific piece of data in two different ways – for instance, you may want both the numerical value (Matrix Editor) and a Bar Chart. To do this, you simply stamp the second probe on top of the first probe: this is called stacking, and the access point in the Inspector of the second component will simply say Stacked Access.

TUTORIAL EXAMPLE 8

This example loads a small network of connected components and allows you to place static probes at various locations on the network.

NeuroSolutions Example

B.4.2 Temporal Probes

Temporal probes display data from multiple iterations of the network. For instance, some of the previous examples used a MegaScope to display the sine wave. The sine wave was presented to the network one sample at a time, but the MegaScope and its cohort, the Data Storage component, displayed the time history of the sinewave.

Icon | Name | Description
 | MegaScope | Displays a line graph of the data similar to an oscilloscope
 | Data Storage | Stores data from multiple iterations of the network for display by the temporal probes
 | Scatter Plot | Displays the data as a set of points on an X,Y set of coordinates

The Data Storage component stores data over time as the network runs so that the temporal probe can display the data. Some of the properties of the Data Storage component are shown below.

Data Storage Inspector

Notice that the Data Storage component allows you to select the amount of data to store (buffer size) and how often it should send the data to the temporal probe (message every). The message every property determines how often the display is updated.

To create a new temporal probe, first stamp a Data Storage component and make sure it is at the correct access point. Then stack a temporal probe on top of it and adjust its parameters if necessary. For instance, the MegaScope allows you to display multiple signals, change the color of each signal's trace, and change the scale of both axes as well.

TUTORIAL EXAMPLE 9

This example allows you to create and experiment with temporal probes and the Data Storage component.

NeuroSolutions Example

B.4.3 Data Transmitters

The probes display the data that they receive from the component they are stamped on. What if you want to display data from two different components on the same probe? The Data Storage Transmitter is the answer. You place a temporal probe on one component then place a Data Transmitter on the second component and tell it to transmit its data to the probe. The Data Storage Transmitter is on the Transmitters palette, and it looks like a miniature barrel [image] when placed on the Breadboard. It is the rightmost component on the palette.

Transmitters palette

To setup the Data Storage Transmitter, you must do two things. First, make sure that it is located at the correct access point (e.g. , will be sending the correct data). Second, you must tell it which Data Storage component to send the data to.

Data Storage Transmitter Inspector

On the Transmitter page of the Inspector you will see a list of possible receivers (Data Storage components). The receivers list shows the name of each Data Storage component on the Breadboard – the name is taken from the Component Name property on the Engine page of each component. When you click on one of the receivers, a list of possible actions shows up in the action list. Double click on the action of choice to make the connection (indicated by a "C"). In this case, we want the AttachToBuffer action, which sends the data from the Transmitter to the Data Storage component.

TUTORIAL EXAMPLE 10

This example uses a Data Storage Transmitter to create a single plot showing the input and the output of the network.

NeuroSolutions Example

Were you were wondering what the other transmitters on the Transmitters pallette do? They can be used to signal events to other components. For instance, the Threshold Transmitter can be used to tell the Static Controller to stop training when the error has reached a certain point.

This section has covered all you need to know to create probes in NeuroSolutions. Remember that probing a neural network may be the only way to fully understand how it is working. The next section covers the Input family, which allow you to use your own data in the neural networks.

Go to next section


 B.5 The Input Family

The Input family of components provide data to the neural network. In a few of the previous examples, we have used a signal generator to input a sine wave into the network.

Input Family palette

The three most common Input components are:

Icon | Name | Description
 | Function Generator | Generates sinusoids, square waves, ramps, or user- defined functions for input to the network
 | File component | Reads a variety of file types and provides the data as input to the network
 | Noise | Generates uniform or Gaussian noise; often stacked on top of other components to add dither
Like the Probe family, the Input family components are stamped onto access points of components in the network. You must select the correct access point to ensure that the data is entering the network at the correct place. The most common location for an Input family component is at the preactivity access point of the first (input) Axon in the network.

The Function Generator is very simple to use. Simply stamp the Function Generator on the appropriate component, check to ensure it is on the proper access point (the access page of the Signal Generator Inspector), and select the desired waveform. You can select both the shape of the waveform (sinusoid, square wave, etc.) and the number of samples per cycle.

Function Generator Inspector

TUTORIAL EXAMPLE 11

This example uses the previous networks but allows you to experiment with the Function Generator.

NeuroSolutions Example

B.5.1 The File Component

The File component is probably the most commonly used Input component. It reads data from the file system (e.g., your computer disk) and provides that data as input to the network.

The File component

Using the File component is rather easy, but there are quite a few options. First add a File component to the Axon and ensure that it is at the proper access point. Next, open the File List page in the Inspector on the File component.

File List page of the File component Inspector

If there are files present that you would like to remove, click on the file and hit the Remove button. If you would like to add a new file to be read into the network, follow these steps:

  • Click the Add button, find the data file you want to add by navigating through the file system, and then double-click the filename. The Associate File panel will appear, where you specify the file type and how it is to be used:

Associate File panel: set for testing with an ASCII-column file

  • Choose a translator from the list. NeuroSolutions offers translators to cover the vast majority of file types:

Translator | Format Description
ASCII | ASCII file format with no column headings
ASCII-Column | ASCII file format with descriptive column headings
Binary | 32-bit floating-point
Bitmap | Series of B&W bitmap pictures
Color Bitmap | Series of color bitmap pictures
DLL | For loading custom formatted files
Interactive Book | Special format designed for use with this book
NeuroSolutions provides three predefined data sets. The Training data set is the default data set and is used to train the network. Even though we haven't started training networks yet, you should always use the training set unless otherwise instructed. The Testing and Cross Validation data sets are used to test or verify that the network is learning appropriately. Since the Testing and Cross Validation sets are not used to train the network, they can be used to determine how the network will perform on data it has not seen yet. For now, always use the Training data set.

  • Next, Click the Close button.

  • If you have a data file with multiple columns, the Customize Column panel will be displayed next and allow you to select which columns you want to use as input. The default is to use every column of data in the file. If there are columns you would like to skip, select each column and hit the Skip button.

The Customize Column panel

  • Click the Close button on this panel when you are done.

  • If the data file has more columns than there are PEs in the Input Axon, NeuroSolutions will ask you if you'd like to modify the number of PEs in the Input Axon – usually you can just click "Yes"

That's all. There are quite a few steps, but it is relatively straightforward. If you add multiple files to the same File component and data set, then the data will be added together before entry into the network.

One other thing that will come in handy is the ability to normalize your files. If you open the Inspector on the File component and go to the stream page, there is a checkbox labeled Normalize. When this is checked, the data file will be normalized between the two values shown in the edit boxes left of the Normalize chechbox. This feature is used quite frequently in neural networks.

The Stream Inspector page

TUTORIAL EXAMPLE 12

This example allows the user to manipulate the file component and load data files into the network.

NeuroSolutions Example

Go to next section


 B.6 Training a Network

This section discusses the extra components necessary to train an adaptive system using gradient descent. Don't worry if you don't understand the concepts just focus on the mechanics. The textbook explains the concepts in detail.

Adaptive learning using gradient descent focuses on using the error between the system output and the desired system output to train the system. The learning algorithm adapts the weights of the system based on the error until the system produces the desired output. The Error Criteria family in NeuroSolutions computes different error measures that can be used to train the network.

The Error Criteria palette

The Error Criteria components are typically connected to the output of the network. They have a Desired access point that requires a member of the Input family to provide it data (e.g. a File component or Function Generator). By far the most common criterion is the L2 or Mean Squared Error (MSE) criterion. It simply computes the difference between the system output and the desired signal and squares it.

Icon | Name | Description
 | L1 Norm or Absolute Value | The absolute value of the error between the system output and the desired output
 | L2 Norm or Mean Squared Error | The squared difference between the system output and the desired output
 | Lp Norm | The difference between the system output and the desired output raised to the pth power

TUTORIAL EXAMPLE 13

This example studies the Error Criteria component. In this example, we will use the StepExemplar button [image] on the Control toolbar to single-step the network so we can analyze the error computation.

NeuroSolutions Example

B.6.1 Learning and Backpropagation

Now that we know how to compute the error, we can use the error to modify the weights of the system, allowing it to learn. The goal of the system is for the system output to be the same as the desired output, so we want to minimize the mean squared error. The method used to do this is called error backpropagation. Essentially, it is a three-step process. First, the input data is propagated forward through the network to compute the system output. Next the error is computed and propagated backward (thus the name backpropagation) through the network, and then it is used to modify the weights.

NeuroSolutions implements backpropagation of the error in a secondary "plane" that sits on top of the Axons and Synapses. This is called the backpropagation plane.

A network with a backpropagation plane

NeuroSolutions shows the backpropagation plane using smaller versions of the Axons and Synapses stacked on top of them. The backpropagation plane passes the errors backward from the Error Criteria component to the beginning of the network (and manipulates the errors along the way). NeuroSolutions adds a third plane that actually uses the errors in the backpropagation plane to change the weights in the network – this is where the learning actually happens.

This plane is called the gradient descent plane and sits on top of the backpropagation plane. A typical gradient descent component is the Momentum component [image]. Notice in the figure below that only the components with weights use the gradient descent components.

Network with all 3 planes – the forward, the backpropagation and the gradient descent planes

The last thing we need to discuss is the Back Static Controller [image]. This controller sits on top of the Static Controller and controls the backpropagation and the gradient descent planes.

The details of all these components are not necessary at this point, so we will summarize what we have learned in an example.\

TUTORIAL EXAMPLE 14

This example uses the Breadboard from the previous example, sets up the backpropagation plane and the gradient descent plane and allows the system to learn.

NeuroSolutions Example

Go to next section


 B.7 Summary

This concludes the tutorial for NeuroSolutions. Remember that the on-line help for NeuroSolutions is quite extensive. Also remember that the help cursor can be used to quickly bring up details on each of the NeuroSolutions components.

We'd like to make one additional comment. We have tried to convey in this tutorial the basics necessary to use the examples in the book, which focus mainly on adapting the example breadboards. NeuroSolutions has a number of features we have not discussed here, such as the Neural Wizard. The Neural Wizard asks you questions through a series of dialog boxes and creates an entire neural network for you, including input files, probes, backpropagation planes, and so on.

As we mentioned at the beginning, don't worry if you don't understand everything in this tutorial. Many of the concepts in NeuroSolutions will make more sense after you understand more about neural networks. As you read through the book, you may want to revisit this tutorial to go through it in more detail or just refresh your memory.


 Backpropagation Derivation

In this derivation we drop the dependence on the pattern for clarity. First assume that the kth PE is the only output PE in the net. With the machinery of the chain rule, let us write the gradient of the cost with respect to the weight as the product of the output error propagated to the PE (the gradient with respect to the PE state) times the sensitivity of the PE output with respect to the weight:

(3B.1)

where we substituted the definition of the local error given by Eq. 3.21. Notice that the sensitivity of the cost with respect to the weights is decomposed into the sensitivity of the cost with respect to the state yi times the sensitivity of the state with respect to the local weights. The state sensitivity [image] J/ [image] y can be computed by the chain rule, which yields from the figure

(3B.2)

Remember that the index k denotes a single PE in the layer to the right (i.e., in the layer closer to the output) of the ith PE. Now let us substitute these partial derivatives with network quantities and plug them in Eq. 1 to yield.

(3B.3)

This expression computes the gradient of the cost with respect to the weight wij. We assumed a single output PE in this derivation. For completeness the case of multiple output PEs is treated next.

The idea is basically the same, the only difference is that now there are many output PEs (denoted by k) connected to the ith PE, each contributing additively to the gradient of the cost with respect to the PE state, that is,

(3B.4)

Substituting back Eq. 3B.1 in the original equation Eq. 3B.1, we finally get

(3B.5)

According to the rules of gradient descent learning, we just change the weights proportional to the negative of Eq. 3B.2

(3B.6)

The algorithm just presented is the backpropagation algorithm. The expression that we arrived at seems pretty daunting, with a summation and lots of indices, but in fact can be easily interpreted.

Computer Algorithm

Let us assume a sample by sample presentation of data, that is,{x(n), d(n)}.

Step 1: Present an input –desired response pair {x1,d1}

Step 2: Compute the outputs of every PE starting from the input layer (l = 1) up to the output layer (l=L). We can formally write this step as

where f(.) is the nonlinearity, the superscript means layer l=1,…L, n is the iteration number, and wij is the weight that links the ith PE to jth PE. In the first layer [image]. If the PEs have biases, [image]. If the PE is in the output layer, make [image].

Step 3: Compute the injected error as ei(n) = di(n) – yi(n)

Step 4: Compute the local errors starting from the output layer and continuing until the first layer. In the output layer the error is (delta rule)

In all the other layers

Once we have these local errors and the activations of step 2, every weight in the network can be updated according to Eq. 3B.6. Note also that the weights to compute the errors and the activations are the "old weights."

Step 5: Repeat this procedure for every input pattern and for the number of iterations required for convergence. For best results the patterns should be randomized from presentation to presentation.

This summarizes the computer algorithm to implement backpropagation.

Return to text


 Backpropagation Versus BPTT

If the network is static or dynamic but feedforward suchas the TDNN, and the desired signal is known for all time, there is no point in using BPTT even for the case of trajectory learning. We can use straight backpropagation and add up the gradients over the specified time interval, as we did in batch learning. This can be easily proved by analyzing Eq. 11.4. However, there are cases where the dynamic network is feedforward (TDNN) but the desired signal is known only at the terminal time. In this case we have to use BPTT, since we do not have an explicit error at each time step.

As specified in the text, once BPTT is applied to one portion of the network (for instance, the feedback parameter in the gamma network), the easiest way is to adapt ALL the weights with BPTT, although strictly speaking, only the weights that lie in the topology to the left of the recurrent connection need to be updated with BPTT. However, it is easier to implement the learning algorithm uniformly.

Return to Text


 Batch Versus On-line Learning

The on-line and batch modes are slightly different, although both will perform well for parabolic performance surfaces. One major difference is that the batch algorithm keeps the system weights constant while computing the error associated with each sample in the input. Since the on-line version is constantly updating its weights, its error calculation (and thus gradient estimation) uses different weights for each input sample. This means that the two algorithms visit different sets of points during adaptation. However, they both converge to the same minimum.

Note that the number of weight updates of the two methods for the same number of data presentations is very different. The on-line method (LMS) does an update each sample, while batch does an update each epoch, that is,

LMS updates = (batch updates) x (# of samples in training set).

The batch algorithm is also slightly more efficient in terms of number of computations.

Return to text


 Bayes' Classifier

The optimal classifier (also called Bayes' classifier) is obtained in the same form as for the 1-D case. We substitute the means and covariances estimated from the data for each class in Eq. 2.8. The inverse of the covariance for the class women is

and the determinant of [image] is 0.3256, yielding

For the class men the discriminant is

The separation surface is obtained by equating gw(x)= gm(x) which yields

This is a quadratic in 2-D space as shown in Figure 2-5. This surface yields the smallest classification error for this problem, but just by inspection of the figure we can see that many errors are going to be made. Thus we have to get used to the idea that optimal does not necessarily means good performance. It simply means the best possible performance with the data we have.

Return to text


 Bayesian Threshold

The general methodology to compute the Bayesian Threshold is to substitute the likelihoods and find the value of x (temperature) which gives us equal a posteriori probabilities. For Gaussian distributions (Eq. 2.3) notice that x appears in the exponent, which complicates the mathematics a bit. Taking the natural logarithm of each side of the equation simplifies the problem without changing the solution (since the logarithm is a monotonically increasing function). Let us do this for the two-class case. We get

Now taking the logarithm

it is clear that the solution involves solving a quadratic equation in x. With simple algebra the solution can be found, which corresponds to the threshold T,

The solution is rather easy to find when [image] 1 = [image] 2, since in this case the second-order term vanishes, and the result is

where k depends on the ratio of a priori probabilities. This solution has a clear interpretation. When the variances are the same and the classes are equally probable, the threshold is placed halfway between the cluster means. If the two variances in our problem were the same the value for T= 37.75 [image] C .

The a priori probabilities shift the threshold left or right. If the a priori probability of class 2 is smaller than the a priori probability of class 1, the threshold should be shifted toward the class with smaller probability. This also makes sense because, if the a priori probability of one class is larger, we should increase the region that corresponds to this class to make fewer mistakes.

For the general case of [image] 1 different from [image] 2 we have to solve the quadratic equation. The distributions intersect at two points (two roots), but only one is the threshold we are looking for (it has to be between the means of our distributions). In our case for 1,000 measurements, the solutions are x1 =34.35 and x2 =37.07 , so the threshold should be set at 37.07 [image] C to optimally classify sick from healthy. Notice that this result was obtained with the assumptions of Gaussianity, the a priori probabilities chosen, and the given population (our measurements).

Note that the different variance of the classes effectively moved the threshold to the left, that is, in the direction of the smallest variance. This makes sense because a smaller variance means that the data is more concentrated around the mean, so the threshold should also be moved closer to the class mean. Therefore we conclude that the threshold selection depends on the variances of each cluster. We have discovered that for classification we need a new distance that is not only a function of the distance between cluster means, but also the variances of the clusters.

Return to text


Bayesian Threshold2

 Bayesian Threshold

The general methodology to compute the Bayesian Threshold is to substitute the likelihoods and find the value of x (temperature) which gives us equal a posteriori probabilities. For Gaussian distributions (Eq. 2.3) notice that x appears in the exponent, which complicates the mathematics a bit. Taking the natural logarithm of each side of the equation simplifies the problem without changing the solution (since the logarithm is a monotonically increasing function). Let us do this for the two-class case. We get

Now taking the logarithm

it is clear that the solution involves solving a quadratic equation in x. With simple algebra the solution can be found, which corresponds to the threshold T,

The solution is rather easy to find when [image] 1 = [image] 2, since in this case the second-order term vanishes, and the result is

where k depends on the ratio of a priori probabilities. This solution has a clear interpretation. When the variances are the same and the classes are equally probable, the threshold is placed halfway between the cluster means. If the two variances in our problem were the same the value for T= 37.75 [image] C .

The a priori probabilities shift the threshold left or right. If the a priori probability of class 2 is smaller than the a priori probability of class 1, the threshold should be shifted toward the class with smaller probability. This also makes sense because, if the a priori probability of one class is larger, we should increase the region that corresponds to this class to make fewer mistakes.

For the general case of [image] 1 different from [image] 2 we have to solve the quadratic equation. The distributions intersect at two points (two roots), but only one is the threshold we are looking for (it has to be between the means of our distributions). In our case for 1,000 measurements, the solutions are x1 =34.35 and x2 =37.07 , so the threshold should be set at 37.07 [image] C to optimally classify sick from healthy. Notice that this result was obtained with the assumptions of Gaussianity, the a priori probabilities chosen, and the given population (our measurements).

Note that the different variance of the classes effectively moved the threshold to the left, that is, in the direction of the smallest variance. This makes sense because a smaller variance means that the data is more concentrated around the mean, so the threshold should also be moved closer to the class mean. Therefore we conclude that the threshold selection depends on the variances of each cluster. We have discovered that for classification we need a new distance that is not only a function of the distance between cluster means, but also the variances of the clusters.

Return to text


Calculation of the Orthonormal Weights

 Calculation of the Weights in Orthonormal Bases

Let us assume that the bases [image] (x) are orthonormal in D. Let f(x) be any square integrable function . The goal is to find the coefficients wi such that

Taking the inner product of f(x) with [image] i(x)

since the vectors are orthogonal. Moreover since they are of unit length (orthonormal), we get

which corroborates the interpretation that the weight can be thought as the projection in each elementary function. In general we get the pair of relations

(Note: If the signals are complex, then the coefficients are given by

where * means complex conjugate). On the other hand, if we are working with discrete spaces, this pair of equations becomes

where once again in the second equation the basis has to be the complex conjugate if the signals are complex. This pair is what we need to apply the projection theorem for orthonormal basis and provides the relationships for the Fourier transforms and Fourier series, for example.

Return to text


 Circular Convolution

It is important to note here that Eq. 8.24 implies that both time functions, x(n) and h(n), are extended with a number of zeros equal to the extent of the other function (let us say that both functions have extent N, so their total length is 2N-1). If we take the inverse DFS we see that

We know that the extent of the convolution is equal to the sum of extents of each signal minus 1, so this expression makes sense. If we did not extend the sequences with zeros, Eq.8.24 would have failed to provide the right answer. The reason can be found in a fundamental fact of discrete sequences and DFS: They are both periodic with the number of samples in the window (N).

The effect of a discrete linear system is still defined as the convolution of the input and the impulse response, but it is a circular convolution. For two sequences of length N the circular convolution also has length N, but we see that the "linear convolution" that we defined produces 2N-1 samples, so the linear convolution and the circular convolution do not provide the same results unless we extend the original time sequences with a number of zeros equal to the other sequence. In this case the circular convolution is equal to the linear convolution. See Oppenheim and Schafer for a full discussion.

Return to text


Computation of Correlation Coefficient

 Computation of the Correlation Coefficient

It is important to remember (Eq. 1B.6) that with the optimal coefficients, the regression error, [image] min interpreted as a vector, is perpendicular to the Adaline output y. This condition is called the orthogonality condition. In fact, from the figure below it is easy to see that the smallest error is obtained when the projection of d on y is the orthogonal projection.

During adaptation the error will always be larger than [image] min, meaning that y can be larger than d, so Eq.1.14 may be larger than 1, which is misleading since |r| < 1. Using the fact that the minimum error is perpendicular to y, we can compute the dot product of [image] with y and subtract it from the numerator of Eq.1. 7. We can prove that this new numerator is always smaller than d and that the dot product is zero at the optimal solution and so will not affect the final value of the correlation coefficient. This is exactly what is done in Eq.1.14 .

Return to text


 Concept of Phasor

A phasor is a unit length vector rotating at a constant speed as proposed by Steinmetz. It can be represented by

where the rotation frequency is w. Euler's relation states that

which means that a phasor is a "complex" waveform that gives rise to our well-known sinusoidal waveforms (see the following Figure).

We can imagine Euler's relation and the relation between phasors and real sinusoids if we observe this figure. The goal is to have the phasor rotating and plot its projection on the real line as a function of time.

When the vector is aligned with the real axis at time zero, the amplitude is 1. When it starts rotating, the amplitude decreases monotonically on the real axis (and increases on the imaginary axis), until it is zero for [image] [image] = [image] /2 (it is 1 on the imaginary axis). Then it starts to be negative and is maximally negative for [image] [image] = [image] (at this point is zero on the imaginary axis). At [image] [image] = 3 [image] /2, its amplitude decreases until zero, and then increases again until [image] [image] = 2 [image]. It repeats this behavior for larger values of the angle [image] [image] [image] so we see that in time it gives rise to a periodic wave with period 2 [image].

We can also show that if the rotating speed is constant, the projections must be sinusoidal with a constant frequency. This, we can think of our well-known sinusoid as the projection on the real line of the complex phasor. Notice also that when the rotating speed is faster, the frequency of the sinusoidal projection increases.

The complex vector u(t) can be represented by its modulus (length) and phase as

For the phasor we see that the modulus is 1, and the arctangent is simply [image] t. If we want to plot each one of these functions (now real) independently over time, we obtain what are called respectively the magnitude and the phase representation for the phasor. Being a complex quantity, the phasor requires two plots to be represented over time.

Notice that the phase plot is discontinuous, due to the arctan function, but can be made continuous by adding 2 [image] at the jumps (phase unwrapping).

Alternatively, we could plot the real and the imaginary parts of the phasor to obtain a cosine for the real part and a sine for the imaginary part, as Euler's relation shows.

We see that the phasor is the elementary function in the complex domain, exactly as the sine wave is the elementary function for time representations. Every time we encounter a phasor, we should think of a vector of unit length rotating at a constant speed [image]. Since we started this study in vector spaces, the phasor seems the best way to introduce harmonic or spectral analysis. However, an equivalent explanation can be given in terms of sine waves. The phasor's rotating speed actually corresponds to the frequency of the sinewave.

Return to text


 Conjugate Gradient Method

A set of vectors {sj} is conjugate with respect to a positive definite matrix (e.g., the Hessian) if

What this expression says is that the rotation by H of the vector s j must by orthogonal to si. In the n dimensional Euclidean space R [image] there are an infinite number of conjugate vector sets. It is easy to show that the eigenvectors of the Hessian form a conjugate set and can then be used to search the performance surface. The problem is that we need to know the Hessian, which is not a practical assumption. However, there is a way to find a conjugate set of vectors that does not require the knowledge of the Hessian. The idea is to express the conditions for a conjugate vector set as a function of differences in consecutive gradient directions as

For this expression to be true, the minimum of the gradient of J(i) in the direction s(j) is needed, so the algorithm works as follows.

Start with the gradient-descent direction, s(0) = [image] [image] J(0). Search the minimum along this direction. Then construct a vector s(j) that is orthogonal to the set of vectors

,

which can be accomplished by

There are basically three well-known ways to find [image], namely the Fletcher-Reeves, the Polak-Ribiere, or the Hestenes-Steifel formulas, which are equivalent for quadratic performance surfaces and are respectively given by

In quadratic performance surfaces, you will find the minimum in n iterations, where n is the size of the search space. The minimization along the line can be accomplished for quadratic performance surfaces as Eq. 4B.1. The problem is that, for non-quadratic performance surfaces such as the ones found in neurocomputing, quadratic termination is not guaranteed and the line search does not have an analytic solution.

The lack of quadratic termination can be overcome by executing the algorithm for n iterations and then reseting it to the current gradient direction. The problem of the line search is more difficult to solve. There are two basic approaches: direct search or the scaled conjugate method (Shepherd). The first involves multiple cost-function evaluations and estimations to find the minimum, which complicates the mechanics of the algorithm. The scaled conjugate is more appropriate for neural network implementations. It uses Eq. 4.24 and avoids the problem of nonquadratic surfaces by massaging the Hessian so as to guarantee positive definiteness, which is accomplished by H + [image] I, where I is the identity matrix. Eq. 4B.1 becomes

(4B.3)

At first, we may think that this method is more computationally expensive than search, because of the Hessian matrix. But in fact, this is not the case, since there are fast methods to estimate the product of a vector by the Hessian (the product has only n components). We can use the perturbation method to estimate the product (see LeCun, Simard, and Pearlmutter):

or use an analytic approach due to Pearlmutter. Both methods are compatible with backpropagation.

We still need to address how to set [image] which is not difficult, but involves trial and error. The idea is as follows: if the error increases from one step to the next, it is because we are in an area of J that is far from quadratic, so the Hessian is not positive definite. In such cases we should increase [image] until the error decreases. Notice that for large [image] the denominator becomes approximately [image]. In this case we are essentially using gradient descent, which is known to be convergent to the minimum (albeit slowly). When the error decreases, [image] should again be decreased to fully exploit the potential of the local quadratic information of the performance surface.

Return to text


Continuous and Digital Fitlers

 Continuous and Digital Filters

This equation corresponds to the simplest analog lowpass filter, which is described by the first-order differential equation

(8B.1)

and can be built with a capacitor and a resistor with time constant [image]. If we approximate the differential by the forward difference,

(8B.2)

Eq. 8.13 is obtained. This equation is called a difference equation due to the form of the difference operator used to approximate the differential.

There are important relationships between continuous time systems described by differential equations and the discrete systems we are studying with difference equations. In fact, the forward difference operator and the substitution of t by nT allow us to go from continuous time to discrete time (but there are other more rigorous ways of linking the two domains). T, the sampling period, must be very small to preserve the goodness of the approximation.

Another important characteristic of both the linear combiner and the recurrent system is that they have responses only after the input is applied. Such systems are called causal, i.e. non-anticipatory, and they are the only ones that can be built in analog hardware. Any physically realizable system is causal. With computers we can play tricks and implement noncausal FIR systems. The idea is to use memory and delay the output of the system with respect to real time so that it responds to events that happen in the near future.

Return to text


 Convergence for Multiple Weights Case

One can show that the condition to guarantee converge (see Widrow and Stearns) is

(1B.20)

where [image] is the eigenvalue matrix,

(1B.21)

which means that in every principal direction of the performance surface (given by the eigenvectors of the input correlation matrix R), we must have

(1B.22)

where [image] i is the corresponding eigenvalue. This equation also means that with a single [image] each weight wi(k) is approaching its optimal value wi* with a different time constant ("speed"), so the weight tracks bend, and the path is no longer a straight line toward the minimum.

This is the mathematical description of our earlier statement that the gradient descent algorithm behaves like many one dimensional univariable algorithms along the eigenvector directions. Notice that Eq. 1B.21 is diagonal, so there is no crosscoupling between time constants along the eigenvector directions.

In any other direction of the space there will be coupling. However, we can still decompose the overall weight tract as a combination of weight tracts along each eigendirection as we did in Figure 1-16. Eq. 1B.22 shows that the step size along each direction obeys the same rule as the unidimensional case (Eq.1.17).

Return to text


 Convolution Operation

Although very important, the convolution operation is tricky, since it can not be computed in a point-by-point fashion, as for instance, a change in scale. If we are given an input and the impulse response, convolution is NOT a simple point-by-point multiplication of the two signals.

The equation specifies

If the system is causal (i.e., if the system does not have an output before the input is applied), the response to an input applied at n = 0 can be simplified to

Let us read this equation carefully. This construction is better visualized in the i domain. Notice that for each sample of y (a value of n), we have a sum of terms over the i variable (see the following Figure). These terms are the multiplication of the input x(i) with the impulse response reversed in time h(n-i). One of the two waveforms is therefore flipped in time. The value of the output is the sum of products, i.e. the area under the intersection of the two waves. When time progresses (the index n increases), the relative positions of the input and the impulse response change, since the impulse response is affected by the index n. Hence the total length of the response is the sum of the lengths of each wave (minus 1). Notice also that the convolution can be computed alternatively by switching the roles of x(.) and h(.). The following Figure exemplifies the graphic construction of the convolution.

Remember that the convolution transforms the product of the overlapped samples (between the input and impulse response) at time n into an amplitude value at the output of the filter. This is strange, but effectively it is what ANY linear system does!

Return to text


 Correlation Coefficient and Linear Regression

Consider estimating a random variable d by a constant b according to the mean square estimation,

where E[.] is the expected value operator (see Appendix A). The best b is obtained by taking the derivative with respect to b and setting the result to zero, yielding b* = E[d]. If we substitute this value in the equation, we obtain the variance of d as the smallest error.

Now if we try to approximate d by wx + b we obtain

and the best b* = E[d]-wE[x]. Therefore the best w can be found solving the problem

It is easy to show by differentiation that the best w*

(1B.4)

Therefore the minimum mean square error linear estimator for d is

(1B.5)

The term in parentheses is a zero-mean unit variance version of the input x, so the product with [image] scales it by the variance of d. The term E[d] just guarantees the correct mean.

It is interesting to note that if d and x are uncorrelated, the best estimate of d is its mean. However, if x and d are exactly correlated, the best estimate is highly improved ( [image] in Eq. 1B.5). The minimum mean square error is

This equation shows that in fact [image] can be interpreted as the amount of variance in the data that is captured by the linear model.

There is a very interesting interpretation of the mean square estimation solution. Note that

(1B.6)

which means that the error (the quantity inside the curly braes) is orthogonal to the input.

Return to text


 Cover's Theorem

Cover's theorem draws from a very interesting (and nonintuitive) result. Let us assume that we have a kernel-based machine with M+1 weights. We want to classify N patterns. There are 2 [image] possible divisions of these patterns into two categories (called dichotomies). The question is what is the probability that a given dichotomy chosen at random is implementable by our kernel-based machine?

Cover showed that the probability is

This means that for spaces of size M larger than the number of samples N the probability is essentially 1, that is, we can always divide any data into two classes with probability 1. This probability has a sharp knee (the probability approaches 1 rapidly) at N = 2(M+1), and this value is normally defined as the capacity C of the learning machine. Thus, if we set the number of free parameters (weights) of our learning machine above its capacity, we are almost certain to classify the input patterns correctly. For a linear classifier in D space the capacity is C = 2D+1. However, kernel-based classifiers allow us to go to a higher-dimensional space and set their capacity independently of the input space dimension. This decoupling between the input space dimension and the machine capacity is the great advantage of the kernel-based machine.

Return to text


Cross Entropy Criterion

 Cross-Entropy Criterion

To understand cross-entropy, we have to give a short definition of what is information and the related concept of entropy. Information theory was invented by Shannon in the late 1940s to explain the content of messages and how they are corrupted through communication channels. The key concept in information theory is that of information (Cover). Information is a measure of randomness of a message. If the message is totally predictable, it contains no information. On the other hand, something very unexpected has a high information content, so the information is inversely associated with the probability of an event. We can define the amount of information in a random event xk with probability p(xk) as

Entropy then becomes the mean value of I(x) over the complete range of discrete messages (2K + 1) with probabilities pk = p(xk) as

Entropy is a measure of the average amount of information contained in the event. Now assume that we have two probability mass functions {pk} and {qk}. The relative entropy of the probability distribution P (function of some event r) with respect to the second distribution Q is given by

This concept is called relative entropy and was introduced by Kullback. It is commonly called Kullback-Leibler distance Cover.

Return to text


Definition of A Posteriori Probability

 A Posteriori Probability

The probability of an event after some measurements are made.


Definition of AC

 AC

AC stands for alternating current, and in practice it qualifies the components of the signal that vary over time.


Definition of Activation

 Activation

The output of a PE in an ANN.


Definition of Adaline

 Adaline

Stands for Adaptive Linear Element, and is the processing element proposed by Widrow that implements a weighted sum of inputs.


Definition of Adaline

 Adaline

ADAptive LINear Element, the processing element proposed by Widrow that implements a weighted sum of inputs.


Definition of Adaptive Filter Theory

 Adaptive Filter Theory

A branch of signal processing that analyzes and designs filters that learn from the data. Instead of having coefficients that are set by designer specifications, adaptive filters have a learning algorithm that modifies their coefficients so that the desired response is approximated.


Definition of Adaptive Systems

 Adaptive Systems

Systems that change their parameters (through algorithms) in order to meet a prespecified goal, which is either an input-output map or an internal constraint.


Definition of Affine Transformation

 Affine Transformation

Any linear transformation of the space coordinates. An affine transformation can be considered as a rotation and a translation of the space.


Definition of Aliasing

 Aliasing

The error that is created when analog signals are sampled below the Nyquist frequency. It is called aliasing because high frequencies in the analog waveform are represented as lower frequencies in the digital signal.


Definition of Analog Signals

 Analog Signals

Signals for which both the independent variable time and the values of the signal are real numbers.


Definition of Artificial Neural Networks

 Artificial Neural Networks

ANNs for short, are adaptive, most often nonlinear distributed systems.


Definition of ASCII

 ASCII

American standard code for information interchange . It has become a standard for coding characters into binary strings.


Definition of Attractor

 Attractor

The trajectory set that defines the limit point of the dynamics.


Definition of Autocorrelation

 Autocorrelation

A measure of similarity of the samples' distribution, which is computed by the sum of the cross products between the data set and its shifted versions. The autocorrelation is a function of the shift.


Definition of Backpropagation

 Backpropagation

A training algorithm for multilayer perceptrons that extends the delta rule to hidden-layer networks. It uses the methodology of gradient-descent learning and solves the credit assignment problem by using the weight values in the topology.


Definition of Batch Training

 Batch Training

The adaptation of the weight based on an epoch update.


Definition of Big O Notation

 Big O Notation

An approximate way to express the complexity of a computer algorithm, where only the most rapidly increasing factor is shown. Normally we are interested in multiplications, since they are the most time-consuming to execute in general purpose computers.


Definition of Bounded Input Bounded Output (BIBO)

 Bounded-Input Bounded-Output (BIBO)

The most widely used criterion of stability for linear systems. It simply says that a system is stable if, for a bounded input, the response is also bounded.


Definition of Bump

 Bump

A function that has large values over a limited space extent, that is, basically a local function. A multidimensional Gaussian is an example of a bump. This loose terminology indicates well the nature of the region being created. Important theorems about function approximation with local functions exist.


Definition of Causal System

 Causal System

A system that produces a response only after an input is applied. Hence it is strictly non-anticipatory.


Definition of Classes

 Classes

Classes are the natural divisions of the input data produced by the phenomenon under study (e.g. sick and healthy).


Definition of Classification Error

 Classification Error

The number of samples that were incorrectly classified (misclassifications) normalized by the total number of samples.


Definition of Classifier

 Classifier

Classifier is a machine that automatically divides input data into classes.


Definition of Clustering

 Clustering

An unsupervised technique that groups data according to similarity (which also implies considering what is dissimilar).


Definition of Code

 Code

A systematic translation of the data into bits.


Definition of Committees

 Committees

Ensembles of ANNs trained with the same data, often with different topologies, whose output is interpreted as a vote for the classification. Committees are appealing because they decrease the variance of the final decision, which is considered one of the most difficult problems in semiparametric classifiers.


Definition of Complex Number

 Complex Number

A number z that can be written as z = Re(z) + j Im (z) where [image].


Definition of Confusion Matrix

 Confusion Matrix

A matrix in which the man-made (rows) and the machine-made (columns) per class are entered. A perfect classifier has only the diagonal populated. Errors appear in nondiagonal positions. The confusion matrix is an efficient way to observe the confusion between classes.


Definition of Connectionist

 Connectionist

Normally taken as a synonym of neural networks. The name comes from the fact that neural networks are highly distributed systems where the computation is done in the links (connections between PEs).


Definition of Context PE

 Context PE

A memory PE formed by a single first order recurrent system.


Definition of Contour

 Contour

A curve linking all the points with the same value of J (J = constant). The contour plot for J is formed by concentric ellipsoids (ellipses for the 2-D case).


Definition of Convex

 Convex

A surface is convex when any point in a line joining two points in the surface belongs to the surface.


Definition of Correlation Coefficient

 Correlation Coefficient

The ratio of the covariance between the input and desired data over the product of their standard deviations.


Definition of Covariance

 Covariance

The sum of the cross products of the two variables with the means removed.


Definition of Cover's Theorem

 Cover's Theorem

A linear classifier to asymptotically separate arbitrary data sets if the data is nonlinearly mapped to a sufficiently large feature space.


Definition of Cross-correlation

 Cross-correlation

A measure of the similarity between two different data sets, computed by the sum of the cross products between the two data sets at different lags (it is a function of the lag).


Definition of Curse of Dimensionality

 Curse of Dimensionality

An often used expression that relates to the fact that classification problems become exponentially more difficult with the size of the space.


Definition of Data-flow Machine

 Data-flow Machine

The chaining of algorithmic operations that are necessary to implement the neural network function and its training. Since a neural network is distributed, data is sent into its input and transformed when it goes through the machine, as are the errors that are used during training.


Definition of DC

 DC

DC stands for direct current, but in practice it specifies the long-term average of the signal. So its spectrum is a delta function at zero frequency, with amplitude equal to the signal average.


Definition of Decibel (dB)

 Decibel (dB)

The log of the ratio of two quantities, i.e. . Negative values mean that the ratio is less than 1.


Definition of Decision Surface

 Decision Surface

The boundary (nominally multidimensional) between the input data classes.


Definition of Decoupled Kalman Filtering Training (dKft)

 Decoupled Kalman Filtering Training (dKft)

and multistream learning have been recently proposed by Feldkamp and co-workers with very promising results. See Puskorius, G., L. Feldkamp and L. Davis, Dynamic neural networks methods applied to on-vehicle idle speed control, Proc. IEEE, vol 84, 1407-1420, 1996.


Definition of Deflation

 Deflation

A method of computing the principal components that is similar to the Gram-Schimdt orthogonalization procedure: first compute the principal direction and subtract it from the data before computing the next principal direction.


Definition of Delay

 Delay

An operator (linear system) that delays the input by one sample without distorting it.


Definition of Delay

 Delay

A type of memory in which the memory mechanism is a pure delay.


Definition of Delay Line

 Delay Line

A single input multiple output system created by a cascade of delay elements.


Definition of Delta Function

 Delta Function

A function that can be thought as the limit of a rectangular pulse of height 1/ [image] and width [image] when [image] goes to zero. Mathematically, it is a function [image] (t) that obeys the relation


Definition of Difference Equation

 Difference Equation

An equation involving two or more variables at different time steps. Of importance for our study are the difference equations that have constant coefficients.


Definition of Digital Signal Processing (DSP)

 Digital Signal Processing (DSP)

An expanding field in electrical engineering that modifies signal properties according to the user specifications using computer algorithms (instead of analog hardware, as done in the past).


Definition of Dirac Delta

 Dirac Delta

A function that is zero everywhere except when the argument is zero, where it has unit area. In discrete time it is defined as

Practically, it yields the value of any function it is multiplied with at the sample number where the impulse occurs, i.e. f(n) [image] (n-n0) = f(n0).


Definition of Discriminant Function

 Discriminant Function

A function g(x) that evaluates every position in pattern space and produces a large value for one class and low values for all the others.


Definition of Dual

 Dual

or transpose network. The network obtained from the original network by reversing the signal flow and changing summing junctions with splitting nodes, and vice versa.


Definition of Dynamic networks

 Dynamic Networks

Neural networks with short term memory or with feedback connections that have outputs depend on more than one time instant.


Definition of Dynamic Systems

 Dynamic Systems

Systems with outputs that depend on more than one time instant


Definition of Eigenfunctions

 Eigenfunctions

The natural modes of a system, a signal is an eigenfunction of a system when the system output is a signal of the same shape, but possibly of a different amplitude and phase. Eigenfunctions are related to the concept of eigenvectors in linear algebra.


Definition of Eigenvalue Spread

 Eigenvalue Spread

The ratio of the largest over the smallest eigenvalue


Definition of Eigenvalues

 Eigenvalues

The scaling constants in the eigenvalue equation of a matrix. Eigenvalues can be considered as the projections of the data along the eigenvectors.


Definition of Eigenvectors

 Eigenvectors

Vectors e for a matrix R that yield R e = [image] e, where [image] are the eigenvalues.


Definition of Epoch

 Epoch

One complete presentation of the input data to the network being trained.


Definition of Ergodic

 Ergodic

A property of some stationary random processes in which the statistical moments obtained by statistical operators equal the moments obtained with time operators. A nonstationary random process can NOT be ergodic.


Definition of Fan-in

 Fan-in

The number of inputs that feed a given PE.


Definition of Fast Fourier Transform (FFT)

 Fast Fourier Transform (FFT)

A computational algorithm that computes the Fourier transform in O(Nlog(N)) time where N is the number of samples of the input signal. Using the definition, the Discrete Fourier Transform takes O(N²). Savings are enormous for large N.


Definition of Feedback

 Feedback

Another memory mechanism that is based on a recurrent connection of the output to the input.


Definition of Feedforward

 Feedforward

Topologies with no recurrent connections.


Definition of Fiducial

 Fiducial

The name given to the point where the function is being approximated.


Definition of Filter

 Filter

A linear system designed with pre-specified frequency responses.


Definition of Filter Attenuation

 Filter Attenuation

The decrease in amplitude from the input to the output of a system at a given frequency and it is measured normally in - dB (20 log since it is amplitude).


Definition of Filtering

 Filtering

A selective removal of frequencies in the input signal to achieve a desired goal.


Definition of Finite Impulse Response (FIR)

 Finite Impulse Response (FIR)

A type of linear system for which the impulse response is of finite extent. The response of FIR systems is constructed from linear combinations of past values of the input only.


Definition of First Order

 First Order

Gradient descent is called a first-order method because it uses only information about the performance surface tangent space. Newton's method is a second-order method because it uses information about the curvature.


Definition of Fixed Point Learning

 Fixed Point Learning

A training regime for dynamic systems where the input - desired response pair is specified for all time, which means using a dynamic system as a static system (i.e. as a MLP).


Definition of Focused Networks

 Focused Networks

A terminology originally utilized to describe the fact that the gradients of such networks are time invariant.


Definition of Fourier Series

 Fourier Series

Joseph Fourier in the late 18th century showed that any periodic signal, no matter how complex, could be decomposed in sums (possibly infinite) of simple sine waves of different frequencies. These decompositions are called the Fourier series. If y(t) is real,


Definition of Frequency

 Frequency

The inverse of the period of a waveform. It is measured in cycles per second (Hertz), or radians per second.


Definition of Frequency Domain Approach

 Frequency Domain Approach

A set of methods that use frequency domain techniques to analyze and quantify linear systems. Examples are the idea of phasors, Fourier transforms, frequency response and the system transfer function.


Definition of Frequency Resolution

 Frequency Resolution

The difference between two consecutive discrete frequencies. It is equal to the inverse of the window length times the sampling period.


Definition of Frequency Response

 Frequency Response

The Discrete Fourier transform of the impulse response, or the transfer function evaluated in the unit circle.


Definition of Fully Recurrent Network

 Fully Recurrent Network

A network that has arbitrary feedback connections from input-to hidden-to output PEs.


Definition of Function Approximation

 Function Approximation

Became a field of study in the 19th century with the formal definition of a limit by Cauchy and Weierstrass. It culminated a long road of discoveries by mathematical giants such as Euler, Legendre, and Gauss, motivated by astronomical observations. The goal was to approximate difficult mathematical functions by ensembles of simpler functions. Approximation requires the definition of an error, which implies a metric space to define the distance between the true function and the approximation. Moreover, the availability of a set of simpler functions is postulated.


Definition of Functional Analysis

 Functional Analysis

The branch of analysis in which the functions do not depend on single numbers but on collections of components (eventually an entire range of a numerical function). An example is a function of a vector.


Definition of Generalization

 Generalization

The ability to correctly classify samples that were not used to train the learning machine.


Definition of Generalize

 Generalize

A machine generalizes when it produces the correct output for inputs that belong to the same class but were not used for training.


Definition of Genetic Algorithms

 Genetic Algorithms

Global search procedures proposed by John Holland that search the performance surface, concentrating on the areas that provide better solutions. They use "generations" of search points computed from the previous search points using the operators of crossover and mutation (hence the name). See Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, Addison Wesley, Reading MA1989.


Definition of Geometric Ratio

 Geometric Ratio

The ratio of two consecutive terms in a geometric progression.


Definition of Global Minimum

 Global Minimum

The minimum that achieves the smallest value of a function (normally MSE).


Definition of Gradient

 Gradient

A vector that always points in the direction of maximum change, with a magnitude equal to the slope of the tangent to the curve at the point.


Definition of Greedy

 Greedy

A greedy method is one that uses resources proportional to the size of the input.


Definition of Hessian

 Hessian

The matrix of the second derivative of the cost with respect to the weights


Definition of Highpass

 Highpass Filter

A filter that attenuates the low frequencies and lets the higher frequencies pass without attenuation.


Definition of Hyperplane

 Hyperplane

A planar surface in four or more dimensions. A hyperplane is still defined by any three points in four or more dimensions. To visualize a hyperplane in 4-D use the "flat land" approach; that is, try to visualize a plane when in a 2-D world.


Definition of Ill-Conditioned

 Ill-Conditioned

A matrix whose determinant is almost zero. When the eigenvalue spread becomes very large, the matrix becomes more and more ill conditioned. See Appendix A.


Definition of Ill-posed

 Ill-posed Problem

A problem is ill-posed when small modifications of the input produce large change in the outcome. Ill-posed problems very often arise when one tries to reverse a cause-effect relation.


Definition of Illposed

 Ill-posed

A term coined by Hadamard to mean that solutions are extremely dependent on minor changes in the form of simple equations. Ill-posed problems very often arise when one tries to reverse a cause-effect relation.


Definition of Impulse Response

 Impulse Response

The response of a linear system to an impulse occurring at n = 0.


Definition of Indicator Function

 Indicator Function

A function that takes only two values 1 and 0 (or -1).


Definition of Infinite Impulse Response (IIR)

 Infinite Impulse Response (IIR)

A linear system for which the impulse response is infinite and has some type of internal feedback mechanism.


Definition of Inverse Z Transform

 Inverse Z Transform

The inverse mapping from the z domain back to time, given by

where C is a counterclockwise closed contour encircling the origin and belonging to the region of convergence of X(z).


Definition of Kernel

 Kernel

A function that implements the delay mechanism (also called operator).


Definition of Learning Curve

 Learning Curve

A plot of the MSE across iterations.


Definition of Learning Rate Annealing

 Learning Rate Annealing

The progressive decrease of the learning rate across iterations.


Definition of Learning Rate Scheduling

 Learning Rate Scheduling

The choice of a variable step size, which starts large in the beginning of training and decreases progressively toward the end of adaptation


Definition of Least Mean Square (LMS)

 Least Mean Square (LMS)

A steepest descent search algorithm that uses a very efficient estimate of the gradient (the product of the error and the input):

w(n + 1 = w(n) + [image] x(n) [image] (n)


Definition of Least Squares

 Least Squares

An analytic procedure that minimizes the MSE in linear optimization problems (i.e., problems that are linear in the unknowns).


Definition of Likelihood

 Likelihood

Likelihood is the probability density function of each event class.


Definition of Linear Combiner

 Linear Combiner

A linear system that produces an output that is a weighted sum of past input samples.


Definition of Linear Machine

 Linear Machine

Linear machine is a parametric classifier in which the discriminant functions are hyperplanes.


Definition of Linear Regression

 Linear Regression

The process of fitting (minimizing the sum of the square of the deviations) a cloud of samples by a linear model.


Definition of Linear Systems

 Linear Systems

A system H is called linear if it obeys the following relation

which means that the principle of superposition applies ( [image] and [image] are constants).

Return to text


Definition of Linear Systems Theory

 Linear Systems Theory

A highly mathematical branch of electrical engineering that studies linear functions, their properties, and their implementations.


Definition of Linearly Separable Patterns

 Linearly Separable Patterns

Patterns that can be perfectly classified by linear machines.


Definition of Local Error

 Local Error

The product of the error that reaches the PE times the derivative of the nonlinearity at the operating point. It is really the "effective error" that is used to correct the weights.


Definition of Local Maps

 Local Maps

The definitions of the PE function and its dual network, which specify how the activations are modified when they go through the topology, and the errors when they flow across the dual topology.


Definition of Long-Term Memory

 Long-Term Memory

Information contained in the network weights.


Definition of Lowpass filter

 Lowpass Filter

A linear system that has unity gain for low frequencies (i.e. it lets low frequencies pass unattenuated up to a frequency called the cutoff frequency) and attenuates high frequencies.


Definition of Manifold

 Manifold

A space in which the local geometry of each point looks like a little piece of Euclidean space.


Definition of Margin of the Hyperplane

 Margin of the Hyperplane

The minimal distance of the set of samples to the hyperplane.


Definition of Mean Square Error (MSE)

 Mean Square Error (MSE)

The average of the square of the difference between the desired response and the actual system output (the error).


Definition of Memory Depth

 Memory Depth

The temporal extent of the system response after the application of an impulse at n = 0, that is, the number of time units at which the response of the system is different from zero.


Definition of Memory Filter

 Memory Filter

A linear structure used to store the past of the input.


Definition of Memory Resolution

 Memory Resolution

The number of taps per samples.


Definition of Memory Structure

 Memory Structure

A multidimensional single-input, multiple-output device that takes an ordered sequence of points and produces a static pattern (i.e. a point in the multidimensional space).


Definition of Memory Trace

 Memory Trace

The signal at the output of the memory PE, which represents the past of the input.


Definition of Misadjustment

 Misadjustment

The normalized excess MSE produced by the rattling.


Definition of Mixture Models

 Mixture Models

A generative model for the data composed of multiple-Gaussians. See Jacobs R., M. Jordan, S. Nowlan and G. Hinton , Adaptive mixtures of local experts, Neural Computation 3, 79-87, 1991.


Definition of Multilayer Perceptrons

 Multilayer Perceptrons

Feedforward neural networks with one or more hidden layers, that is, layers with nonlinear PEs that are not directly connected to the outside world.


Definition of Nonconvex

 Nonconvex

A surface is nonconvex when it displays more than one minimum.


Definition of Normalized LMS

 Normalized LMS

The LMS algorithm with a step size normalized by an estimate of the input data variance


Definition of On-line Training

 On-line Training

A learning procedure that modifies the weights after the presentation of every sample.


Definition of Optimal Classifier

 Optimal Classifier

Optimal classifier is the classifier that minimizes the classification error given the observations.


Definition of Optimal Discriminant Function

 Optimal Discriminant Function

Optimal discriminant function is the discriminant that produces the classification with the fewest errors.


Definition of Optimal Hyperplane

 Optimal Hyperplane

The linear discriminant that maximizes the margin.


Definition of Ordered Derivative

 Ordered Derivative

The partial sensitivity in an ordered network, which is made up of the explicit and implicit (through the network) dependencies.


Definition of Ordered List

 Ordered List

The list that summarizes the dependencies of the network topology. Each variable in the ordered list depends only on the variables to its left in the list. The weights appear first in the list.


Definition of Outlier

 Outlier

A point that, because of observation noise, does not follow the characteristics of the input (or desired response) data.


Definition of Pattern Recognition

 Pattern Recognition

Pattern recognition is the creation of categories from input data using implicit or explicit data relationships. Similarities among some data exemplars are contrasted with dissimilarities across the data ensemble, and the concept of data class emerges. Due to the imprecise nature of the process, it is no surprise that statistics has played a major role in the basic principles of pattern recognition.


Definition of Pattern Space

 Pattern Space

Pattern space is the space of the input data. Each multivariate (N variables) data sample can be thought of as a point in a multidimensional (N-dimensional) space.


Definition of Perceptron

 Perceptron

A multiple-input, multiple-output pattern recognition machine made from one layer of McCulloch - Pitts PEs. The perceptron has adaptive weights.


Definition of Performance Surface

 Performance Surface

The total error surface plotted in the space of the system coefficients (weights).


Definition of Plant

 Plant

In control theory, the system to be modeled or controlled.


Definition of Poles

 Poles

The values of z that make H(z) infinite (i.e. the zeros of the denominator polynomial).


Definition of Polynomial

 Polynomial

A rational function that can be put in the form [image] where both x and a are real numbers.


Definition of Probability Density Function

 Probability Density Function

Probability density function (pdf) intuitively, is the function that specifies the probability of a given event in an experiment. It is the limit of the histogram for arbitrary large number of trials. See the Appendix A for a more formal definition.


Definition of Processing Element

 Processing Element

The fundamental computational block in the system. In neural networks PEs are also called neurons or units


Definition of Rattling

 Rattling

The perturbation around the optimal weight value produced by a nonzero learning rate.


Definition of Recurrent System

 Recurrent System

A system in which the output depends on a previous value of the output. Hence it must have a recurrent loop where the output of an element feeds back into the system.


Definition of Region of Support

 Region of Support

The length of the impulse response, or the number of samples the output of the delay line "remembers" the input. Memory depth quantifies mathematically this concept for recurrent memories.


Definition of RLS

 RLS

An on-line algorithm to compute the optimal weights (as opposed to the batch process to solve the least squares). Unfortunately it is also much more computationally intensive than the LMS. To know more please consult, Haykin, S., Adaptive Filter Theory, Prentice Hall, 1996.


Definition of Robust

 Robust

A method is robust if it is not very sensitive to outliers. The term was coined in statistics and control theory to represent methods that have low sensitivity to perturbations.


Definition of Saddle Point

 Saddle Point

A point of zero curvature along at least one direction but not all directions.


Definition of Saliency

 Saliency

The importance of a weight for the overall input-output map


Definition of Sampling

 Sampling

An operation that creates a sequence of numbers from the analog signal in which each number is the value of the analog signal at the measurement time. Sampling is physically done by an analog-to-digital (A/D) converter.


Definition of Sampling Theorem

 Sampling Theorem

Also called the Nyquist theorem. The theorem states that we can uniquely reconstruct a time signal from its sampled representation if we sample no less than twice the highest frequency present in the signal.


Definition of Scientific Method

 Scientific Method

The methodology for discovery utilized in science. See Karl Popper, "The logic of scientific discovery", Harper Torch, 1968.


Definition of Second-Order Statistics

 Second-Order Statistics

Measured by the covariance function. We saw that a Gaussian is fully described by the mean and variance for one-dimensional data, and by the mean vector and the covariance matrix for multiple dimensions.


Definition of Sensitivity

 Sensitivity

The partial derivative of a function with respect to one of its independent variables. It measures how much a small change in one of the independent variables affects the functional value.


Definition of Series

 Series

Iterated sums of terms produced by a general rule with an eventually infinite number of terms. An example is


Definition of Shallow Networks

 Shallow Networks

Networks with few layers, that is, typically one or two-hidden layers.


Definition of Shift-Invariant System

 Shift-Invariant System

A system that produces the same output whether the signal appears at t = t0 or any other time.


Definition of Short-Term Memory

 Short-Term Memory

Past information available as variables (signals) within the topology.


Definition of Sigmoid

 Sigmoid

Any smooth nonlinear function that is monotonically increasing and has an S shape.


Definition of Signal Subspace

 Signal Subspace

A space of lower dimensionality in the reconstruction space that contains the signal vectors.


Definition of Signal-to-Noise Ratio

 Signal-to-Noise Ratio

Signal to noise ratio or SNR is the ratio of the signal power versus the noise power. It is typically measured in decibels (dB). A good signal has a high signal to noise ratio.


Definition of Simulated Annealing

 Simulated Annealing

A global search criterion by which the space is searched with a random rule. In the beginning the variance of the random jumps is very large. Every so often the variance is decreased, and a more local search is undertaken. It has been shown that if the decrease of the variance is set appropriately, the global optimum can be found with probability one. The method is called simulated annealing because it is similar to the annealing process of creating crystals from a hot liquid). See van Laarhoven and Aarts, Simmulated Annealing: theory and applications, Kluwer, 1988.


Definition of Sinc

 Sinc

A time signal given by sin(at)/at. It is the noncausal response of the ideal reconstruction filter.


Definition of Singular Value Decomposisiton (SVD)

 Singular Value Decomposition (SVD)

An analytical procedure that computes the orthogonal decomposition of data. See Matrix Computations by Golub and Van Loan, Johns Hopkins University Press, Baltimore, 1989.


Definition of Spectral Analysis

 Spectral Analysis

The analysis of the properties of signals using frequency domain information.


Definition of Spectrum or Fourier Spectrum

 Spectrum or Fourier Spectrum

Tthe ordered set of coefficients of the decomposition of the signal in phasors.


Definition of Spline

 Spline

The kth normalized B spline of degree r-1 is given by the Cox-deBoor formula

with


Definition of Square Integrable Function

 Square Integrable Function

Any function for which the integral of the square of the function over the domain is finite (here the real numbers).


Definition of Standard Deviation

 Standard Deviation

The square root of the variance. The variance is the second moment of the data with respect to the mean.


Definition of State Variables

 State Variable

Represents an internal quantity in the system that changes during operation. It may be either the output of the PE or the weight. We use it to represent the output of the PE.


Definition of Static Mappers

 Static Mappers

Systems whose output is dependent on a single time instant (instantaneous response).


Definition of Stationary

 Stationary

A signal is said to be stationary if its statistical properties do not change over time.


Definition of Statistical Independence

 Statistical Independence

Statistical independence between two variables x1 and x2 occurs when the joint pdf (f(x1, x2)) is equal to the dot product of the marginal pdfs (f(x1) · f(x2)).


Definition of Statistical Learning Theory

 Statistical Learning Theory

A new branch of statistics that analyzes the learning process mathematically (in functional spaces).


Definition of Steepest Descent

 Steepest Descent

A search procedure that seeks the next operating point in the direction opposite the gradient.


Definition of Step Size (or Learning Rate)

 Step Size (or Learning Rate)

The constant that scales the gradient to correct the old weights.


Definition of Supervised Learning

 Supervised Learning

Learning or adaptation in which a desired response can be used by the system to guide the learning.


Definition of Support Vectors

 Support Vectors

The samples that are close to the optimal hyperplane.


Definition of Tessellation

 Tessellation

A division of the space into convex regions that totally fill the space.


Definition of Test Set

 Test Set

The ensemble of input-desired response data used to verify the performance of the trained system. This data is NOT used for training.


Definition of Time Constant of Adaptation

 Time Constant of Adaptation

The exponent of the exponentially fitted envelope of the weight's geometric progression.


Definition of Time Delay Neural Network (TDNN)

 Time Delay Neural Network (TDNN)

A dynamic neural network where the memory structure is a tap delay line, the same memory utilized in the linear combiner of Chapter 9.


Definition of Time Domain Approach

 Time Domain Approach

A set of analysis methods that use time domain information to analyze and quantify the response of linear systems. Examples are the impulse response and convolution.


Definition of Time Lagged Feedforward Networks (TLFNs)

 Time Lagged Feedforward Networks (TLFNs)

Dynamic neural networks of intermediate complexity between fully recurrent networks and static networks. They are build from feedforward combinations of nonlinear PEs and linear filters.


Definition of Time Lagged feedforward networks (TLFN)

 Time Lagged Feedforward Networks (TLFN)

A network that has short-term memory structures anywhere in the network (i.e. either input, hidden or output PEs) but still has a general feedforward topology.


Definition of Topological Mapping

 Topological Mapping

A mapping that preserves a metric. In this chapter it means that local distances (neighborhoods) are preserved.


Definition of Topology

 Topology

The way the PEs are connected together in the neural network.


Definition of Training Set

 Training Set

The ensemble of input-desired response pairs used to train the system.


Definition of Training The Classifier

 Training The Classifier

Training involves defining the parameters of the discriminant function from the input data (the training set).


Definition of Trajectory

 Trajectory

The locus of points created by the time series in the reconstruction space.


Definition of Trajectory Learning

 Trajectory Learning

The most general case of training a dynamic system, which specifies the desired response during a time segment.


Definition of Transfer Function

 Transfer Function

The ratio of the transforms of the output and the input signals when the initial conditions are zero.


Definition of Uncorrelated Variables

 Uncorrelated Variables

Variables whose correlation is zero. Practically, this means that changing one variable has no effect on the value of the other.


Definition of Unfolding

 Unfolding

A method to create from a recurrent network an equivalent feedforward network. This is possible only for a finite time span.


Definition of Unsupervised

 Unsupervised

A learning rule in which the adaptation of the weights uses only one source of external information (the input).


Definition of Unsupervised Learning

 Unsupervised Learning

Learning in which the system parameters are adapted using only the information of the input and are constrained by prespecified internal rules.


Definition of Validation

 Validation Set

The ensemble of samples that will be used to validate the parameters used in the training (not to be confused with the test set, which assesses the performance of the classifier).


Definition of VC Dimension

 VC Dimension

Vapnik-Chervonenkis dimension, a measure of the capacity of a learning machine . See Vapnik, The nature of Statistical Learning Theory, Springer, New York, 1995.


Definition of Volterra Expansions

 Volterra Expansions

Volterra expansions of a discrete system (y the output, u the input) have the form


Definition of Weight Track

 Weight Track

A weight space plot of the weight locations during adaptation.


Definition of White Noise

 White Noise

A stochastic signal that contains all frequencies in equal amounts.


Definition of Z Notation

 Z Notation

The delay operator in the complex domain. The Z domain is the transform domain notation for discrete signals. z is effectively an independent (complex) variable given by [image] where s= [image] +jw, T is the sampling period, and j is the square root of -1.


Definition of Z Scores

 Z Scores

Variables that have been normalized to zero mean and unit standard deviation.


Definition of Z transform

 Z Transform

A complex function X(z) given by


Definition of Zeros

 Zeros

The values of z that make H(z)=0.


Defintion of Discrete Fourier Series (DFS)

 Discrete Fourier Series (DFS)

A decomposition of a time signal in terms of phasors (frequencies).


 Derivation of Energy Function

For the discrete-time Hopfield network considered here, it was shown that the function

is a Lyapunov function for the network, provided that the weight matrix is symmetric and the nonlinearity is sufficiently steep. Notice that if the biases are zero, the second term disappears, and if the nonlinearity approaches the step function, the last term approaches a constant.

The existence of a Lyapunov function for a given dynamic system guarantees stability (in the sense of Lyapunov). A Lyapunov function is a scalar function of the state with the following properties:

  • It is continuous and has a continuous first partial derivative over the domain.

  • It is strictly positive except at the equilibrium point.

  • It is zero at the equilibrium point.

  • It approaches infinity at infinite.

  • It has a first difference that is strictly negative in the domain except at the equilibrium point.

A quadratic function of the state is a Lyapunov function for linear systems. For nonlinear systems the form of the Lyapunov function is unknown and is normally difficult to find. Hence the importance of Hopfield's work.

H(y) trivially obeys all the conditions except for the last one. The proof shows that when the Hopfield network evolves according to its dynamics, H either stays the same or decreases, proving the strictly negative condition. See Hopfield for the complete proof.

Return to text


 Derivation of LMS with the Chain Rule

Let us rederive the LMS algorithm for a linear PE using the chain rule. We want to compute the partial derivative of the cost J with respect to w and equate it to zero to find the minimum. Now, J is a function of the weights through the network output yP, that is,

where Jp is the cost for the Pth sample and

so using the chain rule Eq. 3.12 we get

Using the gradient-descent idea to update the weights as we did before,

which is exactly the same result as obtained in the LMS algorithm.

Interpreting this equation with respect to the sensitivity concept, we see that the gradient measures the sensitivity. LMS is therefore updating the weights proportionally to how much they affect the performance, i.e. proportionally to their sensitivity. This makes perfect sense, since if the goal is to decrease the error, the weights that impact the error the most should get the largest modifications.

Return to text


 Derivation of Normal Equations

When the derivative of J with respect to the unknown quantities (the weights) is taken, we end up with a set of D+1 equations in D+1 unknowns:

(1B.15)

The solution is the famous normal matrix equation

(1B.16)

or expanding

(1B.17)

Notice that these equations are linear in the unknowns (the wj), so they can be easily solved.

Return to text


Derivation of Oja's Rule

 Oja's Rule and the Raleigh Quotient

Let X= [x1, x2,….xN] represent a set of data with N samples, where [image]. Without loss of generality we assume that the data is zero mean. To find the principal component we must find a vector [image] that maximizes the Raleigh quotient

where R = XX [image] is the data scattering matrix (N times the autocorrelation function). Analyzing the expression for J, we can conclude that the norm of w is irrelevant for the solution, so we can keep it constant at ||w|| = 1. If we want to use the gradient-descent procedure to maximize J, we have to compute the gradient with respect to the weights, which gives

If we keep the norm equal to 1, this expression defaults to the Oja's rule. Maximizing the output variance subject to the norm constraint produces Oja's rule.

Return to text


 Derivation of Quadratic Discriminant

We saw that Bayes rule chooses classes based on a posteriori probabilities. We can consider P(ci|x) as a discriminant

where p(x|ci) is the likelihood associated with the class ci. In this expression we omitted P(x) (see Eq.2.2) because it is a common factor on all discriminants and will not affect the overall shape nor placement of the boundary. It can therefore be dropped for the definition of the discriminant function.

Now let us take the natural logarithm of this equation and obtain

This is the general form for the discriminant, which depends on the functional form of the likelihood. If the density p(x|ci) is a multivariate normal distribution we get Eq. 2.8.

Estimating all the elements of the [image] matrix in high-dimensional spaces with adequate precision becomes a nontrivial problem. Very often the matrix becomes ill-conditioned due to lack of data, resulting in discriminants that have the wrong shape suboptimal performance.

Return to text


 Derivation of Sensitivity Through Nonlinearity

Let us write

and note that from Eq. 3.1

and

so finally we have the result in the text, Eq. 3.13

Return to text


 Derivation of the Conditional Average

This result can be demonstrated (see Bishop for a full treatment) if we write the MSE for the case with a large number of patterns as an integral:

For clarity we denote the desired response by t. Note that the index k sums over the targets, and the sum over the data exemplars was transformed into the integral, which has to be written as a function of the joint probability of the desired response and the input. This joint probability can be factored in the product of the input pdf p(x) and the conditional of the target data given the input p(tk|x).

The square can be written

where <<tk|x>> is the conditional average given by

We can write further

Now if we substitute back into the MSE equation and simplify, we obtain

The second term of this expression is independent of the network, so will not change during training. The minimum of the first term is obtained when the weights produce

since the integrand is always positive. This is the result presented in the text.

Return to text


 Derivation of the Largest Step Size

The best way to find the upper bound for [image] is to write the equation that produces the weight values. Let us rewrite the ideal performance surface (Eq.1B.10)as

(1B.11)

where

(1B.12)

By computing the gradient of J Eq.1 B.11 , we get

(1B.13)

so the iteration that produces the weight updates Eq.1.11 can be written as

(1B.14)

This is a first-order linear constant-coefficient difference equation, which can be solved by induction. First, let us subtract w* from both sides to yield

Start with a solution w(0).

which provides by induction the equation

Return to text


Derivation of the Optimalsolution

 Derivation of the Optimal Solution

Let us just take the derivative of J (Eq. 1.30) with respect to the weights, using matrix operations.

since the transpose of R is equal to itself because of its Toeplitz strucutre). If we equate this to zero, we obtain the optimal weights,

which is the equation in the text.

Return to text


 Derivation of the Time Constant of Adaptation

Writing exp(-1/ [image])= [image] and expanding the exponential in Taylor series,

we get approximately [image] ~1-1/ [image]. We saw that geometric ratio of the gradient descent is Eq.1.17 so we get

Return to text


 Development of the Phone System

A good example of engineering design is the telephone system. Long and meticulous research was conducted at Bell Laboratories on human perception of speech. This created the specification for the required bandwidth and noise level for speech intelligibility. Engineers perfected the microphone that would translate the pressure waves into electrical waves to meet the specification. Then these electrical waves were transmitted through copper wires over long distances to a similar device, still preserving the required specification. For increased functionality the freedom of reaching any other telephone was added to the system, so switching of calls had to be implemented. This created the phone system. Initially, the switching among lines was done by operators. Then a machine was invented that would automatically switch the calls. Operators were still used for special services such as directory assistance, but now that the fundamental engineering aspects are stable, we are asking machines to automatically recognize speech and directly assist callers.

The development of the phone system is an excellent example of engineering design. Once we have a vision, we try to understand the principles at work and create specifications and a system architecture. The fundamental principles at work are found by applying the scientific method. The phenomenon under analysis is first studied with physics or mathematics. The importance of models is that they translate general principles, and through deduction we can apply them to particular cases like the ones we are interested in. These disciplines create approximate models of the external world using the principle of divide and conquer. First the problem is divided in manageable pieces, each is studied independently of the others, and protocols among the pieces are drawn such that the system can work as a whole, meeting the specifications drawn a priori. This is what engineering design is today.

Return to text


 Dimension of Reconstruction Space

Signals with the same period may have different subspace dimensions. The more complicated the signal is in time, the larger is the dimensionality of the corresponding subspace. Random noise has no time structure and fills the space of any dimension. Static pattern data is similar to random noise. We need a space of maximum dimensionality (i.e., equal to the number of the components of the data vector) to represent it since we do not know a priori any relationships among the data components.

The dimension of the reconstruction space has been object of intense studies. Takens proved that for time series produced by dynamical systems, the size of the reconstruction space should be in the worse case equal to 2D + 1, where D is the size of the attractor of the dynamics. The arguments are rooted in topological theory that an object of dimension D may need a space of size 2D + 1 to be reconstructed without intersection of surfaces. See Kaplan

Return to text


 Distributed TLFN Architectures

In neurocomputing it is known that a general linear delay mechanism can be represented by temporal convolutions instead of multiplicative instantaneous interactions such as those found in MLPs. This model has been called the convolution model [Cohen and Grossberg], and in discrete time the activation of the ith PE reads

(11B.7)

where Ii(n) is a network input (if the PE is not an input Ii(n) = 0), yi(n) is the activation at the ith PE at time n, N is the total number of PEs, and K is the number of states (taps) of the delay subsystem that implements the short-term memory mechanism. The index k is associated with the states of the memory filter. The activation of the kth tap of the jth PE is written yjk(n). The weight wijk connects the kth tap of the jth PE to the ith PE. Since the topology is required to be feedforward, the condition i > j must be imposed.

The activation of each PE depends explicitly on the past value of either the input or other states, so that the time history can be captured more easily during training. We can think of the convolution model as a combination of nonlinear function f(.) with a short term linear memory mechanism represented by net (.), where

(11B.8)

As written, the memory mechanism can be globally recurrent. The memories in TLFNs are a special case of Eq. 11B.8, when neti(n) is a locally recurrent linear filter, created by cascading delay operators. We can recognize this arrangement as a generalized feedforward filter. The memory filter is no longer restricted to be finite impulse response (FIR) as in the TDNN; it can also be an infinite impulse response (IIR) adaptive filter with local feedback. The processing function of TLFNs can be described as a combination of nonlinear PEs and time-space mappings built from local linear projections. The delay operators have been addressed in Chapter 10.

Let us write the TLFN equations in matrix form. Define the signal vectors

and parameter matrices p = diagN(pi) and

where p is the matrix with the parameters of the generalized feedforward memory (GFM). Here k is the index of the state variables associated with the GFM. We can eliminate k if we define the GFM state vector

the nonlinearity and the matrix of decay parameters

and the matrix of weights

Then any TLFN can be written as

which is a N(D+1)-dimensional convolution model as we see later. If we analyze the structure of the matrix W we see that the matrix is not fully populated, meaning that the neural topology is prewired when compared to the most general convolution model.

Return to text


 Dominant Pole Approximation

The dominant pole approximation is very handy when we want to find approximately the center frequency and the bandwidth of a filter with many poles. First, we should rank the poles by their absolute value. We should focus on the pole pair (complex poles appear in conjugate pairs) that has the largest absolute value and discard the other poles. Let us assume that the largest pole is z0 = a + j [image] . The magnitude of the pole is |z0|.

The gain of the filter is approximately

The normalized -3 dB frequency of the maximum is approximately

The normalized bandwidth is approximately

These formulas come from geometric considerations in the Z plane. Remember, they are approximations that depend on how appropriate the approximation of the dominant pole really is.

A similar approximation can be done for zeros, except that the peak is substituted by its inverse, which provides the magnitude of the trough. The effects of the dominant pole can be added to the effect of the dominant zero.

Return to text


 Dynamic Backpropagation

In the control literature we see that instead of BPTT, the preferred method to adapt a mixture of neural networks and dynamic systems is dynamic backpropagation. Dynamic backpropagation is a blend of RTRL with static backpropagation. The parameters of the ANN are adapted with backpropagation, but the sensitivities are passed forward among submodules using the old idea of RTRL.

The advantage of this technique is that it provides adaptation sample by sample, being intrinsically on-line. The disadvantages are that it is computationally demanding and it depends on the particular arrangement of subsystems. As we will see later, there is a proliferation of blocks, basically one per parameter, needed to pass the sensitivity forward. During BPTT the gradients are all local to the topology, so it can be applied irrespective of the topological configuration of the system. However, BPTT is not local in time, which means that the adaptation must be done for a trajectory.

Let us present the ideas of dynamic backpropagation as applied to representation 1 of Figure 11-11. For this representation we need to find the parameter set of the ANN, here represented by wi, but since the output of the net v(n) is not directly accessible, we have to include the system P(v, [image]) in the computation of the error. Assume that the error at the output is [image]. Since y is the output of the cascade, and P(v, [image]) is independent of w,

We already know how to compute [image] for every time tick using backpropagation, but every time we change one of the parameters, there is a corresponding change in the output error (for the present time and for future time, since P(v, [image]) is dynamic). This change can be computed if the sensitivity of the ANN is passed forward through a copy of P(v, [image]) as in the following figure

Notice that in general we need as many copies of P(v, [image]) as outputs in the ANN (one system generates a partial derivative [image]). This is where the computational complexity gets out of hand quickly. But notice that we can update the parameters of the ANN for EACH time step, which is an advantage.

If we want to work with any of the other representations, this simple sensitivity model has to be appropriately modified, but it is possible in all cases to update the weights at each step.

Return to text


 Early Stopping and Model Complexity

We saw before in Section 4.4.2 that early stopping provides a criterion to stop training at the point of smallest error in the validation set, that is, the point of best generalization for that particular combination of topology and training set. From the point of view of model complexity, early stopping effectively controls the complexity of the model, which may seem strange since the number of free parameters is constant. It turns out that in nonlinear systems the model complexity depends not only on the number of parameters (as in linear systems) but also on the actual value of the free parameters, so it may change during training.

Early stopping does not address the size of the learning machine, which is also a determining factor for controling the model complexity. In linear machines, model size is the only way to control model complexity, so parsimonious architectures should be a design goal. The full discussion of this topic is rather theoretical and is left to Chapter 5. Here we will use a heuristic approach.

Return to text


 Eigendecomposition

In engineering we use systems to modify time signals according to user specifications. A system is a piece of equipment that modifies input signals x(t) to produce another signal y(t) (see the figure).

Mathematically we can describe a system by a function H operating on the real (or complex) numbers

and we will call H an operator. The output y of the system H when x is applied at the input is

The response of the system at time t is written y(t) = [Hx](t).

A linear system is a system described by an operator H obeying the following properties

where [image] is a scalar. We are normally interested in linear systems that are shift invariant, that is, where the response does not depend on the particular time of application. Let us define another operator T that delays x by [image] seconds:

In shift-invariant systems, H and T commute, that is, THx = HTx.

Let's ask the following question: Which are the signals x(t) that, when applied to a linear time-invariant system H, produce a response that has the same form as the input, apart from a multiplicative factor (gain factor)?

Mathematically this can be written

This is the same problem that gives rise to the eigenvector problem of matrices, and this is why the input x that obeys this condition is called an eigenfunction of H.

Linear shift-invariant systems have the special property that they commute with the derivative operator D, that is,

Or, in words, if we know the response of the system to an input and we want to know the response to the derivative of the input, then it is enough to take the derivative of the output.

This is what we need to answer the original question. The derivative property shows that the question is equivalent to finding a signal x(t) that is proportional to its derivative:

which we know accepts the solution [image], that is, a complex exponential.

This means that a linear shift-invariant operator H, when applied to a complex exponential [image], will only change its magnitude and phase:

(5B. 1)

where [image] is a complex number. Thus, if an arbitrary function u(x) is decomposed into exponentials,

(5B.2)

then the response of H to u(x) can always be evaluated as a sum of weighted responses to exponentials,

(5B.3)

where the [image] i do not depend on u(x). The importance of this equation has to be noted, since it tells us that no matter how complicated the input might be, we can always compute its output by adding the responses to individual exponential components. It also tells us that all we need to know to describe the linear system are the complex numbers [image] i.

Fourier analysis is a special case of this decomposition in which the complex exponentials have zero real parts, i.e. s = jw , yielding

Now we understand why complex exponentials are so important when studying linear systems.

Return to text


 Eigendecomposition Explained

A linear shift invariant operator H, when applied to a complex exponential [image] changes only its magnitude and phase:

(8B.3)

where [image] is a complex number. Thus, if an arbitrary time function u(t) is decomposed into exponentials

(8B.4)

then the response of H to u(t) can always be evaluated as a sum of weighted responses to exponentials, i.e.

(8B.5)

where the [image] i do not depend on u(t). Fourier analysis is a special case of this decomposition where the complex exponentials have zero real part. For continuous time operators this decomposition gives rise to the Laplace domain analysis of linear systems, where the system is represented by the operator H. In a nutshell, the Laplace analysis is the study of the linear system in the complex plane s = [image] [image] + jw.

In the case of discrete time operators, it is convenient to change the variable for the analysis from s to [image], called the Z transform. The Laplace and Z transforms achieve the same goal, but one is applicable to continuous systems while the other is for discrete systems. Notice that we already saw the z notation when we defined the ideal delay operator.

Let us reiterate the advantage of eigendecompositions. They are very efficient since they provide an orthonormal basis, and they naturally decompose the signals, that is, they achieve perfect reconstruction with just a few bases.

Return to text


 Eigenfilter

An eigenfilter is associated with the eigendecomposition we studied in Chapter 5. Recall that there we were looking for natural bases to decompose functions. Here we look at ways of naturally decomposing data clusters. The eigenfunctions are the bases from which the functions are exactly constructed by a finite weighted sum (the projection theorem), so they are the most efficient way to decompose any function.

Oja's rule when applied to the linear PE network implements a decomposition that finds the weights corresponding to the principal component direction. This direction maximizes the projection of the input data cluster. To find this direction, the input data has to be projected by a "filter" matched to the data, hence the name maximum eigenfilter. We should think of the weights of the network as the bases (as we did in Chapter 5) and the network output as the scalar in the projection theorem.

There is a very important concept hidden here. When we directly use data collected from sensors (measurements), the representation space is given by our measurements. This space may not best capture the relevant properties of the data. The principal directions tell us the bases that best represent the data.

Return to text


 Energy, Power and Variance

From the statistical point of view, the energy of a 1-D signal x(n) is related to its variance. In fact, the energy of a stationary signal x(n) with variance [image] ² and mean m is

where E[.] is the expectation operator. If m = 0 then the energy is equal to the variance, [image], so the energy is related to the second-order statistics of the signal.

The power V (or short-term energy) is defined as the energy in a finite window or for a finite number of samples. The power is also related to an estimation of the second-order statistics with finite data. The condition of zero mean is normally assumed in the discussion.

Return to text


 Eq.1B.6


 Eq.1B.10


 Eq.1B.11


 Eq.1B.18


 Eq.1B.19


 Eq.3B.1


 Eq.4B.1


 Eq.11B.1


 Eq.11B.3


 Eq.11B.7


 Eq.1.3


 Eq.1.4


 Eq.1.5


 Eq.1.6


 Eq.1.7


 Eq.1.8


 Eq.1.9


 Eq.2.1

x belongs to [image] if [image] for all j [image] i


 Eq.2.2


 Eq.2.3


 Eq.2.4


 Eq.2.6

x k belongs to ci if gi(xk)>gj(xk) for all j [image] i


 Eq.2.8


 Eq.2.9


 Eq.3.1


 Eq.3.2


 Eq.3.3


 Eq.3.4


 Eq.3.9


 Eq.4.3


 Eq.4.8


 Eq.5.1


 Eq.5.2


 Eq.5.4


 Eq.5.7


 Eq.5.8


 Eq.6.1


 Eq.6.2


 Eq.6.4


 Eq.6.5


 Eq.6.6


 Eq.6.7


 Eq.7.2


 Eq.7.4


 Eq.8.9


 Eq.9.1


 Eq.9.5


 Eq.9.6


 Eq.10.7


 Eq.10.9


 Eq.1.10


 Eq.1.12


 Eq.11.5


 Eq.1.17


 Eq.11.8


 Eq.1.25


 Eq.1.26


 Eq.1.27


 Eq.1.28


 Eq.1.30


 Eq.1.31


 Eq.1.32


 Eq.1.33


 Eq.1.36


 Eq.1.43


 Eq.1.44


 Eq.3.11


 Eq.3.12


 Eq.3.13


 Eq.3.15


 Eq.3.16


 Eq.3.19


 Eq.3.21


 Eq.3.22


 Eq.3.27


 Eq.3.28


 Eq.3.29


 Eq.3.31


 Eq.3.33


 Eq.3.39


 Eq.3.42


 Eq.4.11


 Eq.4.12


 Eq.4.14


 Eq.4.19


 Eq.4.21


 Eq.5.13


 Eq.5.27


 Eq.5.32


 Eq.5.35


 Eq.6.11


 Eq.6.13


 Eq.6.18


 Eq.6.19


 Eq.6.24


 Eq.6.26


 Eq.6.27


 Eq.6.28


 Eq.6.34


 Eq.7.19


 Eq.8.12


 Eq.8.17


 Eq.8.19


 Eq.8.21


 Eq.8.25


 Eq.8.26


 Eq.8.27


 Eq.8.29


 Eq.8.33


 Eq.9.10


 Eq.9.17


 Eq.9.29


 Eq.11.12


 Eq.11.13


 Eq.11.14


 Eq.11.16


 Eq.11.17


 Eq.11.27


 Eq.11.28


 Eq.11.32


 Eq.11.34


 Eq.11.35


 Eq.11.36


 Eq.11.40


 Eq.10.3


 Eq.1.11


 Eq.11.1


 Eq.1.13


 Eq.11.3


 Eq.1.14


 Eq.11.4


 Eq.1.16


 Eq.11.6


 Eq.1.19


 Eq.11.9


 Eq.3.34


 Eq.3.35


 Eq.3.40


 Eq.3.41


 Eq.4.20


 Eq.8.13


Equation for LMS

 LMS

w(n+1) = w(n) + [image] x(n) [image] (n)


 Error Attenuation

Let us look at the figure to understand how the error is attenuated. The top part of the figure shows the tanh nonlinearity, and the bottom part its derivative. When the input variable net is in the linear region (region A), the derivative is close to 1, so the sensitivity Eq. 3.13 is close to xi, the equivalent sensitivity of the linear PE. However, when the operating point of the nonlinearity is close to saturation (case B or C), the derivative of the tanh is close to zero. This effectively corresponds to an attenuation of the sensitivity when compared with the linear PE. What this means is that y became much less sensitive to a change in xi when the operating point is in region B or C.

We thus see that nonlinear PEs have a dual role of saturating the activations for large values of the input and attenuating the corresponding sensitivities. These two factors bring stability to the learning process, because learning becomes much less dependent on outliers. Moreover, the nonlinearity allows each PE in the network to specialize in a portion of the input space (the weights associated with a given PE learn more when that PE is in the linear region).

Return to text


 Estimation of Eigenvalue Spread

The eigenvalue spread can be computed by an eigendecomposition of R, but this is a time-consuming operation and is hardly ever performed. An estimate of the eigenvalue spread for the multidimensional-data case is the ratio between the maximum and the minimum of the magnitude of the Fourier transform of the input data.

Alternatively, simple inspection of the correlation matrix of the input can provide an estimation of the time to find a solution. The best possible case is when R is diagonal, with equal values in the diagonal, because in this case the eigenvalue spread is 1 and the gradient descent travels in a straight line to the minimum. We cannot have a faster convergence than this, even when second-order methods (such as Newton's method, studied later) are used. When R is diagonal but with different values, the ratio of the largest number over the smallest is a good approximation to the eigenvalue spread. When R is fully populated, the analysis becomes much more difficult.

Return to text


 Filter Shape Factors

The frequency response of a realizable lowpass filter is shown in the following Figure. We can define the passband as the set of frequencies that pass through the filter without being appreciably attenuated. There is always a certain "ripple", deviation from perfect unity gain, primarily for frequencies close to the edges of the passband. This ripple is a design factor that can be controlled.

The edges of the passband are called the cutoff frequencies and are also a design specification. The edge is smooth, so there is always a problem of defining it. Normally the [image] 3dB attenuation from the passband is considered the cutoff frequency .

The gain goes from practically 1 in the passband to very low values in what is called the stopband of the filter. The filter attenuates the input frequencies that fall in the stopband. It is this frequency selectivity that makes filters so useful. The edges of the stopband as well as the stopband attenuation (in dBs) are normally specified.

The edges of the passband and the stopband do not coincide in a practical filter (i.e., a filter with finite orders). The difference in frequency from the passband edge to the stopband edge is called the transition band. Good filters have low ripple in the passband, short transition bands, and high attenuation in the stopband (called the filter-shape factor). Ideal filters have zero ripple, zero transition band and infinite attenuation in the stopband.

Return to text


 Filtering Operation

Filtering, as we have said, is the selective removal of frequency components. In many practical situations, signals are contaminated with noise, so it is often critical to remove this unwanted noise. For instance, the electric power we use in the home or office is modulated at 60 Hz for efficiency (to reduce the power loss in the wiring). As you might expect 60 Hz components exist everywhere there is electricity (in Europe the electric power is 50 Hz). This can be very annoying when we try to measure electric signals of low amplitude. For instance, the human body functions as a great antenna for 60-Hz signals, so when we want to measure faint electrical signals created by our heart or brain, we have to deal with the 60-Hz interference. One way to handle this problem is to prefilter the physiologic signals with a linear system that removes the 60 Hz power line interference and lets the rest of the signal pass undistorted.

The issue, as we will see later in this chapter, is how to design such filters. Here we would like to point out that our discussion naturally was based on ideas from the frequency content of the signals. The time domain description was never mentioned in the discussion. This is normally the case, and it is the other motivating factor for defining signals in the frequency domain. More specifically, we can say that the steady-state behavior of linear systems calls for frequency domain representations. For transient response analysis, the advantage of the frequency domain is not as clear.

Return to text


 Fourier Formulas

Applying again the pair of formulas of Eq 5.7 and Eq. 5.8 we present the Fourier transform pair. Remember that the Fourier transform uses as bases the complex exponentials, that is,

where T is related to the interval where the function f(t) is defined and [image]. The complex exponentials are a set of orthonormal bases. This means that we are going to expand the function f(t) as

In the interval D = [0, T] we can compute the weights as (Eq.5.7)

This means that we have formulas to compute the weights, so we do not need to use adaptation. Note that the complex exponential can be expressed as (Euler relation)

so in fact we are decomposing the signals in sums of pairs of sinusoids.

Return to text


 Gamma Filters as an Extension of Wiener Filters

The gamma filter can be considered as an extension to the famous Wiener filter. Let us define the cost as J = E{e²(n)}, where E{.} is the expectation operator. Let us call the D+1 tap vector

and the weight vector

So

where R and p are, respectively, the autocorrelation function of the tap signals and the cross-correlation vector between the tap signals and the desired response. Note that the input in the linear combiner is effectively replaced by the tap signals. The goal of adaptation is to find the minimum of J in the space of the D weights and [image]. Taking the partial derivatives with respect to the weights and [image] [image] we obtain the optimal filter weights as

(10B.1)

where

and

Note that the first equation in Eq. 10B.1 is the same as the Wiener-Hopf equation for the linear combiner. The difference once again is that the autocorrelation is not of the input but of the tap signals. The extra scalar condition is a result of the extra parameter in the gamma filter, and as we discussed, it represents the optimal memory depth for the application. Although the gamma kernels have infinite extent, they can be computed exactly in the frequency domain (provided that x(n) has a computable Fourier transform). Eq. 10B.1 has a form that applies to all the generalized feedforward filters described by Eq. 10. 7.

Return to text


 Good Initial Weight Values

Very little research has been conducted in finding good initial weight values for training, since the problem is very difficult to formulate. Intuitively we can see that if the initial weights can be chosen close to the optimum, then searching for the minimum will be fast and reliable. Unfortunately, we do not know where the optimal weights are in pattern space (this is why we are adapting the system). Some authors have proposed the linear solution for the initial weight values, but this does not always work. In many cases the linear solution is exactly the point that the search should avoid because it is a strong local minimum.

Return to text


 Gradient Definition and Construction

The gradient is formally defined in terms of partial derivatives of a function f(x, y). Let us consider a function f(x,y) that has partial derivatives at x0,y0. The gradient of f at x0,y0 is defined by

where ux, uy are the unit vectors along x and y, and fx, fy are the partial derivatives of f along the x and y directions, respectively, which are given by

The gradient is associated with the concept of a directional derivative of a function. Let us assume we have a direction u = aux + buy. The directional derivative of f at x0,y0 along u is

The gradient can be thus defined as a function of the ordered derivatives:

where the operation is the dot product of two vectors (for v = c ux + d uy, v [image] u = ac + bd).

This expression means that the maximum value of the directional derivative as a function of the direction u is given by the size of the gradient, and it occurs exactly when the direction u coincides with the gradient direction.

Moreover, we can also find this direction pretty easily. Let us consider the curve C(x,y), defined as the line in the x, y plane where the function f has a constant value (this line is called the level curve or the contour of f). At a point x0, y0 in C the rate of change of f in the direction of the unit vector u tangent to C must be zero (see the preceding definition), that is,

But this implies that the gradient vector is perpendicular to the tangent vector u of the level curve at x0,y0. This explains the graphical construction outlined in the text.

Return to text


 Gram-Schmidt Orthgonalization

The question is very simple. Given a set of vectors {x1,….xm} spanning a space S (i.e. the space of all their linear combinations), can we find a rotation that will orthogonalize all the vectors and preserve the span?

The solution was proposed many years ago by Gram-Schmidt in the form of a recursive procedure. Let us start with one of the vectors and make

Then find a direction orthogonal to the subspace defined by the k vectors already orthogonalized and normalize, which yields

which due to the previously orthogonalized vectors yields

It is interesting to look at this equation in a figure for the case of two vectors.

This method is in fact a deflation procedure, because it removes the contributions of the previous vectors to the current direction. To see this, we first have to define a projector as a matrix such that Z² = Z. We immediately see that ZZ [image] is a projector, so when applied to any vector, it will project it to the span of Z. Now, the form I - ZZ [image] is also a projector, and it will project any vector to the orthogonal space of Z (also called the null space of Z). Gram-Schmidt uses an iterative projection to the orthogonal space of each vector, so it is a deflation procedure. (see Diamantaras).

Return to text


 Hebbian as Gradient Search

To show that Hebbian acts as a gradient search just differentiate Eq. 6.11 with respect to W to obtain

due to the Toeplitz properties of R. Gradient ascent changes the weights according to

which is exactly what we presented for the Hebbian rule (aside from the 2, which is included in the step size). Note that gradient ascent goes in the direction of the gradient, so there is no minus sign in the weight update, as in the LMS rule.

Is the Hebbian update ever stable? Let us write the Hebbian update as

Applying the expectation operator we get

where R is the autocorrelation function of the input and I is the identity matrix. The stability of this iterative equation is determined by the characteristic roots of the matrix [image] . Since R is positive definite, all the roots will be positive hence the iteration will diverge for any value of [image].

Return to text


 Hebbian as the Minimization of the Reconstruction Error

Let us interpret Hebbian learning in a supervised learning context. In fact, if we have a performance criterion exterior to the network, it is equivalent to thinking of an error, hence of a desired response. What is the implicit desired response in Hebbian learning? It is the input signal itself, and the minimization is to reconstruct x from [image], which is the projection of y into W, i.e. . A figure will clarify this procedure

This means that the criterion is the mean square difference between the input and the projected output, that is,

If we substitute the definition of y = Wx we get

since

Now we have a more refined definition of what Hebbian learning is accomplishing from a vector space point of view. In fact Hebbian can be either interpreted as maximizing the variance of y (the projection variance, Eq. 6.11), or minimizing the reconstruction error between the input and its version obtained after projecting the output y on the weight vector (transposed).

Return to text


 hot-links

are underlined in green throughout the text. Just click on the link to jump to the appropriate topic or to read the pop-up panel.


 How to Use the FFT

The fast Fourier transform is an algorithm that computes the DFT very efficiently. To use the FFT appropriately we have to choose its parameters wisely. What we are really interested in is estimating the spectrum from a finite data set, so we have to worry about the resolution and the variance of the estimator.

The resolution is controlled by the number of data samples used in the window, which is normally called the length of the FFT. This length has to be a multiple of 2 for the fast algorithm to work. Once the length is selected, the resolving power of the method (the frequency resolution) is set by [image] f = 1/NT so we have to select N appropriately.

Another issue is the variance of the estimator. Experience shows that the estimator has a large variance: that is, the shape of the FFT is highly dependent on the data in the window. To decrease the variance, we normally divide the data set in several windows, take individual FFTs, and average the FFTs together. The variance decreases with the number of individual FFTs being averaged, so the larger the better.

Unfortunately, this means that there are conflicting requirements when we have a finite number of samples. Either we use a long window to get better resolution but sacrifice on the variance (i.e., the variance is high); or we use smaller windows and average the individual FFTs, which reduces the variance but sacrifices resolution because the window size is reduced. We have to get the best compromise for the problem at hand. Sometimes we overlap the windows to get more individual estimates for the same resolution. NeuroSolutions allows the user to set all of these parameters.

Return to text


 Impulse Response

The impulse response is by definition the response of the system when the input is an impulse at n = 0. This means that Eq. 8.9 can be written by substituting h(n) = y(n) and x(n) = [image] (n), which yields

What this equation means is that the impulse response of a FIR is exactly the copy of its weights spread out in time. For instance, if the weights of the filter are w0 = 0.5, w1 = 0.25, w2 = 1, the impulse response is h(0) = 0.5, h(1) = 0.25, h(2) = 1. By knowing the weights in a FIR, we know the impulse response (or vice versa).

Return to text


 Information and Variance

Extracting information from data is what learning is all about. Here we are using a layperson's concept of information, but we can also provide a technical definition. Shannon in a seminal paper proposed the following definition for entropy

where pk are the probabilities of a set of messages {x1,…xN} occurring with probabilities p1, ….pN. Entropy measures the amount of information in messages. The idea is the following: if we know what the message is (pk = 1), the information carried is zero. On the other hand, if its content is unexpected (small pk), then the amount of information the message carries is rather large. This definition supports well our intuition, although Shannon utilized an axiomatic approach to derive the concept of information. Shannon's entropy definition has been the cornerstone for creating efficient and reliable communication systems (see Cover), and it is also quite important in statistics and learning.

We may note that entropy uses all the information about the probability density function (pdf) of the data, but normally we do not know the pdf. It turns out that if the data is Gaussian distributed, that is,

then only two numbers, the mean m and the variance [image] are sufficient to describe the pdf of the data. This means that only the first- and second-order moments are nonzero. Therefore, for Gaussian distributed data, the entropy can be written

and we conclude that it is proportional to the variance of the data. For Gaussian distributed variables, information is a synonym of variance.

Return to text


 Inventors of Backpropagation

Many researchers worked in the computation of sensitivity across a network. Werbos, with the ordered derivative, was probably the first. In the connectionist arena, the key paper that popularized backpropagation was written by Rumelhart, Hinton and Williams. Le Cun in France presented a method based on Lagrange multipliers at basically the same time.


 Know More Boxes

This is a very short example of a "know more box". These links open a secondary window so the information can be browsed without interfering with the flow of the main text.

These boxes always include a "return to text" link which returns to the original position in the main text, although you can also simply click into the main text window.

Return to Text


 L1 Versus L2

There are minor differences between the two norms that have been used in function approximation. Eq. 5.2 uses the absolute value of the error, which is commonly called uniform approximation. Strictly speaking the L1 norm should be used for function approximation. However, the L2 norm, which minimizes error power not the absolute value of the error, is much easier to apply (in linear systems) and also produces an important interpretation of the results (nonlinear regression), as we saw in the probabilistic interpretation of the mappings. In practical situations either norm can be used.

Return to text


Least Square Derivation

 Least Squares Derivation

In Eq.1.4 we substitute the value of the error given by Eq. 1.3 and take the derivative expressed by Eq.1.5 to obtain

(1B.1)

Note that to simplify the notation we omit the limits of the variable in the sum. We do this throughout the text when no confusion arises. Operating we get further

(1B.2)

The set of Eq. 1B.2 is called the normal equations. The solution of this set of equations is

(1B.3)

which provides the coefficients for the regression line of d on x. The summations run over the input-output data pairs. Eq. 1B.2 is solved by computing the value of b from the first equation and substituting it in the second equation to obtain w as a function of x and d. Then the value of w is substituted in the first equation to finally obtain b as a function of x and d (variable elimination). It is easy to prove that the regression line passes through the point

which is called the centroid of the observations. The denominator of the slope parameter of w and b is the corrected (for the mean) sum of squares of the input.

Return to text


Levenberg-Marquadt Quasi-Newton Method

 Levenberg-Marquardt Quasi-Newton Method

The Levenberg-Marquardt algorithm uses the Gauss-Newton method to approximate the Hessian. Let us assume that the performance function is a sum of individual components

where N is the number of samples in the training set, and ei are the instantaneous errors. Then it is easy to show that the gradient is

(4B.4)

where [image] is the Jacobian matrix given by

The Hessian is easily obtained from Eq. 4.11 as

where

Assuming that S(w) is small, the Hessian can be approximated by

So Eq. 4.8 can be written

This weight update does not require second-order derivatives. The approximation that was introduced may result in difficulties in inverting H, but as we saw earlier for the conjugate gradient, we can add a small value ( [image]) to the diagonal of [image] to make sure that the matrix is full rank. This provides the Levenberg-Marquardt algorithm

(4B.5)

This is a very interesting formula: Notice that if [image] is increased so that the first term of the inverse is negligible, the weight update is basically gradient descent with a step size (1/ [image] [image] (n)). On the other hand, if [image] is zero, the information about the curvature is fully utilized in the search.

We mention this fact because we have to set [image] during the adaptation. The goal is to have [image] as small as possible while still guaranteeing inversion, but this depends on the data. We should therefore start with a small value of [image] ( [image] =0.01) and see whether the next weight vector produces a smaller error. If it does, continue. If the error for this new position is higher than before, we have to return to the old weight vector, increase [image] [image] and try again. At each trial [image] should be increased by a nominal amount (a factor of 5 to 10 is normally recommended). Notice that if we continue doing this, we will default to gradient descent, which is known to be convergent with small step sizes. When the error for the new evaluation of the weights produces a smaller error, then start decreasing [image] by the same factor.

This algorithm is particularly well suited for training neural networks with the MSE. Moreover, it interfaces well with the backpropagation formalism. Note, however, that here we need to keep all the errors separated since we need to use the Jacobian matrix. In conventional backpropagation the errors from all the outputs get added together at each PE, and in batch learning we even add all these errors across the training set. However, here each partial derivative of the error must remain accessible during training of a batch, which causes huge storage requirements. Nevertheless, the backpropagation algorithm can still propagate sensitivities from the output to each node to evaluate each entry in the Jacobian. As a rule, backpropagation must be applied repeatedly and independently to each output to avoid the addition of errors, that is, the injected error vector for a P output system becomes

Each input desired response pair creates P errors. Each of these backpropagation sweeps for a given sample creates a row of the Jacobian (the number of columns is given by the number of weights in the network). Each new sample will repeat the process. Thus we can see that the Jacobian matrix gets very large very quickly. The other difficulty is that the algorithm is no longer local to the weight. Nevertheless, the Levenberg-Marquardt algorithm has been shown to be much faster than backpropagation in a variety of applications. As a side note, the conjugate gradient of Eq. 4.10 also requires the use of the Jacobian matrix, so it uses the same basic procedure.

Return to text


 Line Search Methods

A principled approach to finding the optimal step size at each iteration is to minimize

J(wk = [image] ksk) with respect to [image], where sk represents the direction of the search at sample k. For quadratic surfaces there is an analytic solution given by

(4B.1)

where H is the Hessian and [image] J is the gradient vector. Since the Hessian for quadratic surfaces is independent of the point, it needs to be computed just once. We can prove that the algorithm zigzags to the minimum (in orthogonal directions) with this choice of the step size (see Figure 4-3). Note that at each iteration we need to perform vector computations, which make the adaptation computationally more complex.

In general, the optimal value of [image] must be solved by line search since the performance surfaces are not quadratic. This gives rise to the conjugate gradient algorithms.

Return to Text


 Local Minima for Gaussian Adaptation

After Chapter 3 and 4 we should be able to understand the difficulty of using backpropagation to adapt the centers and variances of the Gaussians. With this method the centers (and variances) are moved in the input space by virtue of the gradient. However, with the RBF, both the local activity and the local error are attenuated by the shape of the Gaussian kernel, while in the MLP only the error was attenuated by the derivative of the sigmoid. The net effect is that training becomes very slow, and the chances of getting stuck in local minima are large. Another problem is that during adaptation the variances of the Gaussians can become very broad and the RBF loses its local nature.

Return to text


 Long and Short Term Memory

Long-term memory refers to the storage of information from the past. Since the weights are adapted with the input information, their values are related to all the data that has been presented to the network. Hence they represent the long-term memory of the network.

It is also convenient to consider the activation in the PEs as short-term memory. The short-term memory we consider in this chapter is instantaneous since the activations of the PEs discussed so far only depend upon the current data sample. But later in Chapter 9 we consider other network topologies where the activations depend on past samples.

Return to text


 Mahalanobis Distance

The Mahalanobis distance is the exponent of the multivariate Gaussian distribution, which is given by

where T indicates the transpose, | [image] | is the determinant of [image], and [image] [image] the inverse of [image]. Note that in the equation [image] is a vector containing the data means in each dimension, i.e. the vector has dimension equal to D.

Normally we estimate [image] by the sample mean Eq. 2.4. The covariance is a matrix of dimension D x D where D is the dimension of the input space. The matrix [image] is

and its elements are the product of dispersions among sample pairs in the ith and jth coordinates (N is the number of samples in the data set)

The covariance measures the variance among pairs of dimensions. Notice the difference in number of elements between the column vector [image] (D components) and the matrix [image] [image] (D² components).

The Mahalanobis distance formalizes what we have said for the 1-D classification example. Notice that this distance is a normalized distance from the cluster center. In fact, if we assume that [image] = I (identity matrix), we have the Euclidean distance between the cluster centers, but for classification the dispersion of the samples around the cluster mean also affects the placement of thresholds for optimal classification. It is therefore reasonable to normalize the Euclidean distance by the sample dispersion around the mean, which is measured by the covariance matrix.

The covariance matrix for each class is formed by the sample variance along pairs of directions in the input space. The covariance matrix measures the density of samples of the data cluster in the radial direction from the cluster center in each dimension of the input space. So, it quantifies the shape of the data cluster.

The covariance matrix is always symmetric and positive semidefinite. We assume that it is positive definite, that is, the determinant is always greater than zero. The diagonal elements are the variance of the input data along each dimension. The off-diagonal terms are the covariance along pairs of dimensions. If the data in each dimension are statistically independent, the off-diagonal terms of [image] are all zero, and the matrix becomes a diagonal matrix.

The structure of the covariance matrix is critical for the placement and shape of the discriminant functions in pattern space. Since the distance metric for classification is normalized by the covariance, if the class means stay the same but the covariance changes, the placement and shape of the discriminant function will change.

Return to text


 Mapping Capabilites of the One-Hidden-Layer MLP

The one-hidden-layer MLP with sigmoid PEs is a universal mapper, that is, it can approximate arbitrarily well any continuous decision region, provided the number of hidden-layer PEs is large enough.

There have been proofs of this statement by Cybenko, Gallant and White, Hornik et al, many of them based on the Stone Weierstrass theorem. These proofs are difficult to follow and will be omitted here. They discuss only the mapping capabilities (existence proofs); they do not describe how to create an MLP with the specified characteristics (constructive proofs), so they are useful to describe the power of the technology.

An interesting thing is that the activation function does not seem to be that important for the mapping capabilities, since the proofs use several of them (even discontinuous nonlinearities). The key aspect is the form of the approximation function, which is an embedding of functions, that is, the result of one function becomes the argument for the next: y = f ( f (.))

Return to text


 Mars' Pathfinder Mission

When machines have to autonomously interact with the environment or operate near the optimal set point, we cannot specify all the functions a priori and in a deterministic way. Take, for instance, the Mars Pathfinder mission. It was totally impossible to specify all the possible conditions that the rover Sojourner would face, even if remotely controlled from Earth, so the problem could not be solved by a sequence of instructions determined a priori in JPL's laboratory. The vehicle was given high-level instructions (way points) and was equipped with cameras and laser sensors that would see the terrain. The information from the sensors was analyzed and catalogued in general classes. For each class a procedure was designed to accomplish the goal of moving from point A to point B. This is the type of engineering system of which we will build more and more in the future.

The big difference between the initial machines and Sojourner is that the environment is intrinsically in the loop of the machine function. This brings a very different set of problems, because, as we said earlier, the environment is complex and unpredictable. If our physical model does not capture the essentials of the environment, then errors accumulate over time, and the solution becomes impractical. We thus no longer have the luxury of dictating the rules of the game, as we did in the early machine-building era. It turns out that animals and humans do Sojourner-type tasks effortlessly.

Return to text


 MDL and Bayesian Theory

There are some technical details in implementing this idea of measuring models and errors by code lengths, but they have been worked out in Information theory. See Rissanen. Another interesting concept is to look at model selection from the point of view of Bayes' theory. The probability of a model M given the data D can be computed using Bayes' rule:

We can ignore P(D), since it is common to all the models, so the most probable model maximizes the numerator. We know that the maximization is not affected if we take the log (since the log is a monotonic function), that is,

This expression is very similar to the result obtained using Rissanen's MDL principle. In fact, we can interpret Rissanen's description length as the sum of the error and the complexity of the model. The minimum amount of information required to transmit a message x is given by [image] ln p(x). If p(x) is the correct distribution for the message x (our model), it will correspond to the smallest message length for a given error. The error in turn can be interpreted as the conditional probability of the data given the model. We can therefore say that the description length can be expressed as

Since maximization is equivalent to the minimization of the negative functional we get the minimal code lengths for the data and the model. See Zemel for a complete treatment.

Return to text


Memory Depth

 Memory Depth of the Context PE

The impulse response of the context PE is

Its Z transform, as we saw in Chapter 8 is

Now the definition of the memory depth in the time domain can be converted to the frequency domain as the derivative with respect to z of the system transfer function evaluated at z = 1,

Return to text


 Minimum Error Rate

The probability of error is computed by adding the area under the likelihood of class 1 in the decision region of class 2 with the area under the likelihood of class 2 in the decision region of class 1. Since the decision region is a function of the threshold chosen, the errors depend on the threshold. As we can see from the figure, the error is associated with the tails of the distributions. To estimate this error, we need to integrate the likelihoods in prescribed areas of the input space, which becomes very difficult in high-dimensional space. The probability of error is

where R1 and R2 are the regions assigned to class 1 and class 2, respectively, so it is a function of the threshold. One can show [Fukunaga] that the minimum error rate is achieved with the Bayes rule, that is, by selecting the threshold such that the a posteriori probability is maximized. This result also makes sense intuitively (see the following figure).

The classification error depends on the overlap of the classes. Intuitively, the larger the difference between the cluster centers (for a given variance), the smaller the overlap, so the smaller the overall classification error. Likewise, for the same difference between the cluster means, the error is smaller if the variance of each cluster distribution is smaller. We thus can conclude that what affects the error is a combination of cluster mean difference and the cluster variance.

Return to text


 Minskowski Measures

An alternate interpretation of the error norms is provided in the statistical literature as the Minskowski measures (see Bishop). In this perspective the L2 norm appears as the maximum likelihood solution when the instantaneous errors are Gaussian distributed. When the data set is such that they produce error pdfs that deviates from the Gaussian distribution, the L2 norm does not provide the maximum likelihood solution. If we have a priori information about the error distributions, the appropriate Minskowski measure can thus be used to establish the best performance.

In classification problems the most reasonable distribution seems to be the Bernoulli distribution, which points to the use of the Kullback-Leibler criterion.

Return to text


More Derivation of Performance Surface

 More Properties of the Performance Surface

For a quadratic performance surface Eq.1.9, computing the gradient and equating it to zero finds the value of the coefficients that minimize the cost, that is,

(1B.7)

or

(1B.8)

This solution is fundamentally the same as found in Eq.1.6 (b = 0 is equivalent to assuming that the average values of x and d are zero). Substituting this value of w* into Eq.1.9 , the minimum value of the error becomes

(1B.9)

Eq.1.9 can be rewritten in the form

(1B.10)

To verify this just sovle Eq. 1B.10 substituting Eq. 1B.8 for w* and Eq. 1B.9 for Jmin. Notice the following observations:

• The minimum value of the error Jmin Eq. 1B.9 depends on both the input signal (xi) and the desired signal (di)

• The location in coefficient space where the minimum w* occurs Eq. 1B.8 also depends on both xi, and di.

• The performance surface shape Eq. 1B.10 depends only on the input signal (xi)

Return to text


More Gamma Model Equations

 Gamma Model Equations

We have seen that changing the memory PE produces a different type of TLFN. The gamma model is built from combining a feedforward topology with memory PEs of the gamma type. The purpose of showing the equations here is to provide an introduction to the mathematical treatment of the gamma model (deVries and Principe). At the same time, the complexity of the equations highlights the advantages of using the data-flow implementation of backpropagation when compared with normal equation based learning. Using the convolution model of Eq. 11.B.7, we can easily obtain the dynamic equations that describe the gamma model as

(11B.9)

The following figure represents the block diagram of the gamma model.

Note that the second equation relates the signals at the gamma PE taps, while the first equation simply combines the temporal signals into a feedforward topology of nonlinear PEs. The equation that is specific to the gamma memory PE is therefore the second one.

To train the gamma model with BPTT, the gradients with respect to both the system states, and to the weights wij and [image] i are necessary as given by Eq. 11B.10. We present just the results here deVries and Principe.

(11B.10)

After analyzing the complexity of these equations, you can imagine the difficulty of making sure the equations are right and the training algorithms translate such procedures without errors. As we have been saying all along, however, the data-flow method avoids writing explicitly these equations. We just need to graphically specify the topology, have available the local maps for the nonlinear and gamma memory PEs, and specify the size of the trajectory in the dynamic controller.

Return to text


 More on Scheduling Step Sizes

If the initial value of [image] 0 is set too high, the learning can diverge. The selection of [image] can be even trickier than the selection of [image] because it is highly dependent on the performance surface. If [image] is too large, the weights may not move quickly enough to the minimum and the adaptation may stall. If [image] is too small, the search may reach the global minimum quickly and must wait a long time before the learning rate decreases enough to minimize the rattling. There are other (more automatic) methods for adapting the learning rate that we discuss later in the book.

Return to text


 More Versatile Memories

There are several linear operators fitting our definition of a generalized feedforward memory PE that can be used as short-term memory mechanisms for neural networks. We review here two useful cases, but the choice of the most appropriate memory PE for a given application remains an open question.

A set of bases intimately related to the gamma functions is built with the Laguerre functions. The Laguerre functions are an orthogonal span of the gamma space, which means that the information provided by both memories is the same. The Z transform of the Laguerre functions is given by

(10B.6)

It can be shown that the Laguerre functions can be recursively computed, following the definition of the generalized feedforward structures given in Eq. 10.7. In fact the Laguerre PE is a cascade of a lowpass filter with impulse response

(10B.7)

followed by a cascade of all-pass functions with a kernel given by

(10B.8)

It is easier to write the equations in the Z domain, as

(10B.9)

It is interesting to analyze the difference between the Laguerre and gamma memory traces: the gamma memory PE attenuates the signals at each tap because it is a cascade of leaky integrators with the same time constant. The Laguerre memory PE has a front-end lowpass filter, but the next stages are all-pass filters. Each all-pass filter places a zero in the mirror location (with respect to the unit circle) of the pole of the previous stage and includes a pole of their own at the same pole location. This topology effectively is creating a basis using a Gram-Schmidt orthogonalization of the tap signals after the lowpass filter, which we know produces an orthogonal basis when the input is an impulse (or white noise). The Laguerre PE does not attenuate the tap signals in amplitude but still delays the signal components. This leads to a set of oscillating basis functions. The following figure compares the response of the Laguerre and the gamma PE to an impulse, which produces the set of Laguerre and gamma bases, respectively.

The advantage of the Laguerre PE is that the tap signals (the bases) are obtained by convolving the lowpass-filtered input with an orthogonal set of functions (the all-pass functions), so the basis is less correlated and the adaptation speed becomes faster, specially when [image] is close to 0 or 2. The Laguerre memory is still very easy to compute and has only one free parameter [image]. The all-pass function is the closest realizable analog circuit implementation to the ideal delay operator.

The gamma memory PE has a multiple pole that can be adaptively moved along the real Z domain axis, i.e. the gamma memory can implement only lowpass (0< [image] <1) or highpass (1 < [image] <2) transfer functions. The highpass transfer function creates an extra ability to model fast-moving signals by alternating the signs of the samples in the gamma PE (the impulse response for 1< [image] <2 has alternating signs). But with a single, real parameter the adaptation is unable to move the poles to complex location in the Z domain. Two conditions may require a memory structure with complex poles: first when the information relevant for the signal processing task appears in periodic bursts; and second, when the input signal is corrupted by periodic noise. A memory structure with adaptive complex poles can successfully cope with these two conditions by selecting in time the intervals where the information is concentrated (or the time intervals where the noise is concentrated).

The following figure shows the gamma II kernel, implementation.

In the Z domain the transfer function of the gamma II kernel is

(10B.10)

which corresponds in the time domain to a kernel given by

(10B.11)

Notice that for stability, the parameters [image], [image] must obey the condition 0 < [image] (1+ [image])<2 and [image] >0. Complex poles are obtained for [image] > 0.

In terms of versatility, the gamma II has a pair of free complex poles, the gamma has a pole restricted to the real line in the Z domain, and the tap delay line has the pole set at the origin of the Z domain (z = 0). A multilayer perceptron equipped with an input memory layer with the gamma II memory structure implements a nonlinear mapping on an ARMA model of the input signal. The following table shows a comparison of the characteristics of the memory PEs.

tap delay
gamma
Laguerre
gamma II
Depth (k taps)

k
k/
No closed form
No closed form
Resolution

No closed form
No closed form
Stability
always
0< [image] <2
0< [image] <2
0< [image] (1+ [image])<2
Filter
allpass
lowpass
low/allpass
bandpass
Pole
0

r,w0

Return to text


 Multilayer Linear Networks

Why did we not discuss multilayer networks with linear PEs? It turns out that from the point of view of input-output mappings, a multilayer network with linear PEs is normally equivalent to a no-hidden-layer network (Adaline), so there is little interest in studying such networks. However, if we want to train one with gradient descent, we will need to use the backpropagation algorithm.

Return to text


 Multiple Variable Correlation Coefficient

The idea of the correlation coefficient is the same for oone or multiple dimensions. The equations get a little more complicated since we are now working with an ensemble of input vectors. So the nice form of Eq.1.7 has to be modified. An ensemble of vectors is better described as a matrix, so we are going to define a new matrix U as

where each column is one of the input samples. We are likewise going to define a column vector d with all the desired responses (this is a vector for the single-output regression, otherwise it also becomes a matrix)

The total error variance can be written as

where w* is the set of optimal coefficients. This expression can be easily derived if the output of the regressor is substituted in the definition of the error (Linear Models). The part of the error that is explained by the linear model is the second term. The variance of the output is expressed in the same way (just subtract the mean of the desired signal). Thus if we normalize this equation by the variance of the desired response we get

which leads to the correlation coefficient for the multivariate case.

Return to text


 Multiple-Hidden-Layer MLPs

The multilayer perceptron architecture is not in the general form of the projection theorem previously discussed. As seen in Chapter 3 the MLP implements an embedding of functions, so its approximation properties cannot be directly studied with the projection theorem except for the case of the one-hidden-layer MLP with linear output PE, as mentioned earlier. For this case we recognize that the output of each hidden PE is producing the elementary functions. When the output PE is a logistic (or tanh) function these values are nonlinearly combined, so the projection space is no longer a hyperplane in the input space.

Remember also the properties discussed in Chapter 3 that the MLP with two hidden layers is a universal approximator and, even with a single hidden layer, can approximate any continuous function on a compact set. There are many theorems that provide proofs of this property irrespective of the choice of nonlinearity. We can therefore conclude that the essence of the power of the approximation is in the topology, not in the specifics of the nonlinearity. As remarked before, these are existence theorems, so the designer still needs to select the topology to actually create the universal approximation properties for the class of functions of interest.

Return to text


 Nature of Analog Signals

Analog signals exist in a continuum. This means that between any two values in the time line there exist an infinite number of intermediate values. This causes problems when we want to represent analog signals in digital machines. To understand why, just think how many memory locations the computer needs to represent a piece of an analog signal between [0,1]. If there are an infinite number of points on the real line between 0 and 1 one would need an infinite number of memory locations. This is the reason that we need to sample or discretize the time line.

Return to text


 Newton's Derivation

The equation can be easily proved if we recall the gradient of the performance surface

left multiply by R [image] to obtain

and then substitute in the optimal solution Eq. 1.28 to obtain

From this equation we can derive the incremental equation presented previously.

Return to text


 Nonlinear System Identification with Neural Networks

System identification is a very important application in engineering. Very often we are interested in modeling the input-output relationship of an unknown plant. We saw in Chapter 9 that this problem is a practical application of function approximation. In fact, we can consider the output of the system d(n) as a fixed but unknown function of x(n), that is, d(n) = f(x(n)). The role of the model is to approximate the function f(.) by using a set of bases that are defined by the topology of the model system.

The linear combiner implements a set of linear bases, so we are effectively doing function approximation with a moving average (MA) linear system. We covered other possibilities in Chapter 9 but we restricted our attention to linear bases. In this chapter we relax this constraint and work with nonlinear systems, of the TLFN class. With TLFNs we can use the same block diagram (Figure 9-24 is copied below for reference) but develop nonlinear models of the unknown plant.

In nonlinear moving-average (NMA) modeling, the output of the model is a nonlinear function of its input, i.e.

(10B.2)

This equation states that the model is fundamentally feedforward. A TDNN is effectively a NMA model of the plant when it is used as the model. The TDNN is a nonlinear version of the linear combiner, and an alternative to Volterra models (which are also NMA).

In nonlinear autoregressive (NAR) models the output of the model is given by

(10B.3)

In this equation the next output is a function of the past values of the output. This type of model is used in prediction, where we train an NMA model, but due to the block diagram form of prediction, we are fitting a NAR model to the data.

In general we use a nonlinear autoregressive with external input (NARX) model for system identification:

(10B.4)

In other words, we drive our TLFN with a signal (also fed to the plant) and the past outputs of the TLFN.

The nonlinear autoregressive moving-average (NARMA) is the most general class of nonlinear models and it is a blend of the two previous types, i.e.

(10B.5)

That is, we drive the TLFN with its past outputs (global feedback) and with the input and its delayed versions.

A fundamental aspect of system identification is to understand when a given model should be used given the knowledge that we have about the plant. In terms of the function approximation paradigm, this means that we have to choose the basis appropriately. A related problem is how we are going to parameterize our nonlinear models. Once again the NARMA (Eq. 10B.5) seems to be a very general parameterization, and in some cases simpler but easier to train parameterizations may exist.

Return to text


Nyquist Theroem

 Nyquist Theorem

Let us take a sine wave of period T, and sample it with a frequency 10/T, that is, so that there are 10 samples per cycle. Frequency is defined as f = 1/T and is measured in Hertz (or radians per sec for [image]. The samples obviously preserve the information of the sine wave (amplitude and frequency) as shown in the following figure.

Now sample the same sine wave at f = 2/T, (i.e. two samples per cycle). We still get a sine wave of the same frequency, but notice that we have only 1 point per half cycle, so we are getting dangerously close to distorting the wave.

Try now f = 1/T. You see that we obtain a flat line, that is, the information in the samples does not match anymore the original signal. Try now f = 1/2T. We get back a sine wave, but now of a different frequency!. We say that we are aliasing the sine wave.

This example shows that sampling has to be done according to the fastest variation contained in x(t). We saw in Chapter 5 that any periodic signal could be decomposed in sine waves, so to preserve the information contained in x(t), we have to find its maximum frequency and choose a sampling frequency at least twice this value. This is called the Nyquist theorem.

The Nyquist theorem makes a lot of sense if we think about it carefully. It basically says that if we want to represent fine detail in a waveform, we have to have sufficiently dense samples. Otherwise we end up with digital signals that do not resemble the original analog ones. This error is called the aliasing error, the error that distorts the analog signals when they are sampled due to too low a sampling frequency.

Given an analog signal x(t) and a sampling frequency fs, sampling preserves the information of x(t) only up to fs/2. All the other frequencies present in x(t) are aliased back to the range 0 - fs/2. So there are two potential types of errors when digitizing an analog signal:

  • Information from frequencies above fs/2 are lost,

  • Higher frequencies in the analog signal can even add noise to the lower frequencies since they appear aliased back in the interval 0-fs/2.

This is the reason why a golden rule of sampling is to always prefilter the input signal with an analog filter with a cutoff at fs/2 (more on filters later in the chapter). This practice avoids aliasing, but it cannot compensate for the missing frequencies above fs/2.

Return to text


 Optimal LAMs

The LMS is an iterated method to train a LAM optimally. One can show that the optimal LAM weights have to meet the solution

which exists as long as the inverse of X exists (here X and Y are the matrices constructed from the full training set). This means that the patterns must be linearly independent (instead of orthogonal, as required for Hebbian training). If we have fewer patterns than inputs, the optimal solution is not unique. We can show (Kohonen ) that in this case

which involves the computation of X+, the pseudo-inverse of X. There is a method to compute W* recursively (Greville's theorem) using a nonlocal algorithm that resembles the LMS. The LMS with a small step size is a good approximation to this recursive algorithm. See also Hecht-Nielsen.

Return to text


 Optimizing Linear and Nonlinear Systems

You may know that our mathematical knowledge is very limited in computing solutions for nonlinear differential equations, so it may seem hopeless to attack the problem of optimizing a nonlinear system. Although it is true that we lose the ability to analytically solve for the minimum, we can still use iterative procedures to find it. This is one of the advantages of optimization techniques.

For the linear problem treated in Chapter 1, the analytical solution can be applied and always produces the best possible results. However, the analytic approach cannot be used for the M-P PE. Iterative optimization (in the form of the gradient-descent procedure) can be applied to problems that are beyond analytical solutions. As we are going to see, we can apply gradient descent as long as the function is smooth, which can include many nonlinearities.

Return to text


Origin of LBG Algorithm

 LBG Algorithm

Stands for Linde, Buzo and Gray who proposed the technique in 1980. See Linde Y., A. Buzo and R. Gray, An algorithm for vector quantization design, IEEE Trans. on Communiations, COM-28, 84-95, 1980.


 Parametric and Nonparametric Classifiers

The classifier we just discussed is called a parametric classifier because the discriminant functions have a well-defined mathematical functional form (Gaussian) that depends on a set of parameters (mean and variance). For completeness we should mention nonparametric classifiers, where there is no assumed functional form for the discriminants. Nonparametric classification is solely driven by the data (as in K nearest neighbors; see Fukunaga). These methods require a great deal of data for acceptable performance, but they are free from assumptions about the shape of discriminant functions (or data distributions) that may be erroneous.

Return to text


 Parseval's Theorem

Parseval's theorem is well known in linear system theory and is a consequence of the property that vectors do not change length when the orthonormal basis is changed. The theorem states that

where C is a closed contour that belongs to the region of convergence of X(z). In practical terms what this theorem says is that the energy of the sequence x(n) can be computed either in the time domain (left side) or through the spectrum (right side). For the discrete Fourier transform (DFT) this expression reduces to

Return to text


 Parzen Window Method

The Parzen window method is a nonparametric density-estimation method widely used in statistics. It is related to learning because it provides a way to estimate the pdf of the data, and it can be applied to a wide range of functions.

In the Parzen window method we start by choosing a symmetric normalized area and unimodal kernel function

and construct the pdf estimator

Commonly used kernels are the Gaussian, the rectangular, and the spectral windows (Tukey, Hanning, Hamming). The Parzen estimator is consistent, and its asymptotic rate of convergence is optimal for smooth densities, but it requires large number of samples to provide good results.

Return to text


 PCA, SVD, and KL Transforms

We have to briefly cover the mathematics of principal component analysis (PCA) to fully understand and apply the concept.

PCA and singular value decomposition (SVD) are intrinsically related. Let us start with the SVD because it is an algebraic operation applicable to any matrix. The goal of SVD is to diagonalize any matrix, that is, to find a rotation where only the diagonal elements are nonzero.

Consider the matrix Z with M rows and N columns (M x N). For every such matrix there are two orthonormal matrices, U (M x M) and V (N x N) and a pseudodiagonal matrix D = diag{ [image] 1, , [image] p} (M x N) where P = min{M,N} such that

The vectors u and v are called the left and right singular vectors of Z, while the [image] are called the singular values of Z.

SVD is intrinsically related to the eigendecomposition of a matrix. In fact, if we postmultiply by Z [image] U we obtain ZZ [image] U = UDV [image] Z [image] U = UDD [image] . We can likewise show that Z [image] ZV = VD [image] D. Now DD [image] and D [image] D are square diagonal matrices, and so the vectors u and v are the eigenvectors of the matrices ZZ [image] and Z [image] Z respectively,

Now let us define PCA. Consider a vector [image] with mean zero, and R = E[xx [image]] which is a symmetric matrix (DxD). PCA produces a linear transformation of the data y = Wx where the columns of W form an orthonormal basis. PCA has a very nice property: it minimizes the mean square error between the projected data (to a subspace M) and the original data. The reconstructed data from the projections is a vector

PCA thus minimizes

where tr (.) means the trace of the matrix. The trace of WRW [image] is effectively the variance of y, i.e.

.

Thus the minimization of J implies the maximization of the variance of y, which is also the variance of the estimated projection. This provides still another interpretation for PCA: PCA is the linear projection that maximizes the variance (power) of the projection to a subspace.

It is interesting to analyze the characteristics of the PCA projection, that is, the structure of W. If the eigenvalues of R {e1,….eD} are ordered in descending order of the eigenvalues { [image] 1,… D}, we can show that

These two equations basically state that if we project the data with PCA to a subspace of dimension M we preserve the variance given by the sum of the first M (principal) eigenvalues. The error can also be easily obtained by adding the D-M-1 (minor) eigenvalues.

The projections are called the principal components of x. They are statistically uncorrelated:

and their variances are equal to the eigenvalues of R,

and are arranged by descending order of variance. Since R is a symmetric matrix, we can say that it can always be decomposed as

where [image] is a diagonal matrix with entries [image] i, the eigenvalues, and ei are the eigenvectors. This equation shows the special function that eigenvectors represent for a matrix. They diagonalize it, that is, they represent the direction in space where we can compute the entries of the matrix using only scalar operations. Alternatively, once the eigenvectors and eigenvalues are known, we can construct R with scalar operations! This means we have found the structure of the data.

We see that the PCA is actually operating with the eigenstructure of R, hence its importance. In general only the data is known, not R. Even when R is known, normally its eigenstructure is not quantified. But when we perform PCA we discover the dependencies on the data, and we can even project it to a subspace to simplify the analysis and lose the least variance.

Now the equivalence between SVD and PCA should be clear. In fact, if Z is square and symmetric, the two orthogonal matrices U and V become the same, and SVD becomes equivalent to PCA.

Last, we would like to define the Karhunen-Loeve transform (KLT). This transform was originally developed to study decompositions of continuous time signals. But for finite duration (D) discrete signals, it can be formulated in the following way:

Consider the stationary random process x(n) with zero mean and autocorrelation matrix R with elements R(l,k) = E[x(n-k)x(n-l)]. The KLT is defined as the set of basis ui(n) that satisfy the relation

We can write this expression in matrix form to read Rui = [image] iui, and we immediately recognize the eigenvalue equation involving the (time) autocorrelation of the data. Therefore, KLT and PCA yield the same solution for the case of finite duration discrete signals.

Return to text


 Perceptron Learning Algorithm

Let us study in more detail the perceptron learning algorithm. Notice that Eq. 3.9 corrects the weights only under the condition that y is different from d. There are two cases to consider: either d = 1 (class 1) and y = -1 in which case

or d = -1 (class 2) and y = 1 in which case

Now let us see how these cases arise in practice. Let us consider the 1-D case, with just one weight, as in the figure. In the first case (case A) xA(n) belongs to class 1 but the output is -1, so the present weight wA(n) must be in the left part of the figure (otherwise the output would be positive and there would be no correction). In this case the update adds a value to the weight, moving it in the correct direction.

In the second case (case B), the input pattern xB(n) belongs to class 2, so if the perceptron is outputting a positive value, it is because the weight wB(n) is in the right part of the figure. In this case the weight is decreased according to the equation. In both cases we see that the weight is moved to the shaded region that produces the correct classification. One can show mathematically that for linearly separable patterns this algorithm converges in a finite number of steps (Haykin).

Return to text


 Performance Surface Properties

The minimum value of the error can be obtained by substituting the optimal weight Eq.1.31 into the cost equation Eq.1.30, yielding

(1B.18)

We can rewrite the performance surface in terms of its minimum value and w* as

(1B.19)

For the one dimensional case, this equation is the same as Eq.1B.10 (R becomes a scalar equal to the variance of the input). In the space (w1,w2) J is now a paraboloid facing upward. The shape of J is again solely dependent on the input data (through its autocorrelation function). One can show that:

  • the principal axes of the performance surface contours (surfaces of equal error) correspond to the eigenvectors of the input correlation matrix R, (see Appendix A)

  • while the eigenvalues of R give the rate of change of the gradient along the principal axis of the surface contours of J (Figure 1-16).

Figure 1-16 Contour plots of the performance surface with two weights

The eigenvectors and eigenvalues of the input autocorrelation matrix are all that matters to understand convergence of the gradient descent in multiple dimensions. The eigenvectors represent the natural (orthogonal) coordinate system to study the properties of R. In fact, along these coordinates the convergence of the algorithm can be studied as a joint adaptation of several (one for each dimension of the space) unidimensional algorithms. Along each eigenvector direction (the axes of the ellipsoids) the algorithm behaves just like the one-variable case that we studied in the beginning of this chapter. The eigenvalue becomes the projection of the data onto that direction, just as [image] in Eq.1B.11 is the projection of the data on the weight direction. But in any other direction the adaptation is coupled.

The location of the performance surface in weight space depends on both the input and desired response Eq.1.31. The minimum error also depends on both Eq.1B.18. Multiple regression finds the location of the minimum of a paraboloid placed in an unknown position in weight space. The input distribution defines the shape of the performance surface. The input distribution and its relation with the desired response distribution define both the value of the minimum of the error and the location in coefficient space where that minimum occurs.

Return to text


 Pop-up Windows

This is an example of a popup window.


 Power Spectrum

The power spectrum is the Fourier transform of the autocorrelation function.


 Probabilistic Interpretation of Sigmoid Outputs

According to Bayes' rule, the a posteriori probability can be written as

Note that for the two class case the denominator can be written as [image], where C1 and C2 are the two classes. Now if the likelihoods are Gaussians of equal variance, it is not difficult to show that

where

Note that this is exactly the definition of the activation function of the logistic PE, so we can interpret the output of this PE as the a posteriori probability of the class given the data.

Return to text


 Projection with the Context PE

The projection implemented by the context PE cannot be analyzed in finite dimension vector spaces, since the impulse response is infinite. However, we can approximately study it taking into consideration that there is a finite effective memory depth given by 1/ [image]. Using this approximation the context PE can be interpreted as a projection on a linear manifold of size int(1/ [image] [image].

The number of degrees of freedom of this projection is nevertheless very different from the delay line. In the delay line the bases are the past values of the input. The output is constructed by weighting the values of each basis with the filter coefficients to obtain a direction. We have as many coefficients as bases to choose the direction of the projection.

In the case of the context PE a similar model can be used to analyze the projection operated by the system, but the details are different. The bases can still be considered the previous samples but now the weighting is not independent from axis to axis, it is fixed by the value of the feedback coefficient. The most recent past sample is weighted by 1- [image], two samples in the past by (1 [image] [image])², an so on. There is no flexibility to choose the values of the filter coefficients independently. This means that the projection vector is always biased toward the first basis, where the weights are larger. Obviously, this limits the projection directions that can be created with the context PE. The context PE is able to create projections that are only lowpass filters of the input.

The projection vector always exists in the first quadrant. The direction approaches the bisector of the first quadrant when [image] approaches 0, and the projection space dimension is very large (since the effective dimension is 1/ [image]). The context PE is therefore practically equivalent to the linear combiner with all weights equal to one which we identified before as a lowpass filter.

In the other extreme of [image] close to 1, the projection approaches the first projection axis (since the largest weight is associated with the sample n-1, and the geometric ratio is close to zero), which is able to represent the input without error in large-dimensional spaces, but the size of the projection space is very small since it is 1-D (just the present sample). Hence there is a clear trade-off between the size of the projection space and the resolution of the representation.

The conclusion of this analysis is the following. With the context PE we do not need to consider the projection space, since we control the location of the projection vector directly (i.e., this system is single-input, single-output). However, we can still analyze the projection as in the case of the linear combiner, but now there is only one free parameter that controls the direction of the projection and the effective size of the memory space at the same time. The context PE is only able to create lowpass projections. Another important characteristic is that the size of the projection space is controlled by the free parameter.

Return to text


Proof of Eigenequation

 Proof of Eigen-Equation

To enhance the readability of the equations, we will use matrix notation and rewrite Oja's rule as

The differential equation that corresponds to this difference equation is

Any solution of this equation has to be an eigenvector ei of R (see, for instance, Diamantaras and Kung). Now writing w(t) as a linear combination of its basis vectors ei

one can further show that the weights adapted with Oja's rule converge with probability 1 to either e1 or -e1 (i.e., to the eigenvector that corresponds to the largest eigenvalue of R). This is the reason a linear network adapted with Oja's rule is sometimes called the maximum eigenfilter.

Return to text


 Properties of DFS

The discrete Fourier series (DFS) is so important in signal processing that we have to provide here a synopsis of its properties. Moreover, we will see that knowing these properties for the DFS is enough to know and understand the properties of the other Fourier analysis transforms.

  1. The DFS is a linear operator:

  2. Parseval's theorem: The energy of a signal can be computed either in the time or in the frequency domain

  3. Delaying a time signal multiplies the transform by equally delayed basis:

  4. The convolution in time is equivalent to a multiplication of the transforms (and vice versa).

Return to text


 Quantized Signal

A quantized signal is a signal whose amplitude is a multiple of a quantity called the quantization step. The quantization step is chosen according to the number of bits of the A/D converter utilized, and it is related to the precision of the voltage measurement performed in the converter. For instance, an 8 bit A/D converter has a quantization step of 2¯ [image]. This gives exactly 256 different values to measure the amplitude of the analog signal. Normally there are 128 positive values and 128 negative values.

We can think that the A/D converter creates a grid in the amplitude axis. When the A/D converter receives the command to convert, it measures the analog voltage and then assigns the voltage to the closest point in this amplitude grid.

We can see that there is an error in going from the analog voltage to the quantized voltage, but this error is always smaller than the quantization step.

A good rule of thumb is to consider that the signal-to-noise ratio (SNR) is 6 dB/bit. A b-bit A/D converter has a SNR of 6(b-1) dB, since one bit is the sign bit. For instance, an 8-bit A/D converter has a 42 dB SNR. This means that the quantification noise floor of an 8-bit A/D is 100 times smaller than the largest signal.

Return to text


 RBF as Kernel Regression

For RBF kernel regression the windows are symmetric multidimensional Gaussian functions (as utilized in RBF networks) that quantify the joint data distribution in the input-desired signal space, given by

As we discussed in Section 5.3.4, regression can be thought of as estimating the condition average of the target data ti conditional to the input xi, i.e. . When the MSE is used, the output of the network approaches this value. The conditional average can be written as a function of the pdf

This yields Eq. 5.35 in the text. In general, we can use fewer Gaussians, as done in the RBF network, yielding

where [image] are the centers of the Gaussians in the desired signal space.

Return to text


 Rederivation of Backpropagation with Ordered Derivatives

With this theory we can rederive the backpropagation procedure for any ordered topology very quickly. We assume a network (with smooth nonlinearities) given by Eq. 3.33. Dependence on the iteration is not included for simplicity. The direct effect in Eq. 3.33 is computed as

(3B.7)

where [image] i is the external injected error. We can rewrite Eq. 3.35 as

(3B.8)

Note that the partial derivative of J with respect to the state is equal to the injected error [image] i for an output PE and zero for the other cases. With the assignment that we used before,

(3B.9)

we can write

(3B.10)

To compute the gradients with respect to the weights, we use Eq. 3.36 to get

(3B.11)

Note that the sum extended to all PEs reduces to one term k = i, because we are computing the local contribution to the weight wij.

The sensitivity of the cost can finally be written

(3B.12)

We can recognize equations 3B.12 and 3B.10 as the equations necessary to implement the backpropagation procedure (respectively Eq. 3.29 Eq. 3.31) that we derived PE by PE in the text. But here they were derived in a much more compact form. The weight update using the gradient descent procedure is

Return to text


Reference for A. Shepherd

 Shepherd, A.

Second-order methods for neural networks, Springer, London, 1997


Reference for Abarbanel (1996)

 Abarbanel, H.

Analysis of Observed Chaotic Data, Springer, New York, 1996.


Reference for Adatron

 Adatron

Anlauf J., and Biehl M., The Adatron: an Adaptive Perceptron Algorithm, Europhysics Letters, 10(7), 687-692, 1989.


Reference for Akaike

 Akaike, H.

A new look at the statistical model identification, IEEE Trans. Auto. Control, AC-19, 716-723, 1974


Reference for Almeida (1987)

 Almeida, L.

A learning rule for asynchronous perceptrons with feedback in a combinatorial environment, 1st IEEE Int. Conf. Neural Networks, vol 2, 609-618, 1987.


Reference for Anderson, Silverstein, Ritz and Jones (1977)

 Anderson, Silverstein, Ritz and Jones

Distinctive features, categorical perception, and probabilty learning: some applications of a neural model, Psychological Review 84, pp. 413-451, 1977.


Reference for Andrew Barron

 Barron, A.

Universal approximation bounds for superpositions of sigmoid functions, IEEE Trans. Information Theory, 39, #3, 930-945, 1993.


Reference for APEX

 APEX

see Principal Component Neural Networks: Theory and Applications by Diamantaras and Kung, Wiley, 1996, page 90.


Reference for Baldi and Hornik (1989)

 Baldi, P. and K. Hornik

"Neural networks and principal component analysis: learning from examples without local minima", Neural networks 1, 53-58, 1989.


Reference for Barak Pearlmutter

 Pearlmutter, B.

Fast exact multiplications by the Hessian, Neural Computation 6 (1), 147-160, 1994


Reference for Barron

 Barron, A.

Approximation and estimation bounds for ANNs, IEEE Trans. Information Theory 39, #3, 930-945, 1993.


Reference for Bengio, Simard and Frasconi (1994)

 Bengio, Y., P. Simard and P. Frasconi

Learning long term dependencies with gradient descent is difficult, IEEE Trans. on Neural Networks, 5, 157-166, 1994.


Reference for Bishop

 Bishop

Neural Networks for Pattern Recognition, Oxford, 1995.


Reference for Bishop

 Bishop C.

Pattern Recognition with Neural Networks, Oxford, 1995.


Reference for Casti

 Casti

Casti, J., Alternate Realities: Mathematical Models of Nature and Man, Wiley, New York, 1989.


Reference for Chris Bishop

 Bishop, C.

Neural Networks for Pattern Recognition, Oxford, 1995, pp 347.


Reference for Cohen and Grossberg (1983)

 Cohen, M. and S. Grossberg

Absolute stability of global patterns formation and parallel storage by competitive neural networks, IEEE Trans. Syst. Man. and Cybernetics, vol. SMC-13, 99 pp. 815-826, 1983.


Reference for Cover and Thomas

 Cover, T. and J. Thomas

Elements of Information Theory, Wiley, New York, 1991.


Reference for Cover and Thomas (1991)

 Cover, T. and J. Thomas

Elements of Information Theory, Wiley, New York, 1991.


Reference for Curt Lefebvre

 Lefebvre, C.

An Object-Oriented Approach for the Analysis of Neural Networks, Master Thesis, University of Florida, 1991


Reference for Cybenko

 Cybenko

Approximation by Superposition of a Sigmoid Function, Mathematics for Control, Signals, and Systems, 2(4):303-314, 1989.


Reference for D. Luenberger

 Luenberger D.

Linear and Nonlinear Programming, Addison Wesley, Reading MA, 1986


Reference for deVries and Principe (1992)

 deVries and Principe

The gamma model: a new neural network model for temporal processing, Neural Networks 5, 565-576, 1992.


Reference for deVries and Principe (1992)

 deVries, A., J. Principe

The gamma model: A new model for temporal processing, Neural Networks, vol 5, pp 565-576, 1992.


Reference for Diamantaras and Kung (1996)

 Diamantaras, K. and Kung, S.

Principal Component Analysis Networks: Theory and Applications, Wiley, New York, 1996.


Reference for Duda and Hart

 Duda and Hart

Duda R. and Hart P., Pattern Classification and Scene Analysis, Wiley, 1973.


Reference for Duda and Hart (1973)

 Duda R. and P. Hart

Pattern Classification and Scene Analysis, Wiley, New York, 1973.


Reference for Elman (1990)

 Elman, J.

Finding strucutre in time, Cognitive Science 14, pp 179-211, 1990.


Reference for Estimation Theory

 Estimation Theory

Melsa J., and D. Cohn, “Decision and Estimation Theory”, McGraw-Hill, 1978.


Reference for Feldkamp and Puskorius (1998)

 Feldkamp L. and G. Puskorius

A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification, Proc. IEEE, vol. 86, pp. 2259-2277, 1998.


Reference for Filter Design

 Filter Design

see Oppenheim and Shafer, Discrete time signal processing, Prentice Hall, Englewood Cliffs, NJ, 1987.


Reference for Foldiak (1989)

 Foldiak, P.

Adaptive network for optimal linear feature extraction, Proc. Int. J. Conf. Neural Networks, vol I, 401-405, 1989.


Reference for Freeman (1975)

 Freeman, W.

Tutorial on Neurobiology: from single neurons to brain chaos, Int. J. of Bifurcation and Chaos, 2, 451-482, 1992.

See also the book "Mass Activation of the Nervous System", Academic Press, 1975.


Reference for Freiss

 Freiss, Thilo

Support Vector Neural Networks: The kernel Adatron with bias and soft margin, University of Sheffield Technical Report, 1998.


Reference for Fukunaga

 Fukunaga

Fukunaga K., Statistical Pattern Recognition, Academic Press, 1990.


Reference for Gallant

 Gallant

On Learning Derivatives of an Unknown Function with MLPs, Neural Networks 5 (1), 129-138, 1992.


Reference for Gamma Net

 Gamma Net

Bert DeVries and Principe, "The gamma model: a new neural model for temporal processing", Neural Networks 5, 565-576, 1992.


Reference for Giles, Lin and Horne (1997)

 Giles, L., T. Lin and B. Horne

Remembering the past: the role of embedded memory in recurrent neural architectures, Proc. IEEE Workshop Neural Networks for Signal Processing VII, IEEE Press, pp. 34- 43, 1997.


Reference for Golub and Van Loan (1989)

 Golub G., Van Loan, C.

Matrix Computation, John Hopkins University Press, Baltimore, MD, 1989.


Reference for Grossberg (1982)

 Grossberg, S.

Studies of Mind and Brain, Reisel, Boston, MA, 1982.


Reference for Grossberg (1992)

 Grossberg, Stephen

Natural Intelligence, MIT Press, Boston, 1992.


Reference for Haykin

 Haykin

Neural Networks: A Comprehensive Foundation. MacMillan, 1994. pp108.


Reference for Haykin

 Haykin, Simon

Neural Networks: A Comprehensive Foundation, McMillan, 1994.


Reference for Haykin (1986)

 Haykin, S.

Adaptive Filter Theory, Prentice-Hall, Englewood Cliffs, NJ, 1986.


Reference for Haykin (1994)

 Haykin, S.

Haykin S., Neural Networks: A Comprehensive Foundation, McMillan, New York, 1994.


Reference for Haykin and Principe (1998)

 Haykin and Principe

Haykin S., Principe J., "Dynamic modeling with neural networks", in IEEE Signal Processing Magazine, vol 15, #3, 66, 1998.


Reference for Haykin and Van Veen (1999)

 Haykin and Van Veen

Signals and Systems, Wiley, New York, 1999.


Reference for Hebb (1949)

 Hebb, Donald

The Organization of Behavior: A Neurophysiological Theory, Wiley, New York, 1949.


Reference for Hecht-Nielsen (1990)

 Hecht-Nielsen, R.

NeuroComputing, Addison Wesley, Reading MA, 1990.


Reference for Hecht-Nielsen (1991)

 Hecht-Nielsen R.

NeuroComputing, Addison Wesley, Reading, MA, 1991.


Reference for Hertz, Krogh, and Palmer

 Hertz J., A. Krogh and R. Palmer

Introduction to the Theory of Neural Computation, Addison Wesley, Reading MA, 1991


Reference for Hopfield and Tank (1986)

 Hopfield, J. and D. Tank

Computing with neural circuits: a model, Science, vol 233, pp 625-633, 1986.


Reference for Hornik, Stinchcombe and White

 Hornik, Stinchcombe and White

MLPs are Universal Approximators, Neural Networks, 2:359-366, 1989.


Reference for Horst, Pardalos and Thoai

 Horst R., P. Pardalos and N. Thoai

Introduction to Global Optimization, Kluwer, 1995


Reference for Jordan (1986)

 Jordan, M.

Attractor dynamics and parallelism in a connectionist sequential machine, Proc. 8th annual Conf. on Cognitive Science Society, pp 532-546, 1986.


Reference for Kaplan and Glass (1995)

 Kaplan, D. and L. Glass

Understanding nonlinear dynamics, Springer Verlag, New York, 1995.


Reference for Karhunen and Loeve

 Karhunen and Loeve

Two European matematicians who independently discovered the eigendecomposition in Hilbert spaces. See Loeve, Probability Theory, Van Nostrand, New York, 1963.


Reference for Kay

 Kay, S.

Modern Spectral Estimation, Prentice Hall, Englewood Cliffs, NJ, 1988.


Reference for Kay (1988)

 Kay, S.

Modern Spectral Analysis, Prentice Hall, Englewood Cliffs, NJ, 1988.


Reference for Kay (1988)

 Kay, S.

Modern Spectral Estimation, Prentice Hall, Englewood Cliffs, NJ, 1988.


Reference for Kohonen (1984)

 Kohonen, Teuvo

Self-Organization and Associative Memory, Springer Verlag, New York, 1984.


Reference for Kohonen (1995)

 Kohonen, T.

Self Organizing Maps, Springer Verlag, New York, 1995.


Reference for Kolmogorov (1977)

 Kolmogorov , A.

(Russian mathematician), see the English translation in the book by Kailath, Linear least-square estimation, Dowden, Hutchinson and Ross, Stroudsburg, Pa, 1977.


Reference for Learning Theory

 Learning Theory

see Vladmir Vapnik, "The nature of statistical learning theory", Springer, 1995.


Reference for LeCun, Denker and Solla

 LeCun, Y., J. Denker and S. Solla

Optimal brain damage, Advances in Neural Information Processing Systems, vol 2, 598-605, 1993.


Reference for LeCun, Simard and Pearlmutter

 LeCun, Y., P. Simard and B. Pearlmutter

Automatic learning rate maximization by on-line estimation of the Hessian eigenvectors, in Advances of Neural Information Processing Systems, vol 5, 156-163, 1993, Morgan Kaufmann.


Reference for Linear Models

 Linear Models

Dunteman G., Introduction to Linear Models, Sage Publications, 1984.


Reference for M. Perrone

 Perrone, M.

General averaging results for convex optimization, in Mozer et al (Ed), Proc. 1993 Connectionist Models Summer School, 364-371, Lawrence Erlbaum, 1994.


Reference for N. Campbell

 Campbell, N.

Campbell N., and Mahon R., A multivariate study of variation in two species of rock crabs of genus Leptograpsus, Australian Journal of Zoology, 22, 417-425, 1974.


Reference for Narendra and Parthasarathy (1990)

 Narendra, K. and K. Parthasarathy

"Identification and control of dynamical systems", IEEE Trans. neural networks 1(1):4-27, 1990.


Reference for Nilsson

 Nilsson

Mathematical Foundation of Learning Machines, 1990 Morgan Kaufmann.


Reference for Oja (1982)

 Oja, E.

A simplified neuron model as a principal component analyzer, J. of Mathematical Biology, 15, 239-245, 1982.


Reference for Oppenheim and Schafer (1989)

 Oppenheim, A. and Schafer, R.

Discrete Time Signal Processing, Prentice Hall, Englewood Cliffs, NJ, 1989


Reference for Orr and Muller

 Orr and Muller

Orr, G and Muller, K., "Neural Networks: Tricks of the Trade," Lecture Notes in Computer Science, vol 1524, Springer Verlag, New York, 1998.


Reference for Park and Sandberg

 Park J. and I. Sandberg,

"Universal approximation using radial basis function networks", Neural Computation, vol 3, 303-314, 1989.


Reference for Parzen

 Parzen, E.

On estimation of a probability density function and mode", Annals of Mathematical Statistics, 33, 1065-1076, 1962.


Reference for Poggio and Girosi

 Poggio, T. and F. Girosi

Networks for approximation and learning, Proc. IEEE, vol 78, 1990.


Reference for Principe and Smith (1986)

 Principe, J. and J. Smith

Design and implementation of linear phase FIR filters for biological signal processing, IEEE Trans. Biomed. Eng., vol BME 33, 6, pp 550-559, 1986.


Reference for Principe deVries and Oliveira (1993)

 Principe, J., A. deVries and P. Oliveira

The gamma filter: a new class of adaptive IIR filters with restricted feedback, IEEE Trans. Signal Proc., vol 41, n2, pp 649-656, 1993.


Reference for Principe, Wang and Wu (1996)

 Principe, J., C. Wang and H. Wu

Temporal decorrelation using teacher forcing anti-Hebbian learning and its applications in adaptive blind source separation, Proc. IEEE Workshop on Neural Networks for Signal Processing, 413-422, 1996.


Reference for Puskorius, Feldkamp and Davis (1996)

 Puskorius, G., L. Feldkamp and L. Davis

Dynamic neural networks methods applied to on-vehicle idle speed control, Proc. IEEE, vol 84, 1407-1420, 1996.


Reference for R.A. Fisher

 Fisher, R.A.

The use of multiple measurements in taxonomic problems, Annals of Eugenics, 7, Pt II, 179-188, 1936.


Reference for Rabiner and Juang (1986)

 Rabiner, L. and B. Juang

An Introduction to Hidden Markov Models, IEEE ASSP Magazine, vol. 3, pp 4-16, 1986.


Reference for Rao and Huang (1996)

 Rao, K. and J. Huang

Techniques and Standards for Image Video and Audio Compression, Prentice Hall, Englewood Cliffs, 1996.


Reference for R.Fletcher

 Fletcher, R.

Practical Methods of Optimization, Wiley, New York, 1987


Reference for Rissanen

 Rissanen, Jorma

Stochastic Complexity in Statistical Inquiry, World Scientific, 1989.


Reference for Rosenblatt

 Rosenblatt, Frank

invented the perceptron in 1958, considered by many as the first learning machine. See Principles of Neurodynamics, Washington DC, Spartan Press, 1961.


Reference for Sandberg (1977)

 Sandberg, I. and L. Xu

Uniform approximation and gamma neural networks, Neural Networks, vol 10, 781-784, 1977.


Reference for Sandberg and Xu (1997a)

 Sandberg, I. and L. Xu

Uniform Approximation of Multidimensional Myopic Maps, IEEE Trans, on Circuits and Systems, vol. 44, pp 477-485, 1997.


Reference for Sanger (1989)

 Sanger, T.

Optimal unsupervised learning in a single layer linear feedforward neural network, Neural Networks, 12, 459-473, 1989.


Reference for Scott Fahlman

 Fahlman, S.

Fast Learning Variations of Backpropagation: an Empirical Study, in Proceedings 1988 Connectionist Models Summer School, 38-51, Morgan Kaufmann, 1988.


Reference for Shannon (1948)

 Shannon, Claude

A mathematical theory of communications, Bell Sys. Tech. Journal, 27:379-423, 623-656, 1948.


Reference for Silva and Almeida

 Silva, F. and L. Almeida

Acceleration techniques for the backpropagation algorithm, in Almeida and Wellekens (Eds), Neural Networks, Lecture notes in Computer Science, 110-119, Springer, 1990.


Reference for Silva and Almeida (1991)

 Silva, F. and L. Almeida

A distributed decorrelation algorithm, in Gelenbe (Ed.), Neural Networks, Advances and Applications, North Holland, 1991.


Reference for Simon Haykin

 Haykin, S.

Artificial Neural Networks: A Comprehensive Foundation, IEEE Press, New York, 1995.


Reference for Tikonov

 Tikhonov, A.

A. Tikhnov and V. Arsenin, "Solution of ill-posed problems", Winston, Washington, 1977.


Reference for Van Trees (1968)

 Van Trees, H.

Detection, Estimation, and Modulation Theory, Wiley, New York, 1968.


Reference for Vapnik

 Vapnik

The nature of statistical Learning Theory, Springer Verlag, 1995, pp 136.


Reference for Vapnik

 Vapnik, Vladimir

The Nature of Statistical Learning Theory, Springer Verlag, 1995 and Statistical Learning Theory, Wiley, 1998.


Reference for Vladimir Vapnik

 Vladimir Vapnik

The Nature of Statistical Learning Theory, Springer Verlag, New York, 1995.

pp 128


Reference for Vladimir Vapnik

 Vapnik, V.

The Nature of Statistical Learning Theory, Springer Verlag, New York, 1995.


Reference for Wahba

 Wahba, Grace

Spline Models for Observational Data, SIAM, 1990.


Reference for Waibel (1989)

 Waibel. A.W.

Phoneme recognition using time delay neural networks", IEEE Proc. ASSP-37, 328-339, 1989.


Reference for Wan (1994)

 Wan, E.

Time series prediction using a a network with internal delay lines, in Time series prediction:Forecasting the future and understanding the past, 195-217, Addison-Wesley, 1994.


Reference for Wan (1996)

 Wan, E.

Diagramatic derivation of gradient algorithms for neural networks, Neural Computation, vol 8, 182-201, 1996.


Reference for Werbos

 Werbos

Werbos, P. "Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences", Ph.D. dissertation Harvard, 1974.


Reference for Widrow and Stearns

 Widrow B. and S. Stearns

Adaptive Signal Processing, (Chapter 4), Prentice Hall, Englewood Cliffs, NJ, 1985.


Reference for Widrow and Stearns (1985)

 Widrow, B. and S. Stearns

Adaptive Signal Processing, Prentice Hall, Englewood Cliffs, NJ, 1985.


Reference for Widrow and Hoff

 Widrow and Hoff

Widrow B. and Hoff, "Adaptive switching circuits", IRE WESCON Convention Record, 96-104, 1960.


Reference for Wiener (1949)

 Wiener, N.

Extrapolation, Interpolation and Smoothing of Stationary Time Series, MIT Press, Cambridge, MA, 1949.


Reference for Zemel

 Zemel, R.

Minimum Description Length Framework for Unsupervised Learning, Ph.D. Thesis, U. of Toronto, 1993.


 Relationship Between DFS, FT, DFT, LT and Z

We recommend that a digital signal processing book (Oppenheim) be consulted for a more in-depth comparison of the transforms. Here we would just like to relate all the signal processing transfers, because this is hardly ever done, and when looked from the point of view of projection, they can be all integrated in the same framework.

Let us first enumerate all the transforms with their applications:

  • Laplace transforms (LT), utilized in the study of linear systems in continuous time

  • Z transforms (ZT), utilized in the study of discrete time linear systems

  • Fourier transforms (FT), applied in the study of continuous time signals

  • Fourier series (FS), applied in the study of continuous time periodic signals

  • Discrete Fourier transforms (DFT), applied in the study of discrete time signals

  • Discrete Fourier Series (DFS), applied in the study of discrete time periodic signals

  • Fast Fourier Transform (FFT), just an efficient algorithn to compute the DFS and the others Fourier transforms in an interval

These transforms are normally taught in different courses, so they are interpreted as being different, but in fact they are all related to each other, as shown in the following Figure.

Here we start with the Laplace transform F(s), which is defined as

where s is the complex variable s = [image] + j [image] , C is a contour in the region of convergence of the function. Hence F(s) is a complex function of a complex variable.

If we digitize the time variable (t = nT) to obtain a discrete signal, we obtain the Z transform F(z)

where z is the complex variable, [image] and C is a contour in the region of convergence of the function. Hence F(z) is also a complex function of a continuous complex variable. Due to the relation between s and z we can consider that the j [image] axis in the s domain is mapped onto the unit circle in the z plane.

The Laplace and Z transforms are used in the study of linear systems, both the transient and the steady state responses. But the Laplace transform is applicable to continuous systems, while the Z transform is applicable only to discrete systems.

Now let us talk about Fourier analysis which includes all of the other transforms. The Fourier transform F( [image]) of f(t), t [image] R is defined as

and it is also a complex function of a complex variable. If we restrict s = j [image] the Laplace transforms becomes the Fourier transform: that is, the Fourier transform is the intersection of the Laplace transform with the plane s = j [image] , so the Fourier transform only partially analyzes linear systems (their steady state response).

The Fourier series F(k) of f(t), t [image] [0,T], is defined as

so it is also a complex function of a complex variable. But notice now that the function f(t) is restricted to the interval [0,T] and the transform is discrete, F(k) but with an infinite number of coefficients. The Fourier series can be used to study periodic signals, since for these signals it is sufficient to know their properties during one period.

Now we digitize our signal f(t) to obtain the discrete transforms. The discrete Fourier transform F( [image]) of f(nT) is defined as

This is also a complex function of a complex variable. Notice that since the time variable was discretized, F( [image]) is computed through a sum. The discrete Fourier transform is related to the Z transform by a restriction on the independent variable. In fact, the DFT is the intersection of the Z transform with a cylinder of radius 1, so it is also only applicable for studying the steady-state response of linear systems.

Finally we define the discrete Fourier series of f(nt) [image] [0,N [image] 1] as

Note that this transform can be thought as a sampling of the discrete Fourier transform at N equally spaced points on the unit circle. It is also equivalent to the Fourier series if we sample the time variable. The DFS assumes both periodic time and frequency signals.

The fast Fourier transform is simply a fast algorithm to compute the DFS, but since the DFS is a sampling of the DFT, the FFT can also be used to estimate the DFT. Moreover, it can also estimate the FT (and likewise for the FS) of the continuous signal provided we sample it according to the Nyquist theorem, so the FFT is a general tool in Fourier analysis.

From this presentation it is obvious that all these transforms are related. Knowing the property of any one of them is sufficient if the appropriate modifications to the domains are observed. Another important observation is that all these transforms are specific instances of the projection theorem on a different, but related set of bases:

LT bases are the continuous variable [image].

ZT bases are the continuous variable [image].

FT bases are the continuous variable [image].

These three continuous transforms must be studied in Hilbert spaces, since the variables are continuous (infinite dimension spaces).

The FS bases are the variables [image].

The DFT bases are the variables [image].

The DFS bases are the variables [image].

These three transforms can be studied in vector spaces, as we present in this book. As we can immediately see, all of these bases belong to the complex exponential family, either discrete or continuous and also purely complex or with real or complex parts. These different bases are what provides the specific properties of each of the decompositions.

Return to text


 RLS Algorithm

The recursive least squares (RLS) algorithm is an on-line implementation of Newton's method that we discussed in Chapter 1. The RLS algorithm utilizes a set of weights that is always the optimum given the data, unlike the LMS algorithm. If we use the analytic approach to solve the Wiener-Hopf equation, the RLS algorithm, although iterative, provides exactly the same coefficient values as the analytic solution (for the same initial conditions) at the end of the data sequence. The RLS algorithm is also appealing since there is no step size. The problem with the RLS algorithm is that it is computationally more demanding than the LMS (complexity of O(D²)). See Haykin. Here we provide only a very brief explanation of the algorithm.

Let us assume that the input of the linear combiner is x(n) and the weights are w(n). The estimation of the autocorrelation function at time n in denoted by R(n). The RLS algorithm specifies that the weights should be adapted according to

where

and

where

We start computing the gain vector k(n) by using the previous estimate of the inverse of R [i.e. P(n-1)] and the present input vector x(n). Once we have k(n) we can update the weights with the present input x(n) and also reestimate the new P(n). Notice that these formulas are recursive, always using the previous available values for the estimates. The initialization is normally done with w(0) = 0 and P(0) = [image] I where [image] =1/ [image] ² is the power of the input.

The computational cost of the algorithm is proportional to O(D²), but notice that we are using second-order information (correlation function information) to update the weights. The convergence is much faster and the final values more precise. The RLS is an on-line implementation of Newton's method.

The RLS has problems in nonstationary environments, since the gain k(n) tends to zero when the algorithm converges. If the properties of the data change, the algorithm cannot update the weights since k(n) is zero (or very small). This is the reason why we normally apply RLS with an exponential decaying window on the estimate of R(n). See Haykin.

Return to text


 Scales in the Unit Circle

The unit circle is the locus of the points [image]. The frequency response is computed when z varies along the unit circle from 0 to 2 [image]. But there is another scale imposed on the unit circle, the frequencies that correspond to the analog-world frequencies.

If the analog frequencies f are normalized by the sampling frequency fs (called the normalized frequency) they also map from 0 to 2 [image]. The highest analog frequency corresponds to ½ the sampling frequency and it appears midway between 0 and 2 [image], i.e. at [image] [image] or z = -1. Zero analog frequency appears at z = 1, and the analog frequencies for the negative part of the spectrum appear between [image] and 2 [image]. The index k of the DFS is measured in normalized frequency. The following Table shows some key frequencies and z values.

norm. freq.
Angles
z values
0
0
  1. 0j
fs/4
90
0+j
fs/2
180
-1+0j
3fs/4
270
0-j
fs
360
  1. 0j

We can also conclude that [image] is periodic with 2 [image] /T, as we would expect. The linear combiner transfer function (in fact, the transfer function of any FIR) has only zeros (the poles at z = 0 are not counted since they represent delays and do not affect the frequency response, i.e. z = 0 is a symmetry point for the unit circle). The transfer function of IIR filters in general has poles and zeros that affect the frequency response.

It should be clear from the study of the first-order recurrent system that a stable system has poles inside the unit circle. Poles outside the unit circle have a modulus greater than 1, which correspond in the time domain to growing exponentials that violate our definition of stability.

Return to text


 Second Order Systems

The system of equation

which we encountered before, is a second-order system. The following Figure shows the block diagram.

Its transfer function is

which shows that the system has a double zero at z = 0 and a pair of poles at

We see that if the square root is positive, both poles are on the real axis. However, for b negative and |4b|>|a²|, the poles have a nonzero imaginary part. In this case they have to appear in complex conjugate pairs if the coefficients a and b are restricted to be real. When the poles are real and positive, the impulse response of this system decays monotonically to zero, and it implements a lowpass filter. When the poles are real and negative, we have a highpass filter (the impulse response alternates exponentially to zero). When the poles are complex, this system implements a bandpass filter, also called a resonator.

Previously, we gave the stability conditions in terms of the impulse response. A much more convenient (and equivalent) formulation is to limit the location of the poles of the system. We can show that the poles of stable causal systems have to be inside the unit circle. Zeros can be placed anywhere.

There is an alternative way to write the resonator equation that is very convenient for finding the location of the poles when they are complex conjugates. We can show that if the poles of the resonator are [image],

So by equating a, b to these values, we can immediately find where the poles are, since r is the magnitude of the poles and [image] the angle from the positive real axis.

Return to text


Separation Surfaces of the Sigmoid PEs

 Decision Surfaces of the Sigmoid PEs

Thinking of the separation surfaces built by ANNs made of sigmoidal PEs as an intersection of hyperplanes is sometimes a crude approximation. Sigmoidal PEs create ridge functions that are functions limited between 0 and 1. The weights and the bias still control the location and orientation of the ridge. Near the ridge region the function is approximately linear, but it deviates from linearity outside the region. The intersection of ridge functions will produce very complex curves. Since the network can control the separation surface by the size of the weights, ANNs can build curved separation surfaces instead of piecewise linear ones, for those problems that require them.

Return to text


 Shapes of 2-D Discriminants

Diagonal Covariance matrix

If the two variables are uncorrelated and of the same variance, then the covariance matrix is diagonal

In this case the Mahalanobis distance defaults to the Euclidean distance:

and the classifier is called a minimum distance classifier. The interesting thing is that the discriminant function for this case defaults to a linear function:

where [image] and [image] since the quadratic term is common to both classes and does not affect the shape of the discriminant. For this case the samples define circular clusters (hyperspherical in multidimensions).

Equal Covariance Matrix

The case of equal covariance matrices for each class ( [image] i = [image]) is still fairly simple. In fact, the discriminant is still linear, but now the weights and bias of gi(x) in the previous equation are given by [image] and [image], which means that each class is a ellipsoidal cluster of equal size and shape. Figure 2-6 shows both cases. Notice that the decision surfaces are both linear functions.

Arbitrary Covariances

This is the most general case, and the general form of the discriminant function of Eq.2.8 must be used. We can see that this discriminant function is quadratic in x:

, [image] [image].

The decision region is either a line, circle, ellipse and parabola, depending upon the shape of the individual clusters and their relative position (Figure 2-7).

These three cases illustrate our previous statement that the covariance matrix is exceptionally important in the definition of the shape (and placement) of the discriminant function. Note that the discriminant function changes from a hyperplane to a quadratic surface, depending on the covariance matrix of each class.

A classifier built from linear discriminant functions (called a linear classifier) exploits only differences in means among different classes, while the quadratic classifier exploits not only the mean difference but also the difference in "shape" of the data clusters. Hence, if we have two classes with the same mean, the linear classifier will always give very poor results. However, a quadratic classifier may perform better, as long as the shapes of the two data clusters are different. In fact, notice that for Gaussian-distributed classes the optimal classifier is a quadratic classifier given by Eq. 2.8. There is no need to find more sophisticated (higher-order) discriminant functions.

The improvement in performance between the linear and the quadratic discriminants comes at a computational cost. In fact, the number of parameters estimated for the case of an arbitrary covariance matrix in 2-D is seven per class for the quadratic (and increases as the square of the dimension due to W), while it is simply three for the linear case. Quadratic discriminants require more data samples for the reliable estimation of their parameters.

Return to text


Shapes of 2-D Discriminants-2

 Shapes of 2-D Discriminants

Diagonal Covariance matrix

If the two variables are uncorrelated and of the same variance, then the covariance matrix is diagonal

In this case the Mahalanobis distance defaults to the Euclidean distance:

and the classifier is called a minimum distance classifier. The interesting thing is that the discriminant function for this case defaults to a linear function:

where [image] and [image] since the quadratic term is common to both classes and does not affect the shape of the discriminant. For this case the samples define circular clusters (hyperspherical in multidimensions).

Equal Covariance Matrix

The case of equal covariance matrices for each class ( [image] i = [image]) is still fairly simple. In fact, the discriminant is still linear, but now the weights and bias of gi(x) in the previous equation are given by [image] and [image], which means that each class is a ellipsoidal cluster of equal size and shape. Figure 2-6 shows both cases. Notice that the decision surfaces are both linear functions.

Arbitrary Covariances

This is the most general case, and the general form of the discriminant function of Eq.2.8 must be used. We can see that this discriminant function is quadratic in x:

, [image] [image].

The decision region is either a line, circle, ellipse and parabola, depending upon the shape of the individual clusters and their relative position (Figure 2-7).

These three cases illustrate our previous statement that the covariance matrix is exceptionally important in the definition of the shape (and placement) of the discriminant function. Note that the discriminant function changes from a hyperplane to a quadratic surface, depending on the covariance matrix of each class.

A classifier built from linear discriminant functions (called a linear classifier) exploits only differences in means among different classes, while the quadratic classifier exploits not only the mean difference but also the difference in "shape" of the data clusters. Hence, if we have two classes with the same mean, the linear classifier will always give very poor results. However, a quadratic classifier may perform better, as long as the shapes of the two data clusters are different. In fact, notice that for Gaussian-distributed classes the optimal classifier is a quadratic classifier given by Eq. 2.8. There is no need to find more sophisticated (higher-order) discriminant functions.

The improvement in performance between the linear and the quadratic discriminants comes at a computational cost. In fact, the number of parameters estimated for the case of an arbitrary covariance matrix in 2-D is seven per class for the quadratic (and increases as the square of the dimension due to W), while it is simply three for the linear case. Quadratic discriminants require more data samples for the reliable estimation of their parameters.

Return to text


 Short-Term versus Long-Term Memory

What is the computational difference between short and long-term memory? This is a critical and difficult question that has not been properly studied. Static mappers such as the MLP have long-term memory, since the information utilized during system training is converted into weight values using the learning rules. Static mappers contain a repository of past information that we associate with memory. However, they are unable to differentiate time relationships, because the information collected through time is collapsed in the weight values.

Dynamic systems are different. Because they have short-term memory structures or recurrent connections they are sensitive to the sequence of presentation of information. They also have weights so they also have long-term memory. But unlike static mappers these weights code differences within the time window of observation.

Return to text


 Sinc Decomposition

The formulas previously derived can be used to find the decomposition obtained when the basis are sinc functions. We would like to write

The bases are

Now applying Eq 5.7 the weights become

which means that the weights become exactly the value of the function at the point (i.e. the sample value). This explains Figure 5-4.

Return to text


 Size of Feature Space

The conventional way of thinking about features in pattern recognition has been as a method of reducing the dimensionality of the input. A feature is a characteristic of the input data that preserves discriminability, for instance, color, or edges in an image. Features are traditionally obtained with projections onto a sub-space, that is, features have smaller dimensionality than the input. For 50 years the quest in pattern recognition has been to find small-dimensional projections that preserve the most information about the input. We realize that a projection onto a sub-space reduces the discriminability somewhat, but experience has shown that for many problems a handful of small-dimensional projections produce workable classifiers.

The motivation for performing subspace projections is related to the "curse of dimensionality" we already mentioned. If we want to build the optimal Bayes' classifier in a high dimensional space, we have to estimate too many parameters with the available data. One way to conquer this difficulty is to project the data to a smaller-dimensional space (feature extraction) and there develop the optimal classifier. The difficulty has been in choosing the features and their number. Moreover, if the data is hard to classify in the original space, very likely it will also be difficult to classify in the smaller-dimensional feature space. Nevertheless, this has been the traditional way to think about feature spaces.

The view we expressed using Cover's theorem to go to a higher-dimensional feature space has been only a curiosity in pattern recognition until the recent work of Vapnik with support vector machines. High-dimensional spaces produce sparse data clusters, that is, no matter how many data samples we might have, if the size of the space is sufficiently large, there will always lots of room among the data clusters. This means that the classification problem is potentially linearly separable, so a very simple classifier can do the job. The interesting thing is that good performance in the test set can still be controlled by placing discriminants halfway between the boundary samples. We will discuss this in more detail later.

Return to text


Static versus Dynamic Modelling

 Static versus Dynamic Modeling

It is important to understand the source of the difference between the two types of systems. As an example, suppose that we have two clusters of points with the same mean and variance but that belong to two different classes. Training an MLP to distinguish these two classes is virtually impossible. However, let us assume that the two classes have different time histories, as were created by two sinusoidal signal sources of different frequencies. It is obvious that we may separate the two classes with a linear combiner, but we will never be able to separate them with an MLP. This is an extreme case to illustrate the difference between static versus dynamic modeling. In static modeling we are using statistical properties of the data clusters to distinguish them. In time phenomena, time imposes a structure in the input space (see Chapter 9) that may be used to separate data clusters with overlapping statistics; in other words, the sequence in which the points are visited is different, and this difference can be used for separation.

In many practical cases we do not know if there is a time structure underlying our data, and so it is not clear whether dynamic modeling will help to improve performance. The trouble (as we will see in Example 10.1) is that the window in time where we seek the information is crucial for good performance.

Return to text


 Statistical Independence

Statistical independence between two variables x1 and x2 occurs when the joint pdf (f(x1,x2)) is equal to the dot product of the marginal pdfs (f(x1)f(x2)).


 System Identification Fundamentals

Linear system identification is such an important problem that we have to spend some time analyzing it. Perhaps it is best to look at system identification from the point of view of function approximation. As a form of function approximation, the three important problems in system identification are:

  • Defining the bases

  • Computing the parameters

  • Selecting the order of the approximation

As we have seen with the linear combiner, we want to work with a linear model and adapt its parameters. Hence the topology utilized in the model system dictates the type of bases we will be using. The parameters of the model have to be adapted, and we have to make decisions regarding the order of the model.

When a linear combiner is selected, the bases are the signal and its past samples as it is clearly seen from Eq. 9.1. Since the model is a weighted sum of the input alone, this type of model is called moving average (MA). Once the topology is set, the only difference among MA models are their parameters and their order. The order can be determined as we did in Chapter 5 by the Akaike or Rissanen (MDL) criteria. The parameters can be adapted with least squares or, if we prefer, with LMS or RLS.

Experience has shown that if the plant belongs to the same family implemented by the model system, results can be very good with very small orders. This can be understood if we think in terms of eigendecompositions. If the plant is MA, then our MA model will be performing an eigendecomposition, which is guaranteed to be very efficient. Unfortunately, not all linear systems are MA models. When the order is very large, we can show that MA models can still approximate any other linear system at the expense of a large number of coefficients.

The most general linear system is given by

which has an output given by a weighted sum of previous (and present) inputs, and previous outputs. Such a system is called autoregressive moving average (ARMA) model. ARMA models are very powerful but very difficult to work with in practice, due to three major reasons: they may become unstable during adaptation, the algorithms to update their parameters are more computationally demanding, and search can be caught in local minima (the performance surface is no longer convex as in the MA). They are therefore less utilized then the MA models, but the appeal is that ARMA modeling is an eigendecompositon for linear systems. Note that the ARMA model decomposes the output in a space created by the previous inputs and its own past states.

When we discussed linear systems, we presented systems that have a response dependent only upon the system output

which is called an autoregressive (AR) model. Note that the AR model is an extension of the first order recurrent system that we studied in Section 8.5.1. AR models have very nice characteristics: First, they are very flexible because they can approximate any other linear system well. Second, they correspond to many physical processes, so they have a large chance of providing eigendecompositions. Third, we can often use a trick to adapt an AR model that avoids the problem of possible instability: we use the prediction framework and adapt an MA model. Once this MA model is adapted, we work with its inverse. Therefore we can find the optimal parameters using the normal equations (Eq. 9.5). Fourth, the AR model is linked to a fundamental theory on the decomposition of time series (Wold decomposition, see Haykin) The workhorse of linear system identification is the AR model.

Return to text


 System Identification versus Modeling

Not all the problems of interest can be formulated in terms of an external, input-output constraint, but the class of these problems is very large indeed, ranging from system identification, prediction, classification, and so on. For instance, when we are predicting the stock market, we are not interested in preserving the individual components of the economy in our network; we are simply interested in a small prediction error.

On the other hand, there are problems that require more than an accurate input-output relation, such as modeling. In modeling we are interested not only in small input-output errors, but also in preserving some (or all) of the underlying features of the phenomenon in which we are interested. Examples of modeling appear in the physical and biological sciences and in some engineering areas.

Return to text


Takens Embedding Theorem

 Takens' Embedding Theorem

Takens proved that some properties (dynamic invariants) of the dynamic system that produced the time series can be preserved if we transform the time series into a sufficiently large reconstruction space: N, the size of the space, should be at least 2M +1, where M is the number of degrees of freedom of the dynamic system. He proposed that the point coordinates of the reconstructed trajectory be read as N-tuples of the time series. For instance, for a reconstruction in a 3-D space, consecutive triples of the time series should be read at a time, with the first time series sample being the x coordinate of the first point in the reconstruction space, the second sample the y coordinate, the third sample the z coordinate, the fourth sample being the x coordinate of the second point, and so on. (see the following figure). When the points of the reconstructed space are connected, a trajectory is found from which properties of the original dynamic system can be estimated (such as dimension and Lyapunov exponents). This means that some of the important dynamical properties of the original signal are preserved in this reconstruction.

A powerful aspect of this theory is that the Takens embedding is naturally implemented by generalized feedforward structures. In fact, a delay line with three taps provides x(n), x(n-1) and x(n-2), which are exactly the coordinates needed to reconstruct the points of the trajectory in the reconstruction space.

This alternative view of memory structures is very enlightening, because we can see that the PE that receives the output of the memory filter is effectively working with the trajectory in state space, that is, it is accessing the dynamical information. At least as a first approximation this methodology helps us set the size N of the memory filter as twice the size of the dimension of the dynamical system that produced the time series.

The dimension of the dynamic system can be estimated from the time series using for instance the correlation dimension algorithm Abarbanel. But unfortunately, there is still a free delay parameter [image] in Takens' embedding theorem that is unspecified. Its function is basically to uncorrelate the samples in time as much as possible. If [image] is not set right, the training can be very slow. We can experiment with the selection of [image] by changing the number of delays between the tap outputs.

Note that Takens' embedding theorem is a mathematical formulation of the ideas of signal space and reconstruction space that were presented in Chapter 9. It shows that the signal trajectory can preserve information about the dynamics of the system that produced the time series.

Return to text


Takens Embedding Theorem (2)

 Takens' Embedding Theorem

Takens proved that some properties (dynamic invariants) of the dynamic system that produced the time series can be preserved if we transform the time series into a sufficiently large reconstruction space: N, the size of the space, should be at least 2M +1, where M is the number of degrees of freedom of the dynamic system. He proposed that the point coordinates of the reconstructed trajectory be read as N-tuples of the time series. For instance, for a reconstruction in a 3-D space, consecutive triples of the time series should be read at a time, with the first time series sample being the x coordinate of the first point in the reconstruction space, the second sample the y coordinate, the third sample the z coordinate, the fourth sample being the x coordinate of the second point, and so on. (see the following figure). When the points of the reconstructed space are connected, a trajectory is found from which properties of the original dynamic system can be estimated (such as dimension and Lyapunov exponents). This means that some of the important dynamical properties of the original signal are preserved in this reconstruction.

A powerful aspect of this theory is that the Takens embedding is naturally implemented by generalized feedforward structures. In fact, a delay line with three taps provides x(n), x(n-1) and x(n-2), which are exactly the coordinates needed to reconstruct the points of the trajectory in the reconstruction space.

This alternative view of memory structures is very enlightening, because we can see that the PE that receives the output of the memory filter is effectively working with the trajectory in state space, that is, it is accessing the dynamical information. At least as a first approximation this methodology helps us set the size N of the memory filter as twice the size of the dimension of the dynamical system that produced the time series.

The dimension of the dynamic system can be estimated from the time series using for instance the correlation dimension algorithm Abarbanel. But unfortunately, there is still a free delay parameter [image] in Takens' embedding theorem that is unspecified. Its function is basically to uncorrelate the samples in time as much as possible. If [image] is not set right, the training can be very slow. We can experiment with the selection of [image] by changing the number of delays between the tap outputs.

Note that Takens' embedding theorem is a mathematical formulation of the ideas of signal space and reconstruction space that were presented in Chapter 9. It shows that the signal trajectory can preserve information about the dynamics of the system that produced the time series.

Return to text


 Training Alternate Memories

To train TLFNs with Laguerre PEs, we need to specify two maps, the state equations,

the state gradients (i.e., the backpropagated errors) and the formula to update the weights of the Laguerre PE. Since the Laguerre PE is just a minor modification to the gamma PE, we present only the results below.

for all taps after the first (k >1). For the first tap the equations read

where [image] and it is the implementation of the front-end lowpass filter. X(n) and [image] (n) represent the joint input to the PE and its dual respectively. The weight update using straight gradient descent is

This is what is needed to train networks with a Laguerre memory PE.

To adapt the gamma II PE in a TLFN the following two local maps are needed:

The weight update equations are

With these maps we can train ANY topology that includes these memory PEs with BPTT.

Return to text


 Training Focused TLFNs

The goal is to be able to adapt the feedback parameter [image] of a locally recurrent memory (memory at the input layer) in a focused TLFN with an algorithm that can be integrated with static backpropagation. Let us treat the case of the gamma memory. The gamma memory forward equation for tap k is

(11B.4)

Let us take the derivative of y with respect to [image] using the direct differentiation method:

(11B.5)

The direct differentiation method thus provides an on-line approximation (for small step sizes) to adapt the [image] parameter. It is necessary to integrate this equation with the information available in a backpropagation environment.

The goal is to adapt [image] using static backpropagation. Let us assume that the gamma memory PE is connected to several (p) hidden PEs of the MLP in the focused TLFN. Then we can write

(11B.6)

where yi(n) are the outputs of the gamma memory PE, the sum is extended to p, and J is the output criterion. During a backpropagation sweep the errors that are propagated from the output through the dual system up to the gamma memory PE are exactly [image]. Thus [image] can be adapted using a mix of static backpropagation and RTRL. According to Eq. 11B.6, the dual system provides [image] which is multiplied by Eq. 11B.5 and then summed across the gamma PEs. Eq. 11B.5 can be thought of as a new "dual" of the gamma memory PE in the static backpropagation framework, since it instantaneously computes the sensitivities.

Notice that the implementation of Eq. 11B.5 (the instantaneous dual gamma PE) must have memory to store each of the previous values of the sensitivities [image]. What we gain is a sample-by-sample update algorithm that does not require the storage of the states of the feedforward components since it is using static backpropagation. Since we have a single adaptive parameter, the implementation of the RTRL part (Eq. 11B.5) is still efficient. The following figure shows a block diagram of the steps involved in the adaptation of [image]. Notice that these equations are valid only when the gamma PE is at the input: when it is connected to a hidden layer all the gradients to the left of the gamma memory PE become time dependent.

Return to text


 Training the Gamma Filter

The gamma filter was presented in Chapter 10 as an alternative to the Wiener filter. It is basically a gamma memory followed by an adder. Since the gamma memory is locally recurrent, the gamma filter cannot be trained with static backpropagation.

The gamma filter is defined as

(11B.1)

where x0(n) = x(n) is the input signal and y(n) is the output. The weights and the gamma parameter [image] is adapted using the gradients of the cost function given by Eq.11.4. We can use RTRL and get immediately,

(11B.2)

where

This gradient can be computed on-line by differentiating the gamma filter equation (Eq.11B.1), yielding

(11B.3)

Using the ideas of RTRL, for small step size [image] we can drop the summation over time in Eq. 47. Notice that these equations are local in time, and the complexity to adapt a filter of order k is O(K). NeuroSolutions does not implement this procedure directly: BPTT must be used to adapt the gamma filter.

Return to text


 Transfer Function versus Frequency Response

We now have two transform domain representations for a linear system: the transfer function and the frequency response. How do they differ? In fact, they are intrinsically related to each other. The frequency response is the transfer function computed for a specific locus (set of points) in the z domain. This locus is the unit circle. If we take a look at the definition of the z variable and make a = 0, we obtain the curve [image] , which defines the unit circle. Along this locus we are finding how the transfer function of the system changes when only the variable w changes.

This is such an important case, however, that people decided to give it a name: the frequency response. The frequency response tells us how the system is responding to phasors of different frequencies (the different [image] values). This is called the steady-state response. However, the knowledge of the frequency response only partially describes the system, since we still do not know the response of the system to points that do not belong to the unit circle.

An important case for which the frequency response does not provide sufficient information is the response to transients, such as when the input is turned on or off or changes very rapidly. This response is sometimes called the transient response. In such cases we need to study the transfer function, because it is not enough to know the frequency response. There may be many different linear systems with the same frequency response (the same steady-state response to phasors) but with markedly different transient responses.

Return to text


 Types of Discrete Time Linear Systems

An interesting property of the most common discrete linear systems (shift-invariant) is that their output can be expressed as a difference equation involving the present input and the past input and output values:

(8B.6)

where P is the number of previous outputs used, Q is the number of previous inputs, and bp and aq are the free parameters of the system that combine the previous outputs and the inputs, respectively. This equation is the most general difference equation that will be studied here and corresponds to a system called an autoregressive moving average system (ARMA) which is shown in the following Figure.

If all the bp are zero, the system output is a weighted combination of past inputs only, and it is called a moving-average (MA) system, finite impulse response (FIR), or linear combiner. If all the aq (q > 0) are zero (except the first), the system is called an autoregressive system (AR). Both the ARMA and the AR belong to the class of infinite impulse response (IIR) systems. We study here mostly the linear combiners, although most of the theory developed also applies to the IIR case.

There is an important way of rewriting these equations using the delay operator. If we apply the Z transforms to Eq. 8B.6 we get

(8B.7)

where X(z) and Y(z) are respectively the Z transforms of the input and the output. The transfer function H is by definition the ratio of Y(z) over X(z),

(8B.8)

From this equation we see that the zeros of the transfer function are controlled by the location of the zeros of the numerator polynomial, while the poles of H(z) are the zeros of the denominator polynomial. Therefore, MA systems have only zeros, while AR systems have only poles, and ARMA systems have both zeros and poles. When we compute the transfer function along the unit circle, we obtain the frequency response of the system:

(8B.9)

The frequency response is useful for studying the steady state of the system, while the transfer function is required to study the transient response and stability of the discrete system.

Return to text


 Understanding Bayes Rule

There are two types of probabilities associated with an event: the a priori and the a posteriori probabilities. Suppose that the event is "x belongs to class c1". The a priori probability is evaluated prior to any measurements. Without a value for x, then the a priori probability is defined by the relative frequency of the classes, which is P(c1). However, we can also estimate the probability of the event after making some measurements. For a given x we can ask what is the probability that x belongs to class c1, denoted by P(c1|x). This is the a posteriori probability.

According to statistical pattern recognition, what matters for classification are the a posteriori probabilities P(ci|x), but they are generally unknown. Bayes' rule provides a way to estimate the a posteriori probabilities. In fact, Eq. 2.2 tells us that we can compute the a posteriori probability by multiplying the a priori probability for the class, P(ci) with the likelihood that the data was produced by class i. The likelihood p(x|ci) is the conditional of the data given the class, that is, if the class is ci what is the likelihood that the sample x is produced by the class? The likelihood can be estimated from the data by assuming a probability density function (pdf). Normally the pdf is the Gaussian distribution. Using Bayes rule we thus has a way to estimate a posteriori probabilities from data.

Return to text


 Universal Approximation of MLPs

We only outline here the proof for the universal mapping characteristics of the MLP. We start by extending the Weierstrass theorem to metric spaces (the Stone- Weierstrass theorem). Polynomials can be extended to metric spaces by defining the concept of an algebra. A family of functions F that map the metric space V to the real line is an algebra if their elements have the properties

where [image] and [image] are real numbers.

The Stone-Weierstrass theorem can be enunciated in the following way. Let V be a metric space and F an algebra that maps V into the reals. If there is a function [image] for which [image] and [image] in V, then F is dense in the mapping of V into the reals. The idea of dense is the same as arbitrarily close approximation, as stated in the Weierstrass theorem.

This theorem has been used to show the universal mapping capabilities of the MLP. In fact, the function f(v) can be expanded in a special type of "Fourier series" with squashing cosine functions,

where

The nonlinearity of the MLP belongs to this family of squashing functions. Notice that f(v) is exactly the output of the one-hidden-layer MLP with a cosine nonlinearity.

Return to text


 Variance

Data collected from experiments is normally very complex and difficult to describe by few parameters. The mean and the variance are statistical descriptors of data clusters that are normally utilized in such cases.

The mean of N samples is defined as

A physical interpretation for the mean is the center of mass of a body made up of samples of the same mass. It is the first moment of the probability density function (pdf).

We can have very different data distributions with the same mean, so the mean is not a powerful descriptor. Another descriptor very often used is the variance, which is defined as

The variance is the second moment around the mean, and it measures the dispersion of samples around the mean. The square root of the variance is called the standard deviation. Mean and variance are much better descriptors of data clusters. In fact, they define univocally Gaussian distributions, which are very good models for many real-world phenomena.

Return to text


Vector Interpretation of the Separation Surface

 Vector Interpretation of the Decision Surface

A vector interpretation of Eq. 3.4 is very useful. We are going to show that the weights, when interpreted as the coordinates of a vector, are perpendicular to the separation surface. Assume that b = 0 for simplicity. Note that w1x1 + w2x2 can be thought of as the dot product of two vectors of coordinates (w1,w2) and (x1,x2) respectively.

Let us consider the weights (1, 2) as the end point of a vector v drawn from the origin. The points of coordinates (x1, x2 ) in Eq.3.4 can be interpreted as the end points of another vector g (drawn from the origin) that exists on a line. To satisfy g(x1,x2) = 0, v and g have to be perpendicular since their dot product is zero. Hence, the linear decision surface g(x1,x2) has to be perpendicular to the weight vector v.

Remember that the PE weights are the coefficients of the discriminant function. Therefore the weights of the PE indicate the normal direction to the separation surface in the input space.

Return to text


 Vector Space Interpretation of TLFNs

The vector space interpretation of a memory PE explains the adaptation of the TLFN weights as a local approximation of the desired response by a weighted sum of memory traces yk(n). These signals are the bases of the local projection space. Let us study the case of a gamma memory PE inside the neural network (following figure).

The gamma delay line of size D produces a basis that represents a memory space of size D, but the bases are no longer a direct use of the past values of the input, as in the ideal delay line. The bases are already a linear projection of the input trajectory (convolution with the gamma kernel). When the parameter [image] adapts to minimize the output MSE, the projections rotate, and the span of the projection space changes as a consequence of the different time constants (the effective length of the impulse response also changes). The relative angle among the gamma bases vectors does not change with [image]. Hence, a decrease in the error must be associated with a decrease in the relative position between the desired signal and the projection space (following figure).

The recursive parameter in the gamma structure changes the span of the memory space with respect to the input signal (which can be visualized as a relative rotation and change in dimensionality between the input signal and the projection space). In terms of time domain analysis, the recursive parameter is finding the length of the time window (the memory depth) containing the relevant information to decrease the output MSE.

The gamma PE is more powerful than the tap-delay-line PE, but the search for the best orientation of the projection space is not trivial. We can show (deVries and Principe) that the performance surface has many local minima. Most of the time any of the local minima provide a better solution than the linear combiner of the same order, so there is a net gain in using the gamma memory.

Return to Text


 Weierstrass Theorem

Weierstrass proved the following important theorem: Let S[a, b] be the space of continuous, real-valued functions defined in the real segment [a, b]. If [image], then there exists a polynomial [image] with real coefficients [image] for which

for [image] [image] > 0 and [image].

In words, this says that any function can be approximated arbitrarily well (i.e., with an error as small as we want) by a sufficiently large-order polynomial. The Weierstrass theorem is the starting point for most proofs of the universal mapping properties of the MLP.

Return to text


Who Are Cauchy and Weierstrass?

 Cauchy and Weierstrass

Augustin Cauchy (1789-1857) and Karl Weierstrass (1815-1897) were the fathers of calculus. They captured the idea of the limit in a precise mathematical way and opened up new horizons in approximation theory.


Who are McCulloch and Pitts?

 McCulloch and Pitts

A very influential duo of researchers (Warren McCulloch was a neurobiologist and Walter Pitts a computer scientist) who proposed a model of the neuron as a system for logic computation. See Brain Theory Vol 1, Ed.. Shaw and Palm, World Scientific, 1988.


Who is Gauss

 Gauss

Karl Friedrich Gauss (1777-1855) was a mathematical genius who proposed the use of least squares to solve sets of linear equations. He realized that a quadratic equation was obtained in optimization problems involving Gaussian distribution models (after taking the logarithm), which leads to an easy solution for the optimum.


Who is Grossberg?

 Grossberg, Stephen

Stephen Grossberg is a very influential neural network researcher who has developed through the years a very sophisticated theory of biological information processing. In particular he was one of the initiators of the self-organizing networks. His adaptive resonant theory (ART) network is a good example of self-organization. However, his models are normally described in continuous time, using sophisticated mathematics, which are beyond the scope of this textbook. See Neural Networks and Natural Intelligence, MIT Press, 1988.


Who is John Hopfield?

 John Hopfield

John Hopfield is a theoretical biologist who studied the properties of a fully recurrent neural network using the ideas of dynamics, which lead to the powerful concept of computational energy . See the paper Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sc. USA, 79, 2554-2558, 1982.


Who is Kohonen?

 Kohonen, Teuvo

Teuvo Kohonen can be credited as the developer of practical self-organization algorithms. See Self-Organizing Maps, Springer Verlag, 1995.


Who is Minsky?

 Minsky

Marvin Minsky is one of the fathers of artificial intelligence who is credited with the downfall of neural networks in the late 1960s. This may be an exaggeration. In a very influential book written with S. Papert called Perceptrons, MIT Press, 1969, he developed the theory of perceptrons for predicate calculus and showed their limitation versus the Turing machine.


Who is Occam?

 Occam

William of Occam was a monk who lived in 14th century in England and enunciated a principle that has echoed across scientific circles for centuries: He said that a scientific model should favor simplicity, hence the name for the principle: Occam's razor (shave-off the fat in the model.)


Who is R. A. Fisher?

 R. A. Fisher

R. Fisher was a British statistician who in the late 1920's proposed the use of the maximum likelihood principle to solve the discriminant analysis problem (i.e., a rule to distinguish between sets of data).


Who is Rudolph Kalman?

 Rudolph Kalman

Invented the Kalman filter, is one of the most influencial electrical engineers alive. The Kalman filter is an optimal filter that solves the optimization in space state using the recursive minimum mean square estimation. See Haykin's book on Adaptive filter theory for a signal processing view of Kalman filters.


Who is Takens?

 Takens

a Dutch mathematician who formulated a very useful theorem showing that we can access the dynamic information of the system that produced the time series if the size of the reconstruction space is large enough.


Who is Widrow

 Widrow

Bernard Widrow was one of the first researchers who explored engineering applications of adaptive systems. We are going to hear a lot about him in this book.


 Why Nonlinear PEs?

The answer to this question is related to better performance and new computing power. Nonlinear systems may provide, for instance, a better fit to data than linear regression. In terms of the performance surface, this means that the global minimum achieved by the nonlinear network is lower than the minimum of the linear network. More important, nonlinear PEs may effect new types of computation such as classification, which cannot be done well with linear systems. There is a real need for nonlinear processing, in spite of the added difficulty of working with nonlinear structures.

It is instructive to use the last example and train a linear system and a nonlinear system with the same data to understand the difference between them. The M-P PE separates the two data clusters. What do you think is going to happen if we use a linear system to provide a desired response of 1s and 0s?

Let us think in terms of least squares. If a set of points in 2-D space is given and a desired response of 1s and 0s is provided, the linear system will do is regression between the input and the desired response. It will provide the best regression plane (this is a 2D input) between the 0 and 1 responses, given the values of the input samples.

Return to text