A Backpropagation Neural Network - notes on the program

We consider a Backpropagation Neural Network with one hidden layer, which is trained to encode patterns. During training the given 1-out-of-n patterns are presented to the input and the output layer and the weights of these layers are adapted based on the resulting output error. The goal is to adapt the weights such that the given input pattern is reproduced at the output layer. If this goal is reached, the hidden layer signals can be considered as an encoding of the given patterns.


The Unit Classes - Using JLayer Annotations

The three unit classes Input, Hidden and Output model the nodes of the input layer, the hidden layer or the output layer, respectively. We'll discuss them in turn.


Unit Class Input

Nodes of the Input class are only used to enter the patterns and present them to the hidden layer. Therefore, they contain only a layer field y of type Signal, which will be connected to all nodes of the hidden layer, and a layer method to set this field. Signal is a simple class that only contains a field val of type double. 

  @LayerUnit
  public class Input {
	
    @LayerField Signal y = new Signal(); // fanout signal
	
    @LayerMethod
    public void setInput(@LayerParam double input){
      y.val = input;
    }

  }

Unit Class Hidden

At the top of the Hidden class you find the elements to make the connections. There is the input vector x for collecting the input layer signals, the fanout signal y for the output signal of the node that is sent to the output layer, and the vector inErr for the error signals coming back from the output layer.

Below that are the so-called bias value b and the weight vector w. These are the parameters that determine a node's fanout signal and are adjusted during training. With the initSignals() layer method, these parameters are initialized randomly at the beginning. 

  @LayerUnit
  public class Hidden {
	
    @LayerField Signal   y = new Signal(); // fanout signal
    @LayerField Signal[] x;                // input vector
    @LayerField Signal[] inErr;            // incoming error signals
	
    Signal   b;                             // bias
    Signal[] w;                             // weight vector
	
    @LayerMethod
    public void initSignals(Random r) {
      b = new Signal();
      b.val = r.nextDouble();
      w = new Signal[x.length];
      for (int i = 0; i < x.length; i++) {
        w[i] = new Signal();
        w[i].val = r.nextDouble();
      }
    }
	
    @LayerMethod
    public void Forward() {
      y.val = sigmoid(dotProd(w, x) + b.val);
    }
	
    @LayerMethod
    public void Backward(double eta) {
      // compute delta
      double sum = 0.0;
      for (Signal s : inErr) { sum += s.val; }
      double delta = y.val * (1 - y.val) * sum;

      // adjust bias and weights
      b.val = b.val + eta * delta;
      for (int i = 0; i < w.length; i++) {
        w[i].val = w[i].val + eta * delta * x[i].val;
      }
    }

  }

The Forward() layer method calculates the node's signal using the so-called sigmoid function applied to the weighted sum of the input vector x plus the bias value b. And the Backward() layer method performs the training mentioned above. The adaptation of the parameters is based on the error values ​​of the output layer present in the vector inErr and a so-called learning rate eta, which is passed to the method as a parameter.


Unit Class Output

The Output class is similar to the Hidden class in many ways. The class has a fanout signal y, an input vector w and a bias value b, which have the same meaning as in the hidden class. Instead of a vector inErr for incoming error signals, there is a vector outErr for outgoing error signals, which result from a comparison of the current value of y with the value of a target signal t to be set externally.

The methods initSignals() and Forward() correspond to those in the Hidden class, the method setTarget() is used to set the target signal and method getError() can be used to read the error. 


  @LayerUnit
  public class Output {
	
    @LayerField Signal   y = new Signal(); // fanout signal
    @LayerField Signal[] x;                // input vector
    @LayerField Signal[] outErr;           // outgoing error signal
	
    Signal   b;                             // bias
    Signal[] w;                             // weight vector
    Signal   t;                             // target signal
    
    @LayerMethod
    public void initSignals(Random r) {
      y = new Signal();
      b = new Signal();
      b.val = r.nextDouble();
      w = new Signal[x.length];
      for (int i = 0; i < x.length; i++) {
        w[i] = new Signal();
        w[i].val = r.nextDouble();
      }
      t = new Signal();
    }
	
    @LayerMethod
    public void setTarget(@LayerParam double target){
      t.val = target;
    }
	
    @LayerMethod
    public double getError() {
      return Math.pow(t.val - y.val, 2);
    }
	
    @LayerMethod
    public void Forward() {
      y.val = sigmoid(dotProd(w, x) + b.val);
    }
	
    @LayerMethod
    public void Backward(double eta) {
      // compute delta
      double delta = y.val * (1 - y.val) * (t.val - y.val);
		
      // set outgoing error signals
      for (int i = 0; i < w.length; i++) {
        outErr[i].val = delta * w[i].val;
      }
		
      // adjust bias and weights
      b.val = b.val + eta * delta;
      for (int i = 0; i < w.length; i++) {
        w[i].val = w[i].val + eta * delta * x[i].val;
      }
    }

  }

The Backward() methods of the Output and Hidden classes only differ in that the auxiliary value delta depends on the deviation from the target t in the first case and on the sum of the incoming error signals in the second case.


Establishing Global Structure and Dynamics - Using the Generated Wrapper Classes

The EncoderNetwork class defines the global structure and dynamics of the backpropagation network. The parts of the code that depend on the JLayer framework are color coded below.

The simple fields at the top determine global network settings: patternSize is the size of the input and output layers, codeSize is the size of the hidden layer, randomSeed ist the starting value for the random number generator and eta is the learning rate during training.

Next, the three arrays inputArray, hiddenArray, outputArray and their wrappers are declared, followed by the definiton of relation full, which is used to connect or associate layers. Relation full is an implementation of the interface Relation from the JLayer Framework. The two dimensional array patternPool serves to store the 1-out-of-n patterns, the coding of which has to be learned.

Method createNetwork() creates the three layers, connects the input vectors x of the hiddenLayer / outputLayer to the fanout signals y of the inputLayer / hiddenLayer and then associates the vectors outErr of the outputLayer to the vectors inErr of the hiddenLayer. Note that since the linking vectors are constructed in ascending order with respect to the underlying indices, the system ensures that any outgoing error signal outErr[i] goes to the hidden node that gave the input y[i]. At last,the 1-out-of-n patterns are stored in the patternPool. 


  public class EncoderNetwork {
	
    int patternSize, codeSize;
    int randomSeed;
    double eta;
	
    Output[] outputArray;
    Hidden[] hiddenArray;
    Input[]  inputArray;
	
    Layer_Output_ outputLayer; 
    Layer_Hidden_ hiddenLayer; 
    Layer_Input_  inputLayer; 
	
    Relation full = new Relation() {
      @Override
      public boolean contains(int[] x, int[] y) {
        return true;
      }
    };
	
    Double[][] patternPool;
	
    void createNetwork() {
		
      inputArray  = new Input[patternSize];
      hiddenArray = new Hidden[codeSize];
      outputArray = new Output[patternSize];
		
      for (int i = 0; i < patternSize; i++) {
        inputArray[i] = new Input();
        outputArray[i] = new Output();	
      }
      for (int i = 0; i < codeSize; i++) {
        hiddenArray[i] = new Hidden();
      }

      outputLayer = new Layer_Output_(outputArray); 
      hiddenLayer = new Layer_Hidden_(hiddenArray);
      inputLayer = new Layer_Input_(inputArray);

      hiddenLayer.x.connect(inputLayer.y, full);
      outputLayer.x.connect(hiddenLayer.y, full);
      outputLayer.outErr.associate(hiddenLayer.inErr, full);

      patternPool = new Double[patternSize][patternSize];
      for (int i = 0; i < patternSize; i++) {
        for (int j = 0; j < patternSize; j++) {
          patternPool[i][j] = (i == j) ? 1.0 : 0.0;
        }
      }
    }
	
    void initNetwork() {
      Random random = new Random(randomSeed);
      hiddenLayer.initSignals(random);
      outputLayer.initSignals(random);
    }
	
    double Forward(int patternNo) {
      BasedLayer thisPatterns = new LayerBase(patternPool[patternNo]);
      inputLayer.setInput(thisPatterns);
      outputLayer.setTarget(thisPatterns);

      hiddenLayer.Forward();
      outputLayer.Forward();

      Double[] outputErrors = outputLayer.getError().getD1().getBase();
      double accumulator = 0.0;
      for (Double error : outputErrors) { 
        accumulator += error;
      }
      return Math.sqrt(accumulator);
    }
	
    void Backward() {
      outputLayer.Backward(eta);
      hiddenLayer.Backward(eta);
    }

    double RunEpoch() {
      double err = 0.0;
      for (int p = 0; p < patternSize; p++) {
        err += Forward(p);
        Backward();
      }
      return err/patternSize;
    }
	
  }

For further dynamic modelling four methods are provided:

  • method initNetwork() intializes the network weights and bias values;
  • method Forward() takes a given pattern and sets it as input and target. After that it performs the forward pass and returns the accumulated error:
  • method Backward() performs the backward pass;
  • method RunEpoch() performs a training cycle, i.e. it performs the forward and backward pass for all patterns, and returns the normalized overall error.

More Information