Kohonen Network - notes on the program

The Kohonen Network is a self-organising map that is trained through unsupervised competitive learning. The aim is to generate a low-dimensional (usually two-dimensional) representation of a higher-dimensional data set.

Each node of the Kohonen map is assigned a randomly initialised weight vector whose length corresponds to the dimensionality of the input data. During the training process, it is first determined for a given input vector which weight vector is closest to this input and then the weight vector of the "winning node" is pulled a little in the direction of the input vector.

The same applies - to a decreasing extent - to the weight vectors of the neighbours of the winning node, whereby the neighbourhood here refers to the topology of the Kohonen map.

In our example, a two-dimensional map is trained with input data from a two-dimensional square space.


The Unit Classes - Using JLayer Annotations

We need two unit types for the network outlined above. The nodes of the two-dimensional Kohonen map are described by the KohonenUnit class, and to determine the "winning node" we need a node of type DecisionUnit.


Unit Class DecisionUnit

Let's start with the DecisionUnit class. A single object of this type decides which node of the Kohonen map is the winning node, and for this decision the Euclidean distances (to the current input) of all weights are maid available to this "decision maker". 


  @LayerUnit
  public class DecisionUnit {
  
    static class DecisionIO {
      double  distance;			// Euclidean distance
      boolean isWinner; 		// smallest distance to input
    } 
	
    @LayerField 
    DecisionIO[] decisionIO;

    @LayerMethod
    void detWinnerIndex() { 
      double winDist = decisionIO[0].distance;
      int    winIx   = 0;
      for (int i = 1; i < decisionIO.length; i++) {
        if (decisionIO[i].distance <  winDist) {
          winDist = decisionIO[i].distance;
          winIx = i;
        }
      }
      for (int i = 0; i <  decisionIO.length; i++) {
        decisionIO[i].isWinner = false;
      }
      decisionIO[winIx].isWinner = true;
    }
	
  }

The nested static class DecisionIO is provided for the interaction of the decision maker with the nodes of the Kohonen map, so that the necessary communication can be handled with the help of the layer vector decisionIO. The Euclidean distances can be read out via the distance field and the decision, which is determined by the layer method detWinnerIndex(), can be communicated via the isWinner field.


Unit Class KohonenUnit

The description of the KohonenUnit class begins with the static field winIx, in which the index of the winning node is entered, and the weight vector w for storing the representative of the input space. This is followed by three parameters, which are used to adjust the weights during training, and two layer fields:

  • LayerField decisionIO is the counterpart to the layer vector of the same name of the unit type DecisionUnit and is used to communicate with the "decision unit".
  • LayerField ix of type int[] is used to store the index of the current node.

Since the parameter isIndex has the value true, LayerField ix is calculated automatically when an array of type KohonenUnit is wrapped. 


  @LayerUnit
  public class KohonenUnit { // for a two-dimensional Kohonen map
	
    static int[] winIx;      // index of winner unit        
    double[] w;              // weight vector for representing the input space

    // parameters for weight adaptation
    static int	   rad = 4;
    static double  eta = 0.2;
    static double  decay = 0.5;
	
    @LayerField 
    DecisionUnit.DecisionIO decisionIO;

    @LayerField(isIndex = true) 
    int[] thisIx;            // index of this unit

    @LayerMethod
    public void initWeights(Random r, double lwb, double upb) { 
      w = new double[2];
      w[0] = r.nextDouble(lwb, upb);
      w[1] = r.nextDouble(lwb, upb);
    }

    @LayerMethod
    public void nextInputSample(double x0, double x1) {
      decisionIO.distance = euDist(x0, x1, w[0], w[1]);
    }
	
    @LayerMethod
    public void setWinnerIndex(){
      if (decisionIO.isWinner) winIx = thisIx;
    }

    @LayerMethod
    public void adaptWeights(double x0, double x1){
      double euDist = euDist(thisIx, winIx);
      if (euDist < rad) {
        w[0] = w[0] + eta * phi(euDist, decay) * (x0 - w[0]);
        w[1] = w[1] + eta * phi(euDist, decay) * (x1 - w[1]);
      }
    }
	
    // utility functions

    // Euclidean distance
    static double euDist(double x0, double x1, double w0, double w1) {
      double delta1 = x0 - w0;
      double delta2 = x1 - w1;
      double result = Math.sqrt(delta1*delta1 + delta2*delta2);
      return result;
    }
    static double euDist(int[] ix0, int[] ix1) {
      double delta1 = ix0[0] - ix1[0];
      double delta2 = ix0[1] - ix1[1];
      double result = Math.sqrt(delta1*delta1 + delta2*delta2);
      return result;
    }

    // adaptation function
    static double phi(double dist, double decay) {
      return 1.0 / Math.exp(decay * dist);
    }
	
  }

Class KohonenUnit contains four layer methods, which support the training process.

  • LayerMethod initWeights() initialises the weight vector.
  • LayerMethod nextInputSample() is used to pass the next input and to calculate the distance to the current weight vector.
  • As soon as the "decision unit" has determined the winning node, LayerMethod setWinnerIndex() is used to set the index of the winner unit.
  • Finally, the LayerMethod adaptWeigths() is used to pull the weight vector in the direction of the current input, whereby the further away the index of the current node is from the index of the winning node, the lower the degree of adaptation.

The utility functions at the end compute the Euclidean distances between weight vectors or indices or are used for weight adjustment. Note that the return value of phi() decreases with increasing parameters.


Establishing Global Structure and Dynamics - Using the Generated Wrapper Classes

From the two unit classes defined above, the Jlayer annotation processor generates the wrapper classes Layer_DecisionUnit_ and Layer_KohonenUnit_. These classes are used by the following KohonenNet class to create a two-dimensional map that is trained with input from a two-dimensional quadratic space.

At the top, the definition of relation full can be found, which applies for any pair of indices, followed by some items to be used, when the weights are randomly initialized.

Next, the central building blocks of the application are defined, the decisionUnit array with the associated decisionMaker wrapper and the kohArray array with the associated kohLayer wrapper.

Method createNet() instantiates and wraps these arrays and finally connects the (one-element) vector layer decisionMaker.decisionIO with the two-dimensional vector field kohLayer.decisionIO. Method initNet() randomly initialises the weights. 


  public class KohonenNet {
	
    Relation full = new Relation() {
      @Override
      public boolean contains(int[] x, int[] y) {
        return true;
      }
    };
	
    Random 	rgen;
    int 	seed = 4711;
    double 	lwb = 0.4, upb = 0.5;

    private DecisionUnit[]           decisionUnit;
    private Layer_DecisionUnit_ decisionMaker; 

    private KohonenUnit[][]          kohArray;
    private Layer_KohonenUnit_  kohLayer; 

    public void createNet(int lines, int cols) {
      // create decisionMaker
      decisionUnit 	= new DecisionUnit[1];
      decisionUnit[0]	= new DecisionUnit();
      decisionMaker 	= new Layer_DecisionUnit_(decisionUnit); 

      // create Kohonen layer
      kohArray = new KohonenUnit[width][heigth];
      for(int i = 0; < width; i++) {
        for(int j = 0; j < heigth; j++) {
          kohArray[i][j] = new KohonenUnit();
        }
      }
      kohLayer = new Layer_KohonenUnit_(kohArray); 

      // establish connections
      decisionMaker.decisionIO.connect(kohLayer.decisionIO, full); 
    }
	
    public void initNet() {
      rgen = new Random(seed++);
      kohLayer.initWeights(rgen, lwb, upb); 
    }
	
    public void trainNet() {
      double x1 = rgen.nextDouble(0.0, 1.0);
      double x2 = rgen.nextDouble(0.0, 1.0);
      kohLayer.nextInputSample(x1, x2); 
      decisionMaker.detWinnerIndex(); 
      kohLayer.setWinnerIndex(); 
      kohLayer.adaptWeights(x1, x2); 
    }  

    public double[] getWeights(int i, int j) {
      double[] weights = new double[2];
      weights[0] = kohArray[i][j].w[0];
      weights[1] = kohArray[i][j].w[1];
      return weights;
    }
	
  }

Method trainNet() implements one training step. At first, an input sample from the input space is generated randomly and then the four layer methods from KohonenUnit, which were designed for that purpose, are executed.

The final method getWeights() can be used to extract the networt weights.


More Information