another geeky blog

As you already know from the previous article, BackPropagation is a supervised learning algorithm for feed-forward neural networks. The network doesn’t need feedback.

After initializing the neurons weights with random values between two values, the network is capable to automatically adjust the weights using a two phase algorithm:

• ˆforward propagation of a training pattern’s input through the neural network to activate the neurons and get the network outputs
• back propagation from the outputs node to the inner ones in which phase the network previous outputs are compared with the expected outputs from the training patterns, an error estimation is computed and back propagated in order to adjust neurons weights.

The network has an input layer, one or more hidden layers and an output layer.
The number of neurons on each layer can be adapted to meet the requirements.
A neuron is an entity that copies the human brain neuron. It responds to a given input based on the activation function and weights. If the activation function’s output is greater than a threshold, the neuron is excited (it produces an output).
In order to be considered valid, the neuron’s activation function must be differentiable.
Being a gradient descent method, it minimizes the total squared error of the output computed by the net.

The aim is to train the network to achieve a balance between the ability to respond correctly to the input patterns that are used for training and the ability to provide a good response to the inputs that are similar.

The main steps for training the network are:

initialize neurons weights randomly in [ -1 ,1]
while not all examples have been classified correctly
    for each example E in the training set
         forward phase -> compute network's output Out
         error = expected output - Out
         compute delta for all weights backward
         update weights
return network

The sigmoid function is the most used in neuron’s activation step:

A hidden neuron’s (neuron on a hidden layer) output (h) is calculated using the next formula:

where are the network inputs, are the weights, is the activation function threshold, the inputs number and the hidden neurons number.The output layer results are calculated using this formula:

where was calculated before and is the output layer’s neurons number.
The errors for the output layer and for the hidden layer are computed with:


where is the network expected output.The weights for the next step are adjusted as follow:


where is the learning rate.

Using the interfaces presented in the ANN Introduction article, I implemented a basic BackPropagation Network. Here’s the source code for the Neuron:

/**
 * Neuron used in BackPropagation Neural Network
 * @author Octavian Sima
 */
public class BackPropagationNeuron implements Neuron<Double> {

 private int inputsNumber;         //number of neuron inputs
 private Double[] inputs;          //neuron inputs
 private Double[] weights;         //neuron weights
 private Double biasWeight;        //bias neuron weight
 private Double output;            //neuron output after calling compute
 private BackPropagationNetwork parent;  //network which contains the neuron

 @Override
 public void initNeuron(NeuralNetwork<Double> parent, int inputsNumber,
         Double initialWeightMinValue, Double initialWeightMaxValue) {

     this.initialWeightMinValue = initialWeightMinValue;
     this.initialWeightMaxValue = initialWeightMaxValue;
     this.initNeuron(parent, inputsNumber);
 }

 @Override
 public void initNeuron(NeuralNetwork<Double> parent, int inputsNumber) {
     this.parent = (BackPropagationNetwork)parent;
     this.inputsNumber = inputsNumber;
     this.inputs = new Double[inputsNumber];
     this.weights = new Double[inputsNumber];
     //generate random weights for neuron predecesors
     this.generateRandomWeights();
 }

 @Override
 public double activationFunction(Double value) {
     //sigmoid function = 1/(1+e^-x)
     return 1.0 / (1.0 + Math.exp(-1.0 * value));

     //asinh function = ln(x+sqrt(x^2+1))
     //return Math.log(value + Math.sqrt(value * value + 1));
 }

 @Override
 public double activationFunctionDerivative(Double value) {
     //sigmoid function derivative = value * (1 - value)
     return value * (1-value);

    //asinh function derivative = 1/sqrt(1+x^2)
    //return 1/Math.sqrt(1+ value * value);
 }

 @Override
 public Double compute(Double[] input) {
     //forward phase - propagate the output
     Double total = 0.0;
     //update inputs and compute output
     for (int i = 0; i < this.inputsNumber; i++) {
         this.inputs[i] = input[i];
         total += this.weights[i] * input[i];
     }
     //add bias
     total += this.biasWeight;

     //apply activationFunction
     this.output = this.activationFunction(total);
     return this.output;
 }

 /**
 * Backward phase - neuron learns (adjust its weights) from error
 * @param error   error used in weights adjusting
 */
 public void adjustWeights(double error) {
     for (int i = 0; i < this.inputsNumber; i++) {
         double delta = error * this.inputs[i] * this.parent.getLearningRate();
         this.weights[i] += delta;
     }
     //adjust bias neuron weight
     this.biasWeight += error * this.parent.getLearningRate();
 }

 /**
 * Generate random weights between initialWeightMinValue and initialWeightMaxValue
 */
 private void generateRandomWeights() {
     //Random rand = new Random();
     for (int i = 0; i < this.inputsNumber; i++) {
         this.weights[i] = 0.2 * (Math.random() - 0.5);
     }
     this.biasWeight = 0.2 * (Math.random() - 0.5);
 }

 public double getOutput() {
     return this.output;
 }

 public double getWeight(int neuronIndex) {
     return this.weights[neuronIndex];
 }
}

You should be able to use the NeuralNetwork interface in order to implement the BackPropagation network. If you have difficulties, write your email and I’ll send you the code.