The neural network is a computing system (or processor) born from the idea of artificially simulating the behavior of the human brain, to exploit its characteristics of
- Complexity
- Non-linearity
- Parallel processing system
- Knowledge is acquired by the network through a learning process
- Neural connections, i.e. synaptic weights, are used to store the acquired information.
The use of networks therefore allows the possibility of generalizing the knowledge acquired during the learning phase.
The idea underlying neural networks consists of the neuron which, based on the functioning of the human brain, is an information processing unit.
The neuron consists of a unit that receives as input a numerical value consisting of a weighted sum of different signals, and processes it by activating or remaining inactive depending on whether the activation threshold is exceeded or not.
More specifically, the artificial neuron uses the input data as an argument for a function, called the activation function \(s()\) which returns as output the values 0, 1 or a value in the range [0, 1].
Therefore, the neuron is characterized by the function and the activation threshold. The latter is usually introduced through a constant input equal to 1, modulated by a coefficient called bias, whose effect is to control the translation of the activation threshold with respect to the origin of the signals. Formally, the bias plays a role no different from that of the weights, which act as regulators of the intensity of the emitted or received signal.
It is common practice to adopt different activation functions, depending on the role that the neuron and the neural network are intended to play. The most common activation functions are:
-
Sigmoid
\[s(x) = \frac{1}{1+e^{-x}}\]
-
Step function
\[s(x) = \begin{cases} 0, & \mbox{if } x \leq a \\ 1, & \mbox{if } x>a \end{cases} \]
-
Hyperbolic tangent
-
Ramp
Assuming a simple neural network:
To train this simple network for a binary classification problem, assuming that the two input variables are \(X_1\) e \(X_2\) and that the output is a dichotomous variable \(Y\) two processes called Back-propagation and Forward-propagation must be adopted:
- Choose a Learning Rate (\(lr\)) and a number of replications \(epoch\);
- Initialize the values \(w_1\), \(w_2\) e \(b=1\) arbitrarily;
- Forward-propagation:
- Randomly select \(x_1 \in X_1\) e \(x_2 \in X_2\) (for simplicity we assume one data point passes through the network at a time: batch size = 1; but the same reasoning can be extended to a vector of information);
- Proceed to estimate \(Y\): \[ \eta = x_1 \space w_1 + x_2 \space w_2 + b\] \[ u = s(\eta)\]
- Calculate the mean squared prediction error: \[MSE=(y-u)^2=(y-s(x_1 \space w_1 + x_2 \space w_2 + b))^2\]
- Back-propagation:
- Calculate the partial derivatives of the cost with respect to \(w_1\), \(w_2\) e \(b\)
- Calculate the partial derivative of the cost function with respect to \(w_1\): \[ \frac{\partial MSE}{\partial w_1} = \frac{\partial MSE}{\partial u} \space \frac{\partial s(x_1 \space w_1 + x_2 \space w_2 + b)}{\partial (x_1 \space w_1 + x_2 \space w_2 + b)} \space \frac{\partial (x_1 \space w_1 + x_2 \space w_2 + b)}{\partial w_1} =\] \[= 2\space (y-u) \space \frac{\partial s(\eta)}{\partial \eta)} \space x_1 \]
- Calculate the partial derivative of the cost function with respect to \(w_2\): \[ \frac{\partial MSE}{\partial w_2} = \frac{\partial MSE}{\partial u} \space \frac{\partial s(x_1 \space w_1 + x_2 \space w_2 + b)}{\partial (x_1 \space w_1 + x_2 \space w_2 + b)} \space \frac{\partial (x_1 \space w_1 + x_2 \space w_2 + b)}{\partial w_2} =\] \[= 2\space (y-u) \space \frac{\partial s(\eta)}{\partial \eta)} \space x_2 \]
- Calculate the partial derivative of the cost function with respect to \(b\): \[ \frac{\partial MSE}{\partial b} = \frac{\partial MSE}{\partial u} \space \frac{\partial s(x_1 \space w_1 + x_2 \space w_2 + b)}{\partial (x_1 \space w_1 + x_2 \space w_2 + b)} \space \frac{\partial (x_1 \space w_1 + x_2 \space w_2 + b)}{\partial b} =\] \[= 2\space (y-u) \space \frac{\partial s(u)}{\partial u)} \]
- Update the weights and bias \[w1 = w_1 - lr \space \frac{\partial MSE}{\partial w_1}\] \[w2 = w_2 - lr \space \frac{\partial MSE}{\partial w_2}\] \[b = b - lr \space \frac{\partial MSE}{\partial b}\]
- Repeat the process \(epoch\) times.
Therefore, in neural networks, we propagate forward to obtain the output and compare it with the real value to get the error, and subsequently, to minimize the error, we propagate backward by finding the derivative of the error with respect to each weight and subtracting this value weighted by the learning rate from the weight value.
The algorithm stops either when the minimum acceptable error is reached or when the maximum number of epochs established during the design phase is reached (to prevent the training process from running indefinitely). An epoch is a complete update cycle of all weights, feeding the entire training set as input.
Deep learning is that field of machine learning research based on multiple levels of representation, corresponding to hierarchies of features, factors or concepts, where high-level concepts are defined on the basis of lower-level ones. In other words, it refers to a set of techniques based on artificial neural networks organized in multiple layers, where each layer computes values for the next so that information is processed in an increasingly complete manner.
A Multi-Layer Perceptron network (MLP) combines multiple processing levels using artificial neurons. It is composed of an input layer, one or more hidden layers, and an output layer. The layers are interconnected through nodes or neurons, with each layer using the output of the previous layer as input.
The idea behind this method is exactly the one explained previously in the construction of a simple network, but instead of neurons, perceptrons are used. The perceptron is a simple neuron equipped with a "teacher" circuit for learning:
