An introduction to neural networks

We can describe a neural network as a mathematical model for information processing. As discussed in Chapter 1, Machine Learning – an Introduction, this is a good way to describe any ML algorithm, but, in this chapter, well give it a specific meaning in the context of neural networks. A neural net is not a fixed program, but rather a model, a system that processes information, or inputs. The characteristics of a neural network are as follows:

• Information processing occurs in its simplest form, over simple elements called neurons.
• Neurons are connected and they exchange signals between them through connection links.
• Connection links between neurons can be stronger or weaker, and this determines how information is processed.
• Each neuron has an internal state that is determined by all the incoming connections from other neurons.
• Each neuron has a different activation function that is calculated on its state, and determines its output signal.

A more general description of a neural network would be as a computational graph of mathematical operations, but we will learn more about that later.

We can identify two main characteristics for a neural net:

• The neural net architecture: This describes the set of connections-namely, feedforward, recurrent, multi or single-layered, and so on-between the neurons, the number of layers, and the number of neurons in each layer.
• The learning: This describes what is commonly defined as the training. The most common but not exclusive way to train a neural network is with the gradient descent and backpropagation.