Introducing popular open source libraries

There are many open-source libraries that allow the creation of deep neural nets in Python, without having to explicitly write the code from scratch. In this book, we'll use three of the most popular: - TensorFlow, Keras, and PyTorch. They all share some common features, as follows:

• The basic unit for data storage is the tensor. Consider the tensor as a generalization of a matrix to higher dimensions. Mathematically, the definition of a tensor is more complex, but in the context of deep learning libraries, they are multi-dimensional arrays of base values. A tensor is similar to a NumPy array and is made up of the following:
  • A basic data type of tensor elements. These can vary between libraries, but typically include 16-, 32-, and 64-bit float and 8-, 16-, 32-, and 64-bit integers.
  • An arbitrary number of axes (also known as the rank, order, or degree of the tensor). An 0D tensor is just a scalar value, 1D is a vector, 2D is a matrix, and so on. In deep networks, the data is propagated in batches of n samples. This is done for performance reasons, but it also suits the notion of stochastic gradient descent. For example, if the input data is one-dimensional, such as [0, 1], [1, 0], [0, 0], and [1, 1] for XOR values, we'll actually work with a 2D tensor [[0, 1], [1, 0], [0, 0], [1, 1]] to represent all of the samples in a single batch. Alternatively, two-dimensional grayscale images will be represented as a three-dimensional tensor. In the context of deep learning libraries, the first axis of the tensor represents the different samples.
  • A shape that is the size (the number of values) of each axis of the tensor. For example, the XOR tensor from the preceding example will have a shape of (4, 2). A tensor representing a batch of 32 128x128 images will have a shape of (32, 128, 128).
• Neural networks are represented as a computational graph of operations. The nodes of the graph represent the operations (weighted sum, activation function, and so on). The edges represent the flow of data, which is how the output of one operation serves as an input for the next one. The inputs and outputs of the operations (including the network inputs and outputs) are tensors.
• All libraries include automatic differentiation. This means, that all you need to do is define the network architecture and activation functions, and the library will automatically figure out all of the derivatives required for training with backpropagation.
• All libraries use Python.
• Until now, we've referred to GPUs in general, but in reality, the vast majority of deep learning projects work exclusively with NVIDIA GPUs. This is so because of the better software support NVIDIA provides. These libraries are no exception – to implement GPU operations, they rely on the CUDA toolkit in combination with the cuDNN library. cuDNN is an extension of CUDA, built specifically for deep learning applications. As was previously mentioned in the Applications of deep learning section, you can also run your deep learning experiments in the cloud.

For these libraries, we will quickly describe how to switch between a GPU and a CPU. Much of the code in this book can then be run on a CPU or a GPU, depending on the hardware available to the reader.