Hardware and software requirements

The examples in this book were implemented and tested using Python version 3.6. I assume that you're already familiar with the language and common concepts such as virtual environments, so I won't cover in detail how to install the package and how to do this in an isolated way. The external libraries we'll use in this book are open source software, including the following:

• NumPy: This is a library for scientific computing and implementing matrix operations and common functions.
• OpenCV Python bindings: This is a computer vision library, which provides many functions for image processing.
• Gym: This is a RL framework developed and maintained by OpenAI with various environments that can be communicated with, in a unified way.
• PyTorch: This is a flexible and expressive Deep Learning (DL) library. A short essential crash course on it will be given in the next chapter.
• Ptanhttps://github.com/Shmuma/ptan): This is an open source extension to Gym created by the author to support the modern deep RL methods and building blocks. All used classes will be described in detail together with the source code.

A significant portion of this book (parts two, three, and four) is focused on the modern deep RL methods that have been developed over the past few years. The word "deep" in this context means deep learning is heavily used and you may be aware that DL methods are computationally hungry. One modern GPU can be 10- to 100-times faster than even the fastest multiCPU systems. In practice, this means that the same code that takes one hour to train on a system with a GPU, could take from half a day to one week even on the fastest CPU system. It doesn't mean that you can't try the examples from this book without having access to a GPU, but it will take longer. To experiment with the code on your own (the most useful way to learn anything), it would be better get access to a machine with a GPU. This can be done in various ways:

• Buying a modern GPU suitable for CUDA
• Using cloud instances: Both Amazon AWS and Google Cloud can provide you with GPU-powered instances

The instructions on how to set up the system are beyond the scope of the book, but there are plenty of manuals available on the internet. In terms of OS, you should use Linux or macOS, as both PyTorch and most of Gym's environments don't support Windows (at least at the time of writing).

To give you the exact versions of the external dependencies that we'll use throughout the book, here is an output of the pip freeze command (it could be useful for the potential troubleshooting of examples in the book, as open source software and DL toolkits are evolving extremely quickly):

numpy==1.14.2
atari-py==0.1.1
gym==0.10.4
ptan==0.3
opencv-python==3.4.0.12
scipy==1.0.1
torch==0.4.0
torchvision==0.2.1
tensorboard-pytorch==0.7.1
tensorflow==1.7.0
tensorboard==1.7.0

All the examples in the book were written and tested with PyTorch 0.4, which can be installed with the pip install pytorch==0.4.0 command.

Now, let's go to the details of the OpenAI Gym API, which are not complicated, but provide us with tons of environments, from trivial to challenging ones.