Keeping track of package versions and history in IPython Notebook

The IPython Notebook was added to IPython 0.12 in December 2011. Many Pythonistas feel that the IPython Notebook is essential for reproducible data analysis. The IPython Notebook is comparable to commercial products such as Mathematica, MATLAB, and Maple. It is an interactive web browser-based environment. In this recipe, we will see how to keep track of package versions and store IPython sessions in the context of reproducible data analysis. By the way, the IPython Notebook has been renamed Jupyter Notebook.

Getting ready

For this recipe, you will need a recent IPython installation. The instructions to install IPython are at http://ipython.org/install.html (retrieved July 2015). Install it using the pip command:

$ [sudo] pip install ipython/jupyter

If you have installed IPython via Anaconda already, check for updates with the following commands:

$ conda update conda
$ conda update ipython ipython-notebook ipython-qtconsole

I have IPython 3.2.0 as part of the Anaconda distribution.

How to do it...

We will install log a Python session and use the watermark extension to track package versions and other information. Start an IPython shell or notebook. When we start a session, we can use the command line switch --logfile=<file name>.py. In this recipe, we use the %logstart magic (IPython terminology) function:

In [1]: %logstart cookbook_log.py rotate
Activating auto-logging. Current session state plus future input saved.
Filename : cookbook_log.py
Mode : rotate
Output logging : False
Raw input log : False
Timestamping : False
State : active

This example invocation started logging to a file in rotate mode. Both the filename and mode are optional. Turn logging off and back on again as follows:

In [2]: %logoff
Switching logging OFF

In [3]: %logon
Switching logging ON

Install the watermark magic from Github with the following command:

In [4]: %install_ext https://raw.githubusercontent.com/rasbt/watermark/master/watermark.py

The preceding line downloads a Python file, in my case, to ~/.ipython/extensions/watermark.py. Load the extension by typing the following line:

%load_ext watermark

The extension can place timestamps as well as software and hardware information. Get additional usage documentation and version (I installed watermark 1.2.2) with the following command:

%watermark?

For example, call watermark without any arguments:

In [7]: %watermark
… Omitting time stamp …

CPython 3.4.3
IPython 3.2.0

compiler : Omitting
system : Omitting
release : 14.3.0
machine : x86_64
processor : i386
CPU cores : 8
interpreter: 64bit

I omitted the timestamp and other information for personal reasons. A more complete example follows with author name (-a), versions of packages specified as a comma-separated string (-p), and custom time (-c) in a strftime() based format:

In [8]: %watermark -a "Ivan Idris" -v -p numpy,scipy,matplotlib -c '%b %Y' -w
Ivan Idris 'Jul 2015'

CPython 3.4.3
IPython 3.2.0

numpy 1.9.2
scipy 0.15.1
matplotlib 1.4.3
watermark v. 1.2.2

How it works...

The IPython logger writes commands you type to a Python file. Most of the lines are in the following format:

get_ipython().magic('STRING_YOU_TYPED')

You can replay the session with %load <log file>. The logging modes are described in the following table:

We used a custom magic function available on the Internet. The code for the function is in a single Python file and it should be easy for you to follow. If you want different behavior, you just need to modify the file.

See also

• The custom magics documentation at http://ipython.org/ipython-doc/dev/config/custommagics.html (retrieved July 2015)
• Helen Shen (2014). Interactive notebooks: Sharing the code. Nature 515 (7525): 151–152. doi:10.1038/515151a
• IPython reference documentation at https://ipython.org/ipython-doc/dev/interactive/reference.html (retrieved July 2015)