Configuring pandas

The pandas library has more than a dozen configuration options, as described in http://pandas.pydata.org/pandas-docs/dev/options.html (retrieved July 2015).

Note

The pandas library is Python open source software originally created for econometric data analysis. It uses data structures inspired by the R programming language.

You can set and get properties using dotted notation or via functions. It is also possible to reset options to defaults and get information about them. The option_context() function allows you to limit the scope of the option to a context using the Python with statement. In this recipe, I will demonstrate pandas configuration and a simple API to set and reset options I find useful. The two options are precision and max_rows. The first option specifies floating point precision of output. The second option specifies the maximum rows of a pandas DataFrame to print on the screen.

Getting ready

You need pandas and NumPy for this recipe. Instructions to install NumPy are given in Learning to log for robust error checking. The pandas installation documentation can be found at http://pandas.pydata.org/pandas-docs/dev/install.html (retrieved July 2015). The recommended way to install pandas is via Anaconda. I have installed pandas 0.16.2 via Anaconda. You can update your Anaconda pandas with the following command:

$ conda update pandas

How to do it...

The following code from the options.py file in dautil defines a simple API to set and reset options:

import pandas as pd

def set_pd_options():
    pd.set_option('precision', 4) 
    pd.set_option('max_rows', 5)

def reset_pd_options():
    pd.reset_option('precision') 
    pd.reset_option('max_rows')

The script in configure_pd.py in this book's code bundle uses the following API:

from dautil import options
import pandas as pd
import numpy as np
from dautil import log_api

printer = log_api.Printer()
print(pd.describe_option('precision'))
print(pd.describe_option('max_rows'))

printer.print('Initial precision', pd.get_option('precision'))
printer.print('Initial max_rows', pd.get_option('max_rows'))

# Random pi's, should use random state if possible
np.random.seed(42)
df = pd.DataFrame(np.pi * np.random.rand(6, 2))
printer.print('Initial df', df)

options.set_pd_options()
printer.print('df with different options', df)

options.reset_pd_options()
printer.print('df after reset', df)

If you run the script, you get descriptions for the options that are a bit too long to display here. The getter gives the following output:

'Initial precision'
7

'Initial max_rows'
60

Then, we create a pandas DataFrame table with random data. The initial printout looks like this:

'Initial df'
 0 1
0 1.176652 2.986757
1 2.299627 1.880741
2 0.490147 0.490071
3 0.182475 2.721173
4 1.888459 2.224476
5 0.064668 3.047062 

The printout comes from the following class in log_api.py:

class Printer():
    def __init__(self, modules=None, name=None):
        if modules and name:
            log(modules, name)

    def print(self, *args):
        for arg in args:
            pprint.pprint(arg)

After setting the options with the dautil API, pandas hides some of the rows and the floating point numbers look different too:

'df with different options'
 0 1
0 1.177 2.987
1 2.300 1.881
.. ... ...
4 1.888 2.224
5 0.065 3.047

[6 rows x 2 columns]

Because of the truncated rows, pandas tells us how many rows and columns the DataFrame table has. After we reset the options, we get the original printout back.