Using HDF5 with h5py

The h5py module is the most popular way to handle HDF5 files in Python. A new or existing HDF5 file can be opened with the h5py.File() function. After the file is open, its groups can simply be accessed by subscripting the file object as if it was a dictionary object. For example, the following code opens an HDF5 file with h5py and then prints the array stored in the /global_power group:

import h5py
hdf5file = h5py.File('pytable_demo.hdf5')
ds=hdf5file['/global_power']
print(ds)
for i in range(len(ds)):
    print(arr[i])
hdf5file.close()

The arr variable prints an HDF5 dataset type:

<HDF5 dataset "global_power": shape (9, 2), type "<f8">
[2.58  0.136]
[2.552 0.1  ]
[2.55 0.1 ]
[2.55 0.1 ]
[2.554 0.1  ]
[2.55 0.1 ]
[2.534 0.096]
[2.484 0.   ]
[2.468 0.   ]

For a new hdf5file, datasets and groups can be created by using the hdf5file.create_dataset() function, returning the dataset object, and the hdf5file.create_group() function, returning the folder object. The hdf5file file object is also a folder object representing /, the root folder. Dataset objects support array style slicing and dicing to set or read values from them. For example, the following code creates an HDF5 file and stores one dataset:

import numpy as np
arr = np.loadtxt('temp.csv', skiprows=1, usecols=(2,3), delimiter=',')

import h5py
hdf5file = h5py.File('h5py_demo.hdf5')
dataset1 = hdf5file.create_dataset('global_power',data=arr)
hdf5file.close()

h5py provides an attrs proxy object with a dictionary-like interface to store and retrieve metadata about the file, folders, and datasets. For example, the following code sets and then prints the dataset and file attribute:

dataset1.attrs['owner']='City Corp.'
print(dataset1.attrs['owner'])

hdf5file.attrs['security_level']='public'
print(hdf5file.attrs['security_level'])

For more information about the h5py library, refer to the documentation at the following link: http://docs.h5py.org/en/latest/index.html.

So far, we have learned about different data formats. Often, large data is stored commercially in databases, therefore we will explore how to access both SQL and NoSQL databases next.