Data Access and Distributed Processing for IoT

Data is everywhere: images, speech, text, weather information, the speed of your car, your last EMI, changing stock prices. With the integration of Internet of Things (IoT) systems, the amount of data produced has increased many-fold; an example is sensor readings, which could be taken for room temperature, soil alkalinity, and more. This data is stored and made available in various formats. In this chapter, we will learn how to read, save, and process data in some popular formats. Specifically, you will do the following:

• Access data in TXT format
• Read and write csv-formatted data via the CSV, pandas, and NumPy modules
• Access JSON data using JSON and pandas
• Learn to work with the HDF5 format using PyTables, pandas, and h5py
• Handle SQL databases using SQLite and MySQL
• Handle NoSQL using MongoDB
• Work with Hadoop's Distributed File System