Data Transformation

One of the fundamental steps of Exploratory Data Analysis (EDA) is data wrangling. In this chapter, we will learn how to merge database-style dataframes, merging on the index, concatenating along an axis, combining data with overlap, reshaping with hierarchical indexing, and pivoting long to wide format. We will come to understand the work that must be completed before transferring our information for further examination, including, removing duplicates, replacing values, renaming axis indexes, discretization and binning, and detecting and filtering outliers. We will work on transforming data using a function, mapping, permutation and random sampling, and computing indicators/dummy variables. 

This chapter will cover the following topics:

Background

Merging database-style dataframes 

Transformation techniques

Benefits of data transformation