Python - Data Preprocessing with Pandas

Python Data Preprocessing with Pandas

Introduction to Pandas

Data preprocessing is one of the most crucial steps in data analysis and machine learning. Python, along with its powerful library Pandas, provides versatile tools to clean, transform, and prepare data efficiently. This guide will cover all essential aspects of data preprocessing using Pandas, with practical examples and outputs.

Pandas is a popular Python library used for data manipulation and analysis. It provides two main data structures: Series (1-dimensional) and DataFrame (2-dimensional). These structures allow efficient data handling and preprocessing for various data types.

Installing Pandas

Before starting, ensure you have Pandas installed in your Python environment.

!pip install pandas

Output:

Collecting pandas
Successfully installed pandas-1.x.x

Importing Pandas

To use Pandas, import it at the beginning of your script or notebook.

import pandas as pd

Loading Data with Pandas

Pandas can read data from various sources such as CSV, Excel, SQL, JSON, and more. CSV files are commonly used in data preprocessing.

# Load CSV file
data = pd.read_csv('data.csv')
# Display first 5 rows
print(data.head())

Output:

   Name   Age   Gender  Salary
0  John   28    Male    50000
1  Jane   32    Female  60000
2  Mike   25    Male    45000
3  Emma   29    Female  52000
4  Dave   35    Male    70000

Handling Missing Data

Missing values are common in datasets. Pandas provides methods to detect, remove, or fill missing data.

Detecting Missing Values

# Check for missing values
print(data.isnull().sum())

Output:

Name      0
Age       2
Gender    0
Salary    1
dtype: int64

Removing Missing Values

# Drop rows with missing values
data_cleaned = data.dropna()
print(data_cleaned)

Output:

   Name  Age  Gender  Salary
0  John  28   Male    50000
1  Jane  32   Female  60000

Filling Missing Values

Instead of dropping, we can fill missing values using mean, median, or mode.

# Fill missing Age with mean
data['Age'].fillna(data['Age'].mean(), inplace=True)
# Fill missing Salary with median
data['Salary'].fillna(data['Salary'].median(), inplace=True)

Output:

   Name   Age   Gender  Salary
0  John   28.0  Male    50000
1  Jane   32.0  Female  60000
2  Mike   25.0  Male    45000
3  Emma   29.0  Female  52000
4  Dave   35.0  Male    70000

Data Transformation

Data transformation includes modifying data formats, scaling numerical values, or encoding categorical variables for machine learning models.

Renaming Columns

# Rename columns for clarity
data.rename(columns={'Salary':'Income', 'Age':'Years'}, inplace=True)
print(data.head())

Output:

   Name  Years  Gender  Income
0  John  28.0  Male    50000
1  Jane  32.0  Female  60000
2  Mike  25.0  Male    45000
3  Emma  29.0  Female  52000
4  Dave  35.0  Male    70000

Encoding Categorical Variables

Machine learning models require numerical inputs. Pandas allows one-hot encoding and label encoding.

# One-hot encoding
data_encoded = pd.get_dummies(data, columns=['Gender'])
print(data_encoded.head())

Output:

   Name  Years  Income  Gender_Female  Gender_Male
0  John  28.0  50000   0              1
1  Jane  32.0  60000   1              0
2  Mike  25.0  45000   0              1
3  Emma  29.0  52000   1              0
4  Dave  35.0  70000   0              1

Scaling Numerical Data

Scaling ensures that numerical features are on the same scale, which improves model performance.

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
data['Years'] = scaler.fit_transform(data[['Years']])
data['Income'] = scaler.fit_transform(data[['Income']])
print(data.head())

Output:

   Name     Years  Gender  Income
0  John  0.166667  Male    0.166667
1  Jane  0.666667  Female  0.500000
2  Mike  0.000000  Male    0.000000
3  Emma  0.333333  Female  0.250000
4  Dave  1.000000  Male    1.000000

Removing Duplicates

Duplicate records can skew analysis. Pandas makes it easy to identify and remove duplicates.

# Drop duplicate rows
data = data.drop_duplicates()

Handling Outliers

Outliers can distort statistical analysis. Use methods like IQR or Z-score to detect and remove outliers.

# Detect outliers using IQR
Q1 = data['Income'].quantile(0.25)
Q3 = data['Income'].quantile(0.75)
IQR = Q3 - Q1
data_no_outliers = data[~((data['Income'] < (Q1 - 1.5 * IQR)) | (data['Income'] > (Q3 + 1.5 * IQR)))]
print(data_no_outliers)

Feature Engineering

Feature engineering involves creating new features from existing data to improve model performance. Examples include extracting day, month, year from dates or creating ratios between variables.

# Example: Create Income per Year ratio
data['Income_per_Year'] = data['Income'] / (data['Years'] + 1)
print(data.head())

Data preprocessing with Pandas is a foundational skill in Python for data analysis and machine learning. By mastering techniques such as handling missing data, encoding categorical variables, scaling, removing duplicates, and feature engineering, you can prepare datasets effectively for analysis and modeling.