Python - Pandas

Python Pandas

Introduction

Pandas is a powerful and widely-used open-source library in Python that provides high-level data structures and tools designed to make data analysis and manipulation fast and easy. It is especially well-suited for working with structured data, such as tables, spreadsheets, and SQL databases. Built on top of NumPy, Pandas enables rich data operations with convenient indexing, grouping, filtering, merging, and time series capabilities.

Installing and Importing Pandas

Installation

pip install pandas

Importing the Library

import pandas as pd

Core Data Structures in Pandas

Series

A Series is a one-dimensional labeled array capable of holding any data type.

import pandas as pd

s = pd.Series([10, 20, 30, 40])
print(s)

DataFrame

A DataFrame is a two-dimensional labeled data structure with columns of potentially different types.

data = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 35],
    'Salary': [50000, 60000, 70000]
}

df = pd.DataFrame(data)
print(df)

Creating DataFrames

From Dictionary

data = {
    'Product': ['Apple', 'Banana', 'Mango'],
    'Price': [100, 40, 60]
}
df = pd.DataFrame(data)
print(df)

From Lists

data = [['Tom', 25], ['Jerry', 30]]
df = pd.DataFrame(data, columns=['Name', 'Age'])
print(df)

From CSV File

df = pd.read_csv('data.csv')
print(df.head())

Data Inspection

Head and Tail

print(df.head())     # First 5 rows
print(df.tail())     # Last 5 rows

Shape and Info

print(df.shape)
print(df.info())

Summary Statistics

print(df.describe())

Indexing and Selection

Accessing Columns

print(df['Name'])

Accessing Rows

print(df.loc[0])     # Access by label
print(df.iloc[0])    # Access by position

Slicing

print(df[1:3])

Data Manipulation

Adding Columns

df['Tax'] = df['Salary'] * 0.1
print(df)

Renaming Columns

df.rename(columns={'Salary': 'Income'}, inplace=True)
print(df)

Dropping Columns and Rows

df.drop('Tax', axis=1, inplace=True)      # Drop column
df.drop(1, axis=0, inplace=True)          # Drop row

Filtering Data

print(df[df['Age'] > 28])

Changing Data Types

df['Age'] = df['Age'].astype(float)

Handling Missing Data

Detecting Missing Data

print(df.isnull())
print(df.isnull().sum())

Filling Missing Values

df.fillna(value=0, inplace=True)

Dropping Missing Values

df.dropna(inplace=True)

Data Aggregation

GroupBy Operations

grouped = df.groupby('Age')
print(grouped['Income'].mean())

Aggregation Functions

print(df.agg({'Income': ['sum', 'mean'], 'Age': ['min', 'max']}))

Merging, Joining, and Concatenating

Concatenation

df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'A': [5, 6], 'B': [7, 8]})

result = pd.concat([df1, df2])
print(result)

Merge

df1 = pd.DataFrame({'key': ['K0', 'K1'], 'A': ['A0', 'A1']})
df2 = pd.DataFrame({'key': ['K0', 'K1'], 'B': ['B0', 'B1']})

merged = pd.merge(df1, df2, on='key')
print(merged)

Join

df1 = df1.set_index('key')
df2 = df2.set_index('key')

joined = df1.join(df2)
print(joined)

Sorting and Ranking

Sorting by Values

df.sort_values(by='Age', ascending=False, inplace=True)
print(df)

Sorting by Index

df.sort_index(inplace=True)

Ranking

df['Rank'] = df['Income'].rank()
print(df)

Working with Dates and Times

Creating DateTime Objects

df['JoinDate'] = pd.to_datetime(['2023-01-01', '2023-02-15', '2023-03-01'])

Extracting Date Components

df['Year'] = df['JoinDate'].dt.year
df['Month'] = df['JoinDate'].dt.month

Filtering by Date

print(df[df['JoinDate'] > '2023-01-31'])

Reading and Writing Files

Read CSV

df = pd.read_csv('data.csv')

Write CSV

df.to_csv('output.csv', index=False)

Read Excel

df = pd.read_excel('data.xlsx')

Write Excel

df.to_excel('output.xlsx', index=False)

Pivot Tables

Basic Pivot Table

pivot = df.pivot_table(values='Income', index='Age', aggfunc='mean')
print(pivot)

Visualization with Pandas

Line Plot

df.plot(x='Age', y='Income', kind='line')

Bar Plot

df.plot(x='Name', y='Income', kind='bar')

Histogram

df['Age'].plot(kind='hist')

Advanced Topics

MultiIndex DataFrames

arrays = [['bar', 'bar', 'baz'], ['one', 'two', 'three']]
index = pd.MultiIndex.from_arrays(arrays, names=('first', 'second'))
df = pd.DataFrame({'A': [1, 2, 3]}, index=index)
print(df)

Reshaping with Stack and Unstack

df_unstacked = df.unstack()
df_stacked = df_unstacked.stack()

Window Functions

df['RollingMean'] = df['Income'].rolling(window=2).mean()
print(df)

Pandas is a cornerstone tool in the Python data science ecosystem. It provides intuitive, flexible, and efficient methods for data analysis and preprocessing, enabling rapid insights and modeling. With features such as powerful indexing, missing data handling, groupby and aggregation, file I/O, and time series support, Pandas is indispensable for both beginners and professionals working with real-world data.

Learning Pandas thoroughly equips you to handle almost any kind of tabular data manipulation or analysis task in Python. As you continue exploring, combining Pandas with libraries like NumPy, matplotlib, seaborn, and scikit-learn will further expand your data science capabilities.