Python - Advanced NumPy for Numerical Data

Advanced NumPy for Numerical Data in Python

Introduction

NumPy, short for Numerical Python, is the cornerstone of scientific computing in Python. While the basics of NumPy (like array creation and simple operations) are widely used, advanced NumPy functionalities enable high-performance operations on large datasets, broadcasting, linear algebra, memory efficiency, and integration with compiled languages. This guide explores those powerful features in-depth, making it essential for numerical analysts, data scientists, and machine learning practitioners.

Understanding NumPy Internals

NumPy ndarray Structure

Every NumPy array is an instance of the ndarray class. Internally, it consists of:

• Pointer to a block of memory
• Data type (dtype) of the elements
• Shape of the array
• Strides (steps to jump to the next element)

Inspecting Internal Attributes

import numpy as np

arr = np.array([[1, 2, 3], [4, 5, 6]])

print("Shape:", arr.shape)
print("Strides:", arr.strides)
print("Item size:", arr.itemsize)
print("Total bytes:", arr.nbytes)

Advanced Indexing Techniques

Boolean Indexing

x = np.array([10, 20, 30, 40, 50])
mask = x > 25
print(x[mask])

Fancy Indexing

x = np.array([5, 10, 15, 20, 25])
indices = [1, 3]
print(x[indices])

Multi-dimensional Fancy Indexing

arr = np.arange(16).reshape(4, 4)
print(arr[[1, 3], [2, 0]])

Conditional Replacement

x = np.array([1, 2, 3, 4, 5])
x[x > 3] = 99
print(x)

Broadcasting Mechanism

What is Broadcasting?

Broadcasting allows NumPy to perform arithmetic on arrays of different shapes by expanding the smaller array implicitly.

Broadcasting Rules

a = np.array([1, 2, 3])
b = np.array([[10], [20], [30]])

print(a + b)

Manual Broadcasting with np.newaxis

x = np.array([1, 2, 3])
y = x[:, np.newaxis]  # Adds a new axis
print(x.shape, y.shape)

Advanced Array Manipulations

Reshape vs Resize

arr = np.arange(12)
reshaped = arr.reshape((3, 4))
print("Reshaped:\n", reshaped)

arr.resize((2, 6))
print("Resized:\n", arr)

Flattening vs Ravel

arr = np.array([[1, 2], [3, 4]])
print(arr.flatten())  # Returns a copy
print(arr.ravel())    # Returns a view

Stacking Arrays

a = np.array([1, 2])
b = np.array([3, 4])

print(np.stack((a, b), axis=0))
print(np.hstack((a, b)))
print(np.vstack((a, b)))

Linear Algebra with NumPy

Matrix Multiplication

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

print(np.dot(A, B))
print(A @ B)  # Python 3.5+ operator

Inverse and Determinant

from numpy.linalg import inv, det

matrix = np.array([[4, 7], [2, 6]])
print("Inverse:\n", inv(matrix))
print("Determinant:", det(matrix))

Eigenvalues and Eigenvectors

from numpy.linalg import eig

vals, vecs = eig(matrix)
print("Eigenvalues:", vals)
print("Eigenvectors:\n", vecs)

Random Number Generation

Random Array

np.random.seed(0)
rand_arr = np.random.rand(3, 3)
print(rand_arr)

Normal Distribution

normal_arr = np.random.normal(loc=0.0, scale=1.0, size=(2, 3))
print(normal_arr)

Random Integers

int_arr = np.random.randint(0, 10, size=(3, 3))
print(int_arr)

Permutation and Shuffle

arr = np.arange(10)
np.random.shuffle(arr)
print(arr)

perm = np.random.permutation(10)
print(perm)

Memory Efficiency and Views

Copy vs View

a = np.array([1, 2, 3])
b = a.view()
b[0] = 100
print("Original:", a)

Using strides for slicing

x = np.arange(16).reshape(4, 4)
print(x[::2, ::2])  # Strided slicing

Performance Optimization

Vectorized Operations vs Loop

x = np.arange(1_000_000)
y = np.arange(1_000_000)

# Fast vectorized addition
result = x + y

Using np.where

x = np.array([1, 2, 3, 4])
y = np.where(x > 2, x * 10, x)
print(y)

Using numexpr for faster computation

import numexpr as ne

a = np.arange(1e6)
b = np.arange(1e6)

c = ne.evaluate("a + b * 2")
print(c)

Masked Arrays

Introduction to MaskedArray

import numpy.ma as ma

data = np.array([1, 2, 999, 4])
masked_data = ma.masked_equal(data, 999)
print(masked_data)

Using Structured Arrays

Creating Structured Arrays

structured = np.array([(1, 'A'), (2, 'B')],
                      dtype=[('id', 'i4'), ('label', 'U1')])

print(structured['id'])
print(structured['label'])

Advanced UFuncs and Custom Operations

Universal Functions

x = np.array([1, 2, 3])
print(np.exp(x))
print(np.sqrt(x))

Creating Custom UFuncs

def custom_op(x, y):
    return x**2 + y

vec_func = np.frompyfunc(custom_op, 2, 1)
result = vec_func([1, 2, 3], [4, 5, 6])
print(result)

Broadcasting with Higher Dimensions

Practical Broadcasting Example

matrix = np.array([[1], [2], [3]])
vector = np.array([10, 20, 30])
result = matrix + vector
print(result)

Practical Applications of NumPy

1. Image Processing

from PIL import Image

img = Image.open("sample.jpg").convert("L")
img_np = np.array(img)
print(img_np.shape)

# Apply thresholding
img_np[img_np < 128] = 0
img_np[img_np >= 128] = 255

2. Financial Data Analysis

prices = np.array([100, 101, 102, 98, 105])
returns = np.diff(prices) / prices[:-1]
print("Returns:", returns)

3. Simulation with Random Walk

steps = np.random.choice([-1, 1], size=1000)
position = np.cumsum(steps)

Tips and Best Practices

• Prefer vectorized operations over loops for performance
• Use np.where and np.select for conditionals
• Leverage broadcasting to avoid unnecessary reshaping
• Use np.ravel or views for memory efficiency
• Explore masked arrays for handling missing or invalid data

Advanced NumPy capabilities go far beyond simple array creation and slicing. Mastering these features unlocks high-performance numerical computing in Python. From broadcasting and advanced indexing to structured arrays, masked operations, linear algebra, and random simulations, NumPy enables efficient and scalable data processing. It's the backbone of many scientific computing and machine learning tasks, forming the core of the Python data ecosystem. Practicing these advanced techniques will prepare you for real-world numerical analysis challenges with large datasets and performance constraints.