Python - Combined Example: Using Both Requests and Beautiful Soup

Combined Example: Using Both Requests and Beautiful Soup

Web scraping is the process of extracting data from websites. In Python, the two most commonly used libraries for this purpose are requests and BeautifulSoup. The requests library allows you to send HTTP/1.1 requests, and BeautifulSoup allows you to parse HTML or XML documents efficiently. This document provides a detailed explanation of how to use these libraries together for effective web scraping, covering everything from installation to advanced scraping examples.

Introduction to Requests and BeautifulSoup

Why Combine Requests and BeautifulSoup?

While requests is used for fetching the HTML content of a webpage, BeautifulSoup is used for parsing and navigating that HTML content. Together, they form a powerful toolkit for scraping static web pages.

Basic Workflow of Web Scraping

1. Send an HTTP request to the target website using requests. 
2. Receive the HTML content from the response.
3. Parse the HTML using BeautifulSoup.
4. Extract the required data by navigating the parsed HTML tree.

Installing the Required Libraries

Using pip

pip install requests
pip install beautifulsoup4

Optional Parser Installation

pip install lxml
pip install html5lib

These optional parsers may improve speed and accuracy of HTML parsing.

Step-by-Step Example: Scraping Quotes from a Website

Website Used: quotes.toscrape.com

This website is specifically built for practicing web scraping. It displays quotes from famous people, along with authors and tags.

Step 1: Import Libraries

import requests
from bs4 import BeautifulSoup

Step 2: Make an HTTP Request

url = "http://quotes.toscrape.com"
response = requests.get(url)
print(response.status_code)  # Should return 200 if successful

Step 3: Parse HTML with BeautifulSoup

soup = BeautifulSoup(response.text, 'html.parser')

Step 4: Extract Quotes

quotes = soup.find_all('div', class_='quote')

for quote in quotes:
    text = quote.find('span', class_='text').text
    author = quote.find('small', class_='author').text
    tags = [tag.text for tag in quote.find_all('a', class_='tag')]
    
    print(f"Quote: {text}")
    print(f"Author: {author}")
    print(f"Tags: {', '.join(tags)}\n")

Analyzing the HTML Structure

Inspecting the HTML

Each quote is wrapped in a <div class="quote">  element. Inside this div:

• The quote text is inside <span class="text">
• The author is inside <small class="author">
• Tags are nested within <div class="tags"> under <a class="tag">

Handling Multiple Pages

Paginated Content

The site includes pagination using a "Next" button. We can loop through all pages until there is no "next" button.

Code Example for Pagination

url = "http://quotes.toscrape.com"
while url:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    
    quotes = soup.find_all('div', class_='quote')
    for quote in quotes:
        text = quote.find('span', class_='text').text
        author = quote.find('small', class_='author').text
        print(f"{text} - {author}")
    
    next_button = soup.find('li', class_='next')
    if next_button:
        next_url = next_button.find('a')['href']
        url = f"http://quotes.toscrape.com{next_url}"
    else:
        break

Handling Errors Gracefully

HTTP Error Codes

response = requests.get(url)
if response.status_code != 200:
    print(f"Failed to retrieve the page: {response.status_code}")

Using Try-Except for Robustness

try:
    response = requests.get(url)
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print("Error fetching data:", e)

Working with Headers and User-Agent

Why Set Headers?

Some websites block requests without proper headers. Setting a fake User-Agent can help mimic a real browser.

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
}
response = requests.get(url, headers=headers)

Extracting and Saving Data

Storing in a CSV File

import csv

with open('quotes.csv', 'w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    writer.writerow(['Text', 'Author', 'Tags'])
    
    for quote in quotes:
        text = quote.find('span', class_='text').text
        author = quote.find('small', class_='author').text
        tags = [tag.text for tag in quote.find_all('a', class_='tag')]
        writer.writerow([text, author, ', '.join(tags)])

Using pandas for Data Analysis

import pandas as pd

data = []

for quote in quotes:
    text = quote.find('span', class_='text').text
    author = quote.find('small', class_='author').text
    tags = [tag.text for tag in quote.find_all('a', class_='tag')]
    data.append({'Text': text, 'Author': author, 'Tags': ', '.join(tags)})

df = pd.DataFrame(data)
df.to_csv('quotes_pandas.csv', index=False)

Advanced Techniques

Using CSS Selectors with select()

quotes = soup.select('div.quote')
for q in quotes:
    print(q.select_one('span.text').text)

Scraping with Authentication

Some websites require login credentials. Use requests.Session() to manage login sessions.

session = requests.Session()
login_data = {'username': 'your_username', 'password': 'your_password'}
session.post('http://example.com/login', data=login_data)
response = session.get('http://example.com/protected-page')

Limitations of BeautifulSoup and Requests

• Cannot scrape JavaScript-rendered content (use Selenium for this)
• Some sites detect and block bots using rate limiting or captchas
• Not suitable for heavy-duty parallel scraping (use Scrapy for large-scale tasks)

Ethical and Legal Considerations

Respect robots.txt

Always check if the site allows scraping:

http://example.com/robots.txt

Be Polite: Add Delays

import time

time.sleep(2)  # Sleep for 2 seconds between requests

Avoid Personal Data Collection

Do not collect sensitive data or violate privacy policies. Ensure your scraping complies with local data protection laws like GDPR or CCPA.

Combining Python's requests and BeautifulSoup libraries provides a powerful, flexible, and accessible way to extract data from static websites. This approach is ideal for beginners and intermediate users who want to automate data collection and analysis tasks. Whether you're scraping quotes for inspiration, job listings for research, or tables for analysis, understanding how to use these tools together is a foundational skill in modern data science and automation workflows.

Always remember to respect the legal and ethical boundaries of web scraping. Use public APIs when available, read website terms of use, and never overload a site’s server with unnecessary requests. With responsible and efficient use, web scraping can unlock tremendous potential for data extraction and insight generation.