Python - Working Code Sample for Requests/Beautiful Soup/Selenium

Working Code Sample for Requests, Beautiful Soup, and Selenium in Python

Introduction

Web scraping and browser automation are essential for data extraction, testing, and automating manual web-based workflows. Python offers three widely-used tools for such tasks: Requests for sending HTTP requests, Beautiful Soup for parsing HTML, and Selenium for automating web browsers.

In this document, we explore working code samples that demonstrate how to use these libraries effectively. Each section includes clear examples and explanations for tasks like fetching content, parsing HTML, navigating DOM trees, and automating browser interaction using Selenium WebDriver. These practical examples will provide a strong foundation for scraping and automation projects.

Part 1: Python Requests – Fetching Web Pages

Installing Requests

pip install requests

Sending a GET Request

import requests

url = "https://www.example.com"
response = requests.get(url)

print("Status Code:", response.status_code)
print("Headers:", response.headers)
print("Content:", response.text[:500])  # Print first 500 characters

Sending a POST Request

url = "https://httpbin.org/post"
data = {"username": "test", "password": "123456"}
response = requests.post(url, data=data)

print("POST Response:", response.json())

Using Headers and User-Agent

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}
response = requests.get("https://httpbin.org/headers", headers=headers)
print(response.json())

Handling Exceptions and Timeouts

try:
    response = requests.get("https://www.example.com", timeout=5)
    response.raise_for_status()
except requests.exceptions.HTTPError as e:
    print("HTTP error:", e)
except requests.exceptions.Timeout:
    print("Request timed out")
except requests.exceptions.RequestException as e:
    print("Error:", e)

Downloading an Image

img_url = "https://www.example.com/logo.png"
response = requests.get(img_url)

with open("logo.png", "wb") as f:
    f.write(response.content)

Part 2: Beautiful Soup – Parsing HTML

Installing Beautiful Soup and Parser

pip install beautifulsoup4
pip install lxml

Basic Parsing

from bs4 import BeautifulSoup
import requests

url = "https://www.example.com"
html = requests.get(url).text
soup = BeautifulSoup(html, "lxml")

print(soup.title.text)

Finding Elements

print(soup.find("h1"))
print(soup.find_all("a"))

Accessing Attributes

for link in soup.find_all("a"):
    print(link.get("href"))

Using CSS Selectors

for item in soup.select("div.content > ul > li"):
    print(item.text)

Extracting Table Data

html = """

Name
Age
Alice
30
Bob
25

"""

soup = BeautifulSoup(html, "lxml")
rows = soup.find_all("tr")

for row in rows:
    cols = row.find_all(["td", "th"])
    print([col.text for col in cols])

Cleaning and Navigating HTML

# Removing script and style tags
for tag in soup(["script", "style"]):
    tag.decompose()

print(soup.get_text())

Real-World Example: Extract Headlines

url = "https://www.bbc.com/news"
html = requests.get(url).text
soup = BeautifulSoup(html, "lxml")

for headline in soup.select("h3"):
    print(headline.text.strip())

Part 3: Selenium – Automating Browsers

Installation

pip install selenium
# Download ChromeDriver from https://chromedriver.chromium.org

Basic Setup

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.example.com")

print(driver.title)
driver.quit()

Finding and Interacting with Elements

from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.google.com")

search_box = driver.find_element(By.NAME, "q")
search_box.send_keys("Selenium Python")
search_box.submit()

print(driver.title)
driver.quit()

Clicking Buttons and Links

button = driver.find_element(By.ID, "submit")
button.click()

Waits and Delays

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.ID, "content")))
print(element.text)

Extracting Data with Selenium

driver.get("https://quotes.toscrape.com")
quotes = driver.find_elements(By.CLASS_NAME, "text")

for q in quotes:
    print(q.text)
driver.quit()

Headless Mode

from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)
driver.get("https://www.example.com")
print(driver.title)
driver.quit()

Handling Pagination

while True:
    quotes = driver.find_elements(By.CLASS_NAME, "text")
    for quote in quotes:
        print(quote.text)

    try:
        next_button = driver.find_element(By.CSS_SELECTOR, ".next > a")
        next_button.click()
    except:
        break

Downloading JavaScript Rendered Data

driver.get("https://www.example.com/js-page")
content = driver.find_element(By.ID, "dynamic-content").text
print(content)

Combining Requests, Beautiful Soup, and Selenium

Use Case: Scraping Search Results with Fallback

# First try with Requests + BS4
try:
    response = requests.get("https://quotes.toscrape.com")
    soup = BeautifulSoup(response.text, "lxml")
    quotes = [q.text for q in soup.select(".quote .text")]
    for q in quotes:
        print(q)
except:
    # If fails, use Selenium
    from selenium import webdriver
    driver = webdriver.Chrome()
    driver.get("https://quotes.toscrape.com")
    quotes = driver.find_elements(By.CLASS_NAME, "text")
    for q in quotes:
        print(q.text)
    driver.quit()

Scraping Data Table with Dynamic JavaScript

# Load with Selenium
driver.get("https://www.worldometers.info/world-population/population-by-country/")

# Extract table rows
rows = driver.find_elements(By.XPATH, "//table[@id='example2']/tbody/tr")

for row in rows[:5]:  # first 5 countries
    cols = row.find_elements(By.TAG_NAME, "td")
    print([col.text for col in cols])

driver.quit()

This document showcased the combined power of three essential Python libraries for web data handling:

• Requests: Ideal for basic page fetching and static content.
• Beautiful Soup: Perfect for parsing, navigating, and extracting structured data from HTML.
• Selenium: The go-to tool for interacting with dynamic pages, JavaScript-rendered content, and simulating full user interaction in a browser.

When building robust scraping pipelines, it’s common to start with Requests and Beautiful Soup for efficiency, and only fall back to Selenium when dynamic JavaScript content or interaction is required. Understanding when and how to use these tools allows you to extract data effectively while balancing performance and complexity.

Whether you are building data ingestion pipelines, testing workflows, or creating bots, mastering these libraries will significantly enhance your Python web automation toolkit.