Python - Selenium

Python Selenium

Introduction to Selenium

Selenium is a powerful browser automation tool used for testing web applications, web scraping, and automating repetitive web tasks. It supports multiple languages including Python, Java, C#, JavaScript, and Ruby, and can control browsers such as Chrome, Firefox, Safari, and Edge via WebDriver.

This document explores Selenium in depth: installation, setup, locating elements, interacting with elements, waits, advanced usage like headless browsing, handling alerts, frames, navigation, browser options, and integrating with testing frameworks.

Installation

Install Selenium Package

pip install selenium

Download WebDriver

# Example for Chrome:
# Download chromedriver matching Chrome version from:
# https://chromedriver.chromium.org/downloads
# Place the executable in PATH or specify its location in your code.

Setting Up WebDriver

Basic ChromeDriver Setup

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.example.com")
print(driver.title)
driver.quit()

FirefoxDriver Setup

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.example.com")
driver.quit()

Locating Elements

By ID, Name, Class, Tag

from selenium.webdriver.common.by import By

element = driver.find_element(By.ID, "username")
element = driver.find_element(By.NAME, "password")
element = driver.find_element(By.CLASS_NAME, "btn")
element = driver.find_element(By.TAG_NAME, "a")

By CSS Selector

element = driver.find_element(By.CSS_SELECTOR, "div.main > p.intro")

By XPath

element = driver.find_element(By.XPATH, "//div[@id='main']//a[text()='Next']")

Multiple Elements

elements = driver.find_elements(By.CLASS_NAME, "list-item")
for el in elements:
    print(el.text)

Interacting with Elements

Clicking

button = driver.find_element(By.ID, "submit-btn")
button.click()

Typing Text

search_box = driver.find_element(By.NAME, "q")
search_box.send_keys("Selenium Python")
search_box.submit()

Clearing Input

search_box = driver.find_element(By.NAME, "q")
search_box.clear()

Select & Dropdowns

from selenium.webdriver.support.ui import Select

select = Select(driver.find_element(By.ID, "options"))
select.select_by_visible_text("Option 1")
select.select_by_value("option2")

Checkboxes & Radio Buttons

checkbox = driver.find_element(By.ID, "agree")
if not checkbox.is_selected():
    checkbox.click()

Waiting for Elements

Implicit Wait

driver.implicitly_wait(10)  # seconds
driver.get("https://www.example.com")
element = driver.find_element(By.ID, "my-id")

Explicit Wait

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, "submit-btn")))

Browser Navigation & Actions

Browser Navigation

driver.get("https://www.start.com")
driver.get("https://www.nextpage.com")
driver.back()
driver.forward()

Refreshing Page

driver.refresh()

Actions: Hover, Drag & Drop

from selenium.webdriver import ActionChains

actions = ActionChains(driver)
element_to_hover = driver.find_element(By.ID, "menu")
actions.move_to_element(element_to_hover).perform()

source = driver.find_element(By.ID, "drag")
target = driver.find_element(By.ID, "drop")
actions.drag_and_drop(source, target).perform()

Handling Alerts, Frames, Windows

Alerts

alert = driver.switch_to.alert
print(alert.text)
alert.accept()    # or alert.dismiss()

Frames / iFrames

driver.switch_to.frame("frame_name")
# do stuff inside frame
driver.switch_to.default_content()

Multiple Windows / Tabs

main_window = driver.current_window_handle
driver.find_element(By.LINK_TEXT, "Open new tab").click()
for handle in driver.window_handles:
    if handle != main_window:
        driver.switch_to.window(handle)
        break
driver.close()
driver.switch_to.window(main_window)

Screenshots & Page Source

Taking Screenshot

driver.save_screenshot("screenshot.png")

Getting HTML Source

html = driver.page_source
with open("page.html", "w", encoding="utf-8") as f:
    f.write(html)

Headless Browsing

Chrome Headless Mode

from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

Firefox Headless Mode

from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)

Advanced Usage

Execute JavaScript

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Network Conditions & Performance Tuning

# Requires using advanced settings or devtools protocol
# Example is browser/driver dependent and advanced

Integration with PyTest

# conftest.py
import pytest
from selenium import webdriver

@pytest.fixture
def driver():
    driver = webdriver.Chrome()
    yield driver
    driver.quit()

# test_example.py
def test_title(driver):
    driver.get("https://example.com")
    assert "Example Domain" in driver.title

Web Scraping with Selenium

Infinite Scrolling

import time

driver.get("https://example.com/scroll")
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Downloading Dynamic Content

driver.get("https://example.com/data-page")
table = driver.find_element(By.ID, "data-table")
rows = table.find_elements(By.TAG_NAME, "tr")
for row in rows:
    cells = row.find_elements(By.TAG_NAME, "td")
    print([cell.text for cell in cells])

Headless Testing with CI/CD

Integrate headless browser tests into continuous integration systems like Jenkins, GitLab CI, Travis CI, or GitHub Actions. Use base images or containers with browsers and drivers installed, run Selenium scripts in headless mode, and report results.

Best Practices & Troubleshooting

Explicit Waits over Implicit

Prefer explicit waits to avoid unpredictable timeouts and synchronization issues.

Keep WebDriver up to Date

Match driver with browser version to avoid compatibility issues.

Use Unique Identifiers

Always prefer locating elements by unique IDs or meaningful CSS/XPath to minimize brittleness.

Manage Sessions Cleanly

Quit driver in teardown routines or fixtures to avoid locking sessions or consuming resources.

Logging and Screenshots on Failure

Capture screenshots, HTML dumps, and logs when tests fail to aid debugging.

Python’s Selenium binds to a robust ecosystem for automating browsers for testing, scraping, or workflows. From basic navigation and element interaction to headless mode, advanced waits, JavaScript execution, multi-window handling, and CI/CD integration, Selenium provides a full-featured API. When combined with frameworks like pytest, it supports scalable test suites and resilient automation.

Mastering Selenium involves learning to locate elements accurately, managing async page loads, handling dynamic content, and organizing tests into maintainable suites. With careful design, explicit waits, clean session controls, and integration into CI/CD pipelines, Selenium empowers reliable, high-quality web automation for development, testing, and scraping needs.