Python - Requests Library

Requests Library in Python

Introduction

The Requests library is one of the most powerful and user-friendly HTTP libraries available in Python. It is widely used for making HTTP requests and handling responses in a simple and human-friendly manner. Whether you're interacting with a REST API, scraping web content, or downloading files, Requests makes it easy to send HTTP/1.1 requests and work with responses.

Before Requests, handling HTTP connections in Python required complex and verbose code using modules like urllib and http.client. Requests abstracts these complexities and provides a clean, consistent interface.

Installation

Requests is not included in the standard Python library, so it needs to be installed separately using pip:

pip install requests

To verify installation:

import requests
print(requests.__version__)

Basic GET Request

Sending a Simple GET Request

import requests

response = requests.get("https://www.example.com")
print(response.status_code)
print(response.text)

Inspecting the Response

print(response.status_code)  # HTTP status code
print(response.headers)      # Response headers
print(response.url)          # Final URL after redirection
print(response.content)      # Binary content
print(response.encoding)     # Encoding used by response

Common HTTP Methods

The Requests library supports all major HTTP methods:

• GET: Retrieve data from the server.
• POST: Submit data to be processed to a specified resource.
• PUT: Update an existing resource.
• DELETE: Delete a specified resource.
• HEAD: Same as GET but without response body.
• OPTIONS: Describe the communication options for the target resource.

POST Example

data = {'username': 'john', 'password': 'doe'}
response = requests.post("https://httpbin.org/post", data=data)
print(response.text)

PUT Example

data = {'name': 'Updated Name'}
response = requests.put("https://httpbin.org/put", data=data)

DELETE Example

response = requests.delete("https://httpbin.org/delete")

Passing Parameters

URL Parameters

params = {'search': 'python', 'page': 2}
response = requests.get("https://example.com/search", params=params)
print(response.url)

POST Data

payload = {'key1': 'value1', 'key2': 'value2'}
response = requests.post("https://httpbin.org/post", data=payload)
print(response.json())

Working with JSON

Sending JSON in a Request

json_data = {'name': 'Alice', 'email': 'alice@example.com'}
response = requests.post("https://httpbin.org/post", json=json_data)
print(response.json())

Parsing JSON in a Response

response = requests.get("https://api.github.com")
data = response.json()
print(data['current_user_url'])

Custom Headers

You can modify request headers by passing a dictionary:

headers = {
    'User-Agent': 'my-app/0.0.1',
    'Accept': 'application/json'
}
response = requests.get("https://httpbin.org/headers", headers=headers)
print(response.json())

Timeouts

Timeouts help avoid hanging requests. Always set a timeout:

try:
    response = requests.get("https://httpbin.org/delay/5", timeout=3)
except requests.exceptions.Timeout:
    print("The request timed out")

Handling Cookies

Sending Cookies

cookies = {'session_id': '123abc'}
response = requests.get("https://httpbin.org/cookies", cookies=cookies)
print(response.text)

Reading Cookies

response = requests.get("https://httpbin.org/cookies/set?name=value")
print(response.cookies)

Sessions

The Session object allows you to persist cookies and headers across multiple requests.

Creating a Session

s = requests.Session()
s.headers.update({'User-Agent': 'my-app'})
s.get("https://httpbin.org/cookies/set/sessioncookie/123456789")
r = s.get("https://httpbin.org/cookies")
print(r.text)

Authentication

Requests provides support for HTTP Basic and Digest authentication.

Basic Authentication

from requests.auth import HTTPBasicAuth

response = requests.get("https://httpbin.org/basic-auth/user/pass", auth=HTTPBasicAuth('user', 'pass'))
print(response.status_code)

Digest Authentication

from requests.auth import HTTPDigestAuth

response = requests.get("https://httpbin.org/digest-auth/auth/user/pass", auth=HTTPDigestAuth('user', 'pass'))
print(response.status_code)

Redirection

By default, Requests automatically follows redirects:

response = requests.get("http://github.com")
print(response.url)  # Final URL
print(response.history)  # List of Response objects for each redirect

Disabling Redirects

response = requests.get("http://github.com", allow_redirects=False)
print(response.status_code)

Uploading Files

files = {'file': open('test.txt', 'rb')}
response = requests.post("https://httpbin.org/post", files=files)
print(response.text)

Downloading Files

url = 'https://example.com/image.jpg'
r = requests.get(url)
with open('image.jpg', 'wb') as f:
    f.write(r.content)

Proxies

proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080',
}
response = requests.get("http://example.com", proxies=proxies)

Error Handling

Handling Exceptions

try:
    response = requests.get("https://example.com", timeout=5)
    response.raise_for_status()
except requests.exceptions.HTTPError as errh:
    print("Http Error:", errh)
except requests.exceptions.ConnectionError as errc:
    print("Error Connecting:", errc)
except requests.exceptions.Timeout as errt:
    print("Timeout Error:", errt)
except requests.exceptions.RequestException as err:
    print("OOps: Something Else", err)

SSL Certificate Verification

# Default (verifies SSL)
response = requests.get("https://example.com")

# Disable SSL verification (not recommended)
response = requests.get("https://example.com", verify=False)

Advanced: Custom Transport Adapters

from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount("https://", HTTPAdapter(max_retries=3))
response = s.get("https://example.com")

Working with APIs

The Requests library is heavily used to interact with APIs such as RESTful web services.

Example: GitHub API

url = "https://api.github.com/users/octocat"
response = requests.get(url)
data = response.json()
print("Username:", data['login'])
print("ID:", data['id'])
print("URL:", data['html_url'])

Rate Limiting

Many APIs enforce rate limits. Respect these by checking headers like:

response.headers.get('X-RateLimit-Remaining')
response.headers.get('Retry-After')

Performance Tips

• Reuse Session objects for performance gains.
• Use timeouts to prevent hanging.
• Handle exceptions gracefully.
• Avoid unnecessary redirects.

Common Use Cases

• Fetching web page data for scraping
• Automating form submission
• Consuming REST APIs
• Testing endpoints in development

Limitations

• No built-in support for JavaScript rendering
• Not ideal for browser automation (use Selenium for that)

Requests vs urllib

Although both can handle HTTP, Requests is more concise and easier to use than urllib:

Conclusion

The Python Requests library is a simple yet powerful tool for interacting with HTTP servers. It abstracts the complexities of making requests behind a clean and user-friendly API. Whether you are fetching data from a website, communicating with a RESTful API, uploading files, or managing sessions and cookies, Requests makes your life easier.

Its wide adoption and extensive documentation make it a favorite among Python developers. For more advanced needs like browser automation or scraping JavaScript-heavy websites, consider integrating it with other tools such as BeautifulSoup, Selenium, or using frameworks like Scrapy.