How to Set Custom User-Agent Headers in Python Requests   

In this short tutorial, you'll see how to set custom User-Agent headers in Python using the requests library. Whether you're web scraping, accessing APIs, or testing web applications, setting proper User-Agent strings prevents 403 Forbidden errors, 429 rate limits, and helps your requests appear as legitimate browser traffic.

The User-Agent header identifies your client to web servers. Many sites block the default Python requests User-Agent, making custom headers essential for successful HTTP requests.

1. Understanding User-Agent Headers

A User-Agent header tells the server what browser, device, and operating system is making the request. Websites use this to serve appropriate content, detect bots, and enforce access policies.

Default User-Agent in Requests

The requests library uses a default User-Agent that clearly identifies itself as Python, which many sites block.

import requests

response = requests.get('https://httpbin.org/headers')
data = response.json()

print(f"Default User-Agent: {data['headers']['User-Agent']}")

Output Result:

Default User-Agent: python-requests/2.31.0

Why this matters: Sites like Google, Yahoo Finance, and Amazon often block or rate-limit the default python-requests User-Agent to prevent automated scraping.

2. Setting a Custom User-Agent

The simplest way to set a custom User-Agent is passing a headers dictionary to the get() method.

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}

response = requests.get('https://httpbin.org/headers', headers=headers)
data = response.json()

print(f"Custom User-Agent: {data['headers']['User-Agent']}")

Output Result:

Custom User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

Real-world use: This Chrome User-Agent makes your request appear as a Windows Chrome browser, bypassing most bot detection systems.

3. Using Session Objects for Persistent Headers

For multiple requests, use Session objects to set headers once and reuse them automatically.

import requests

session = requests.Session()
session.headers.update({
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
})

response1 = session.get('https://httpbin.org/headers')
response2 = session.get('https://httpbin.org/user-agent')

print(f"Session User-Agent: {response2.json()['user-agent']}")

Output Result:

Session User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36

Performance benefit: Sessions reuse TCP connections, making multiple requests 30-50% faster than individual requests.

4. Complete Headers for Maximum Compatibility

For production web scraping, combine User-Agent with other browser headers to avoid detection.

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Referer': 'https://www.google.com/'
}

response = requests.get('https://httpbin.org/headers', headers=headers)
print(f"Status: {response.status_code}")

Output Result:

Status: 200

5. Best Practices for User-Agent Strings

Popular User-Agent Strings

Tip: Rotate User-Agents randomly for large-scale scraping to avoid pattern detection.

6. Error Handling with Custom Headers

Always implement error handling when setting custom headers to catch connection errors and HTTP errors.

import requests

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}

try:
    response = requests.get('https://api.github.com/users/github', headers=headers, timeout=5)
    response.raise_for_status()
    print(f"Success: {response.json()['login']}")
except requests.exceptions.RequestException as e:
    print(f"Error: {e}")

Output Result:

Success: github

7. Using with JSON API Requests

Combine User-Agent with Content-Type headers for JSON API requests.

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Content-Type': 'application/json'
}

payload = {'title': 'Test', 'body': 'Content', 'userId': 1}
response = requests.post('https://jsonplaceholder.typicode.com/posts', json=payload, headers=headers)

print(f"Status: {response.status_code}, ID: {response.json()['id']}")

Output Result:

Status: 201, ID: 101