How to Send and Generate User-Agent in Python Requests

Softhints -

In Python, we can use the requests library to make HTTP requests with custom User-Agent headers. The User-Agent header is used to identify the client making the request and can be useful for websites and APIs that may have different responses or behavior based on the client's user-agent.

Python send User-Agent header

Here's how you can set a custom User-Agent header using the requests library:

import requests

custom_user_agent = "Mozilla/5.0  Chrome/39.0.2171.95 Safari/537.36"
headers = {"User-Agent": custom_user_agent}
url = "https://httpbin.org/"

response = requests.get(url, headers=headers)

if response.status_code == 200:
	print("Request successful")
	print(response.text)
else:
	print(f"Request failed with status code {response.status_code}")

How does it work:

  • define your custom User-Agent
  • make a dictionary with the User-Agent header
  • HTTP GET request with the custom User-Agent
  • check the response
  • process the response

Python Fake User-Agent

Python offers libraries fake-useragent. This library offers up-to-date simple user agent faker with real world database.

To install the library we can do:

pip install fake-useragent

To use this library:

from fake_useragent import UserAgent
import requests
   

ua = UserAgent()
print(ua.chrome)
header = {'User-Agent':str(ua.chrome)}
print(header)
url = "https://httpbin.org/"
htmlContent = requests.get(url, headers=header)
print(htmlContent)

which give us the following user agent:

{'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36'}

Python Mobile, Tablet or Desktop User Agents

Another useful Python library which helps to identify and analyze mobile, tablet or Desktop user agents is: user-agents.

Library is installed by - pip install user-agents and can be used by:

from user_agents import parse

# iPhone's user agent string
ua_string = 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B179 Safari/7534.48.3'
user_agent = parse(ua_string)

# Accessing user agent's browser attributes
user_agent.browser  # returns Browser(family=u'Mobile Safari', version=(5, 1), version_string='5.1')
user_agent.browser.family  # returns 'Mobile Safari'
user_agent.browser.version  # returns (5, 1)
user_agent.browser.version_string   # returns '5.1'

Summary

Make sure to use diverse and up to date User-Agents. User agents help in data extraction and web scraping.

Keep in mind that it's essential to respect website policies and not use User-Agent headers to impersonate other clients or perform actions that violate the website's terms of service.