To impersonate a web browser in Python we can set a custom User-Agent header. This makes HTTP requests look like they are coming from a specific web browser.
To impersonate a web browser in Python, we can set a User-Agent header to match that of a specific browser. Here's how you can : Send and Generate User-Agent in Python Requests
From this article you can learn:
- Can we scrape - websites blocked by Python requests?
- Why requests is not working in Python?
- How to resolve 403 forbidden error in rest api Python?
- python requests avoid captcha
- python requests prevent bot detection
Python impersonate browser
A better solution is to use a Python library called curl_cffi.
To install library
curl-cffi you can run:
pip install curl-cffi
This library offers a range of browsers which can be used with few lines:
from curl_cffi import requests r = requests.get("https://httpbin.org/", impersonate="chrome99_android") r.status_code
This gives us code 200.
Workaround for blocked GET requests in Python
If you are trying to extract web data and get frequent bans - then you can try different profiles.
The best one which works for a higher level of protection is -
chrome99_android (based on my tests).
import curl_cffi.requests import random clink = "https://httpbin.org/" borwtype=['chrome99','chrome100','chrome101','chrome104','chrome107','chrome110','edge99','edge101','safari15_3','safari15_5'] response = curl_cffi.requests.get(clink,impersonate=borwtype[random.randint(1, len(borwtype) - 1)]) print(response .text)
The code above solves the problem with banned requests and extract data from the most complex sites.