To impersonate a web browser in Python we can set a custom User-Agent header. This makes HTTP requests look like they are coming from a specific web browser.

To impersonate a web browser in Python, we can set a User-Agent header to match that of a specific browser. Here's how you can : Send and Generate User-Agent in Python Requests

From this article you can learn:

  • Can we scrape - websites blocked by Python requests?
  • Why requests is not working in Python?
  • How to resolve 403 forbidden error in rest api Python?
  • python requests avoid captcha
  • python requests prevent bot detection

Python impersonate browser

A better solution is to use a Python library called curl_cffi.

To install library curl-cffi you can run:

pip install curl-cffi

This library offers a range of browsers which can be used with few lines:

from curl_cffi import requests

r = requests.get("https://httpbin.org/", impersonate="chrome99_android")

r.status_code

This gives us code 200.

Workaround for blocked GET requests in Python

If you are trying to extract web data and get frequent bans - then you can try different profiles.

The best one which works for a higher level of protection is - chrome99_android (based on my tests).

import curl_cffi.requests
import random

clink = "https://httpbin.org/"
borwtype=['chrome99','chrome100','chrome101','chrome104','chrome107','chrome110','edge99','edge101','safari15_3','safari15_5']
response = curl_cffi.requests.get(clink,impersonate=borwtype[random.randint(1, len(borwtype) - 1)])
print(response .text)

The code above solves the problem with banned requests and extract data from the most complex sites.