When using requests.get() in Python, a common mistake is assuming the timeout parameter limits the total request time. In reality, timeout controls how long requests waits for a connection and for data chunks — not the full response duration.

We can control connection and response wait time for python requests:

(1) Connection

response = requests.get(url, timeout=10)

(2) Connection and response wait time

response = requests.get(url, timeout=(3, 10))

This can lead to scripts hanging indefinitely if a server is slow or stops sending data.

1. Correct Way to Use Timeouts

Use a tuple timeout to separately control:

  • Connection timeout
  • Read/response timeout
python
Copy code
import requests

url = "https://httpbin.org/delay/5"

response = requests.get(url, timeout=(3, 10))
print(response.text)

Meaning:

  • 3 seconds to establish the connection
  • 10 seconds to wait for data between bytes

2. Python Exceptions ReadTimeout

Python will raise exception if the connection timeouts:

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='httpbin.org', port=443): Read timed out. (read timeout=2)

3. Why This Matters

If you use:

requests.get(url, timeout=10)

This sets both connect and read timeouts to 10 seconds, but a slow streaming response can still keep the connection alive indefinitely.

4. Enforcing a Hard Timeout (Advanced)

For strict total-time limits, you can:

  • Use signal (Unix only)
  • Run requests in a separate thread/process
  • Use async libraries like httpx with timeouts

How to Timeout HTTP Requests in Linux

curl --max-time 10 https://example.com

httpx

import httpx

# Using the top-level API:
httpx.get("https://httpbin.org/delay/7", timeout=10.0)

# Using a client instance:
with httpx.Client() as client:
    client.get("http://example.com/api/v1/example", timeout=10.0)

If the timeout is passed exception is raised:

httpx.ReadTimeout: The read operation timed out

5. Best Practice

  • Always use tuple timeouts
  • Handle exceptions
python
Copy code
try:
    r = requests.get(url, timeout=(3, 10))
except requests.exceptions.Timeout:
    print("Request timed out")

6. Handling exceptions and partial requests

To ensure you still capture partial data if a timeout occurs, wrap the request in a try/except block:

import requests

try:
    response = requests.get("https://httpbin.org/delay/7", timeout=10)
    data = response.text  # full text (if received)
    print(data)
except requests.exceptions.Timeout as e:
    print("Request timed out, partial response may be available.")

    # If a response was partially received
    if hasattr(e, "response") and e.response is not None:
        print(e.response.text)

7. Control With stream=True

If you want even more control (e.g., read chunks and impose your own timeout logic):

import requests

with requests.get("https://example.com", timeout=5, stream=True) as r:
    for chunk in r.iter_content(chunk_size=1024):
        # process chunk …
        print(chunk)

This lets you manage large downloads and timeouts while still reading data incrementally.

Resource