When using Python's requests library, you may sometimes encounter network instability, connection interruptions and other problems, so it is necessary to implement a reconnection mechanism. The requests library itself does not directly provide a retry mechanism, but it can be implemented by combining urllib3's Retry class and requests.adapters.HTTPAdapter.

Retry class (urllib3.util.retry.Retry)

Retry class is a tool in the urllib3 library for setting the retry logic of requests. It allows you to specify parameters such as the number of retries when a request fails, which error types trigger retries, and the retry interval. Retry class can be used to implement a powerful network request fault tolerance mechanism.

Common parameters:

  • total: Total number of retries. Set to None or 0 to not retry.
  • read: Number of retries when read fails.
  • connect: Number of retries when connection fails.
  • backoff_factor: Factor used to control the retry interval. The first retry interval is backoff_factor, the second retry interval is backoff_factor * 2^1, the third is backoff_factor * 2^2, and so on (exponential backoff algorithm).
  • status_forcelist: A list of status codes that trigger a retry when the request returns these HTTP status codes (such as 500, 502, 503, etc. server errors).
  • allowed_methods: Specifies which HTTP methods are allowed to be retried (e.g. GET, POST). By default, retries are only performed for idempotent methods (e.g. GET, PUT, DELETE).
  • raise_on_redirect: If set to True, a MaxRetryError will be thrown after the number of redirects has been exceeded.
  • raise_on_status: If set to True, a MaxRetryError will be thrown after the number of status code triggers has been exceeded.

Retry usage example:

from urllib3.util.retry import Retry

retry_strategy = Retry(
    total=3,  # Retry 3 times in total
    status_forcelist=[500, 502, 503, 504],  # These status codes will trigger a retry
    method_whitelist=["HEAD", "GET", "OPTIONS"],  # Methods that allow retries
    backoff_factor=1  # The retry interval is 1 second, increasing gradually
)

HTTPAdapter class (requests.adapters.HTTPAdapter)

HTTPAdapter is an adapter in the requests library that manages the parameters and behaviors of the underlying HTTP connection, including connection pools, retry logic, and how to transmit HTTP requests.

By default, requests use the standard HTTPAdapter, which does not have a retry function. However, we can implement a retry mechanism by customizing the HTTPAdapter and combining it with the Retry class.

Common parameters:

  • max_retries: Set the retry strategy. This parameter can directly pass a Retry object.
  • pool_connections: The maximum number of connections in the connection pool. The default is 10. It can reduce the repeated TCP handshake time and improve performance.
  • pool_maxsize: The maximum number of connections in the connection pool. If it exceeds the number, the request will be blocked and wait for an idle connection.
  • pool_block: If the connection pool is full, whether to block and wait. The default is False, which means that an exception will be thrown after exceeding the connection pool.

HTTPAdapter usage example:

from requests.adapters import HTTPAdapter

adapter = HTTPAdapter(max_retries=retry_strategy)
session = requests.Session()
session.mount("http://", adapter)
session.mount("https://", adapter)

Reconnection

You can add a request retry strategy to your HTTP session by passing a Retry instance as a parameter to a HTTPAdapter. 

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Define retry strategy
retry_strategy = Retry(
    total=3,  # Total number of retries
    status_forcelist=[500, 502, 503, 504],  # Which status codes to retry for
    backoff_factor=1  # The factor of the retry interval, increasing gradually
)

# Create HTTPAdapter and set retry strategy
adapter = HTTPAdapter(max_retries=retry_strategy)

# Create a requests session and mount the adapter
session = requests.Session()
session.mount("http://", adapter)
session.mount("https://", adapter)

# Make a request
response = session.get('http://example.com',timeout=5)
print(response.status_code)