In Python's requests library, using a proxy server allows you to send HTTP requests through different network routes. Proxy servers can help hide real IP addresses, bypass geographic restrictions, or perform load balancing.

What is a proxy?

A proxy server is an intermediate server that sits between the client (your code) and the target server (the server you want to request). When using a proxy server, your request will first be sent to the proxy, which will then forward the request to the target server, and the target server's response will also be returned to you through the proxy.

The use of proxies is very useful in scenarios such as data scraping, accessing restricted websites, and improving privacy protection.

Use a proxy in requests

requests supports HTTP, HTTPS, SOCKS and other proxy requests;

Use HTTP or HTTPS proxy

If you want to use HTTP proxy or HTTPS proxy, you can pass the proxies parameter in any request method to configure a single request:

import requests

proxies = {
  "http": "http://proxy1:8080",
  "https": "https://proxy2.com:8080",
}

#Format with authentication
proxies2 = {
    'http': 'http://user:password@proxy.example.com:8080',
    'https': 'https://user:password@proxy.example.com:8080',
}

response = requests.get("http://example.org", proxies=proxies)

In this example, http requests will be sent through http://proxy1:8080, and https requests will be sent through https://proxy2.com:8080;

Setting proxies through environment variables

In addition to explicitly passing the proxies parameter in the code, you can also configure proxies through environment variables. The requests library will automatically read these environment variables and apply the proxy settings.

Windows environment variable setting command

set HTTP_PROXY=http://proxy.example.com:8080
set HTTPS_PROXY=https://proxy.example.com:8080

Linux, macOS environment variable setting commands

export http_proxy=http://proxy.example.com:8080
export https_proxy=https://proxy.example.com:8080

Use SOCKS proxy

requests supports the proxy function of the SOCKS protocol; this is an optional feature. If you want to use it, you need to install it:

pip install requests[socks]

There is not much difference in usage from HTTP and HTTPS proxies;

proxies = {
    'http': 'socks5://user:password@proxy.example.com:1080',
    'https': 'socks5://user:password@proxy.example.com:1080',
}

Proxy Pool

If you need to change proxies frequently to avoid being blocked by websites, you can create a proxy pool and randomly select proxies. The sample code is as follows:

import requests
import random

proxies_list = [
    {'http': 'http://proxy1.example.com:8080', 'https': 'http://proxy1.example.com:8080'},
    {'http': 'http://proxy2.example.com:8080', 'https': 'http://proxy2.example.com:8080'},
    {'http': 'http://proxy3.example.com:8080', 'https': 'http://proxy3.example.com:8080'},
]

proxies = random.choice(proxies_list)

response = requests.get('https://httpbin.org/ip', proxies=proxies)