How to use proxies with Python requests, httpx and aiohttp

The proxy URL#

Every Python HTTP library takes a proxy as a URL. With Proxy Cyber the URL carries your credentials and the targeting parameters in the username; the hostname is the gateway and the port picks the protocol.

# Rotating: a new United States household on every new connection
http://USER-cc-us:PASS@res.proxy-cyber.com:9000

# Sticky: one Berlin household held for 30 minutes under the id "job42"
http://USER-cc-de-city-berlin-sid-job42-ttl-30m:PASS@res.proxy-cyber.com:9000

# SOCKS5 with remote DNS (socks5h), same parameters
socks5h://USER-cc-us:PASS@res.proxy-cyber.com:9001

requests#

Connection reuse and IP assignment are different controls. A client session reuses sockets; only the assigned gateway determines rotation and sticky-session behavior. Use the session parameters supplied with your service, and verify the exit over several requests. Closing a socket or sending Connection: close does not guarantee a different IP.

import requests

PROXY = "http://USER-cc-us:PASS@res.proxy-cyber.com:9000"
proxies = {"http": PROXY, "https": PROXY}

# One request, one exit
r = requests.get("https://api.ipify.org", proxies=proxies, timeout=30)
print(r.text)

# A sticky flow: same exit for every request of this session
sticky = "http://USER-cc-de-city-berlin-sid-cart42-ttl-15m:PASS@res.proxy-cyber.com:9000"
s = requests.Session()
s.proxies = {"http": sticky, "https": sticky}
s.get("https://example.com/product/1", timeout=30)
s.post("https://example.com/cart", json={"id": 1}, timeout=30)

httpx#

httpx takes a single proxy= argument (older versions: proxies=). It supports HTTP/2 to the destination through the CONNECT tunnel and works the same way in async code.

import httpx

PROXY = "http://USER-cc-us:PASS@res.proxy-cyber.com:9000"

with httpx.Client(proxy=PROXY, timeout=30) as client:
    print(client.get("https://api.ipify.org").text)

# async, one sticky exit per task
import asyncio

async def fetch(sid: str, url: str) -> str:
    proxy = f"http://USER-cc-us-sid-{sid}-ttl-10m:PASS@res.proxy-cyber.com:9000"
    async with httpx.AsyncClient(proxy=proxy, timeout=30) as client:
        return (await client.get(url)).text

asyncio.run(fetch("task1", "https://api.ipify.org"))

aiohttp#

aiohttp takes the proxy per request or per session and speaks HTTP proxies natively; SOCKS5 needs the aiohttp-socks connector.

import aiohttp, asyncio

PROXY = "http://USER-cc-us:PASS@res.proxy-cyber.com:9000"

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get("https://api.ipify.org", proxy=PROXY, timeout=aiohttp.ClientTimeout(total=30)) as r:
            print(await r.text())

asyncio.run(main())

Rotation, keep-alive and retries#

• A new TCP connection is a new transport connection, not proof of a new exit. The gateway’s assigned rotation mode decides how that connection is routed.
• Use a distinct session identifier for each independent workflow, where the assigned gateway supports it. Confirm the permitted duration and location parameters in your connection details. A residential device can disconnect before the requested duration; a sticky session is not a guarantee of the same IP for days. For a fixed address, compare a dedicated static product.
• Classify the response before retrying. A 403 may indicate an access rule that another IP will not resolve. For 429, respect Retry-After and reduce the request rate; use bounded backoff for temporary connection or server failures. Retry state-changing operations only when you can prevent duplicate actions.
• Set timeouts to the exit type: 15 s connect and 45 s read for residential, more for mobile.

Saving gigabytes#

Residential traffic is billed per GB in both directions. Ask for compressed responses (Accept-Encoding: gzip, br is on by default in these libraries), prefer JSON endpoints to HTML, and never download images, fonts or media unless they are the data. A 350 KB page is cheap; a 4 MB page with assets is not.

Questions people ask#