If you’re using cURL to test APIs or scrape data, that’s a great start. But when you want to scale those calls into automated scripts or integrate them with scraping pipelines, Python is your best friend.
In this guide, we’ll show you how to convert cURL commands into Python code using the requests library. Also, adding rotating proxy support from Proxying.io helps avoid blocks, stay anonymous, and geotarget your requests.
Understanding the Structure of a cURL Command
cURL is usually preinstalled on most laptops, so you can start right away. If not, you can learn how to install it in our guide on using cURL with a proxy.
.
Let’s start with the example of a simple cURL request:
curl https://httpbin.org/getThis command sends a GET request to the given URL, but it is limited.
Equivalent in Python:
import requests
response = requests.get("https://httpbin.org/get")
print(response.text)Now you are working in a language where you can add loops, conditions, sessions, and proxy handling.
Converting Headers, POST Data, and Other Options
Example cURL with headers and POST data:
curl -X POST https://httpbin.org/post \
-H "Content-Type: application/json" \
-H "Authorization: Bearer TOKEN123" \
-d '{"name":"John"}'Converted to Python:
import requests
url = "https://httpbin.org/post"
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer TOKEN123"
}
data = '{"name":"John"}'
response = requests.post(url, headers=headers, data=data)
print(response.json())Python allows better error handling and response parsing which is something cURL alone doesn’t offer.
Adding Proxy Support in Python Requests
Let’s now integrate a proxy from Proxying.io into your requests.
proxies = {
"http": "http://USERNAME:[email protected]:8000",
"https": "http://USERNAME:[email protected]:8000",
}
response = requests.get("https://httpbin.org/ip", proxies=proxies)
print(response.json())Remember: Use residential or rotating proxies to bypass blocks and ensure high success rates for your requests.
Sessions and Persistent Headers with Requests
Python’s request.Session() lets you reuse TCP connections and keep consistent headers or cookies across multiple requests.
session = requests.Session()
session.proxies = proxies
session.headers.update({
"User-Agent": "Mozilla/5.0",
"Authorization": "Bearer TOKEN123"
})
r1 = session.get("https://httpbin.org/headers")
r2 = session.post("https://httpbin.org/post", json={"name": "John"})
print(r1.status_code, r2.status_code)This becomes critical when scraping authenticated or JavaScript-heavy websites.
Handling Redirects, Timeouts, and Retries
Robust scraping requires more than just making requests; you must handle failures gracefully
try:
response = requests.get("https://httpbin.org/redirect/1",
proxies=proxies,
timeout=10,
allow_redirects=True)
print(response.status_code)
except requests.exceptions.RequestException as e:
print(f"Request failed: {e}")- Use allow_redirects=False if you want manual control.
- Set timeouts to avoid hanging scripts.
- Use libraries like tenacity or custom logic for retries.
Rotating Proxies with Retry Logic
When scraping at scale, you will want rotating IPs and retry logic.
import random
from time import sleep
proxy_list = [
"http://user:[email protected]:8000",
"http://user:[email protected]:8000",
"http://user:[email protected]:8000",
]
def get_random_proxy():
return {"http": random.choice(proxy_list), "https": random.choice(proxy_list)}
for i in range(5):
try:
proxy = get_random_proxy()
r = requests.get("https://httpbin.org/ip", proxies=proxy, timeout=8)
print(r.json())
except requests.RequestException:
print("Proxy failed, retrying...")
sleep(2)Rotating proxies + retry logic = fewer blocks and better coverage.
Working with Cookies and Authentication
Many APIs and websites require session-based authentication, something not easily done with cURL but much more flexible in Python.
session = requests.Session()
# First, log in and store cookies
login_payload = {
"username": "myuser",
"password": "mypassword"
}
login_url = "https://example.com/login"
session.post(login_url, data=login_payload)
# Now, access a protected page with cookies
dashboard_url = "https://example.com/dashboard"
response = session.get(dashboard_url)
print(response.text)This approach lets you stay logged in, follow redirects, and reuse session cookies, making it ideal for scraping authenticated content.
Scraping APIs and Websites at scale
Once you have converted cURL to Python and added proxies, the next challenge is scaling your operation. Python gives you all the tools for this:
- Concurrency with asyncio, aiohttp, or concurrent.futures.
- Job scheduling with cron or APScheduler.
- Data output into CSV, JSON, or a database.
- Proxy rotation to avoid blocks.
For example:
from concurrent.futures import ThreadPoolExecutor
urls = [f"https://httpbin.org/get?page={i}" for i in range(10)]
def fetch(url):
return requests.get(url, proxies=get_random_proxy(), timeout=8).json()
with ThreadPoolExecutor(max_workers=5) as executor:
results = list(executor.map(fetch, urls))
for result in results:
print(result)Conclusion
While cURL is great for quick tests, switching to Python lets you do much more. With the requests library, you can handle sessions, and authentication, manage retries, and use rotating proxies to avoid getting blocked.
Python is a better choice when you want to automate tasks, scrape data at scale, or rebuild reliable tools.
You can try it for free by signing up at Proxying.io. You will get 25MB of proxy usage without needing a credit card. required
