Three hundred requests in, my crawler went quiet. Not an error, not a crash, just page after page of the same generic “Access Denied” template where product data used to be. I’d been so focused on the parsing logic that I never stopped to think about how the request itself looked from the other side.
That’s the part most people skip, and it’s the part that actually determines whether your crawler survives past the first few hundred pages. Getting blocked usually has very little to do with your code being wrong. It has everything to do with your traffic looking like traffic a machine would send. Let’s walk through what actually gets you flagged and how to crawl a website without getting blocked.
How Websites Detect and Block Web Crawlers
Modern anti-bot systems don’t check one thing; they check a stack of things at once. IP reputation gets scored against known datacenter ranges and abuse databases. Request volume and timing get compared against what a human could plausibly click through. TLS handshakes get fingerprinted because the way your HTTP client negotiates a connection is surprisingly distinct from how Chrome does it. Browser fingerprints, headers, and behavioral patterns all feed into the same decision.
You’ll usually see the block in one of a few familiar shapes:
- 403 Forbidden: flat rejection, no ambiguity.
- 429 Too Many Requests: You tripped a rate limit.
- Cloudflare 1020: The WAF decided your request doesn’t look human.
- A CAPTCHA wall: soft block, still recoverable if you handle it right.
- Shadow-banning: the sneaky one. Server returns 200 with a normal-looking page, but the content’s been stripped or swapped for junk. Your crawler thinks it succeeded. It didn’t.
That last one is what actually caught me. No error to catch, no status code to branch on, just empty data quietly sailing past my validation checks.
Plan Before You Crawl
I used to skip straight to writing the parser. I don’t anymore.
Check robots.txt first. It won’t stop a determined scraper technically, but ignoring it is a fast way to get your IP range blacklisted outright, and some sites treat robots.txt violations as grounds for legal notice. Read the Terms of Service too, specifically anything about automated access.
Decide early whether you actually need to parse HTML at all. A lot of sites load their data through a JSON API behind the scenes, and hitting that endpoint directly is faster, cleaner, and far less likely to trip fingerprint checks than rendering a full page. More on that later. If you’re still fuzzy on how web scraping actually works end-to-end, get that settled before you write a single line of parser code; it shapes every decision after this one.
Run a small test batch, maybe 20 to 50 requests, before you scale up. And don’t hold long-lived sessions open longer than the task needs. A session that stays alive for six hours doing nothing but idling between requests is its own kind of red flag.
Rotate IP Addresses and Manage Proxies
A single IP sending hundreds of sequential requests is the easiest pattern in the world to flag. This is where IP rotation actually earns its keep, and the three proxy types behind it behave very differently under load.
Datacenter proxies. Fast, cheap, and the first thing most anti-bot systems check against known hosting-provider IP ranges. Fine for low-sensitivity targets, gets burned quickly on anything with real protection.
Residential proxies. Traffic routes through real ISP-assigned household connections, so it reads as an ordinary visitor rather than a server. Slower per request, but this is what holds up against sites that actively fingerprint datacenter ranges.
ISP proxies. A middle ground: datacenter-level speed with an IP that’s registered to an actual internet service provider rather than a hosting company. Useful when you need consistency across a session without paying the residential latency cost. If you’re still weighing the two, our ISP vs datacenter proxies comparison lays out the trade-off in more detail.
Whichever pool you pick, watch it. Log error rates and latency per IP as you go, since a proxy that was clean an hour ago can get flagged mid-run without warning.
Use Realistic Headers and Browser Fingerprints
Your IP can be perfectly clean, and you’ll still get flagged if your headers don’t match what a browser actually sends. This is the mistake I see most often: people rotate IPs religiously and leave the default python-requests User-Agent sitting right there in every request.
Send a full, realistic header set. Accept, Accept-Language, Referer, and the Sec-Fetch-* family all need to be present and consistent with each other, not just a bare User-Agent string bolted onto an otherwise empty request.
And consistency matters more than any single header. Your User-Agent, TLS fingerprint, timezone, and screen resolution all need to describe the same device. A User-Agent claiming Chrome on Windows paired with a TLS handshake that fingerprints as Python’s urllib3 is an instant tell, and it’s one of the fastest ways to get caught even with a perfectly rotated proxy pool underneath it. If juggling all of that by hand feels error-prone, an anti-detect browser manages fingerprint consistency across sessions for you.
Craft Human-Like Request Patterns
Real users don’t request 200 pages a second apart down to the millisecond. A few things close that gap:
- Randomize delays between requests using something like a Gaussian distribution instead of a fixed sleep, so timing doesn’t form a detectable pattern.
- Follow a plausible navigation path through the site’s actual structure, category page to listing to detail page, instead of hitting URLs in sequential or alphabetical order.
- Avoid predictable ID progressions like /product/1001, /product/1002, /product/1003.
- Back off dynamically when you notice server latency climbing. A human would slow down too if a site felt sluggish; your crawler should behave the same way.
When to Use Headless Browsers
You don’t need a full browser for every job, and you should default to not using one until something forces your hand.
You do need one when the content only renders after JavaScript runs, common on React, Vue, and Angular-heavy sites. Same for complex authentication flows with multi-step logins, and for visual challenges like slider CAPTCHAs that require actual interaction.
The tradeoff is real. Headless browsers eat far more CPU and memory than a plain HTTP client, and out of the box, tools like Playwright and Puppeteer leave detectable fingerprints of their own unless you apply stealth patches. Reach for a headless browser only when a plain request genuinely can’t get you the data.
Handle CAPTCHAs, Honeypots, and Other Traps
CAPTCHAs come in a few flavors now. Traditional reCAPTCHA v2 and v3 are still common. Cloudflare Turnstile runs invisibly in the background and rarely shows the user anything at all. Older image and audio CAPTCHAs still show up on legacy sites.
Honeypots are the quieter trap. A link hidden with display:none or positioned off-screen exists for exactly one reason, to catch bots that blindly follow every href on the page. A real visitor never clicks it, so a crawler that does gets flagged instantly.
Third-party CAPTCHA solving services exist as a fallback, and they work. But if you’re hitting CAPTCHAs constantly, that’s usually a signal that something upstream, your fingerprint, your request pattern, your proxy pool, is already broken. Solving the CAPTCHA treats the symptom, not the cause.
Optimize Request Patterns and Error Handling
A crawler that runs for more than a few minutes needs actual error handling, not just a try/except wrapped around the whole thing.
Randomize your crawl path rather than working through URLs in a fixed order. Set real rate limits per domain, and back off adaptively when you see errors climbing instead of a flat retry-after-N-seconds rule. Log unexpected status codes and response shapes as you go, since that shadow-banning problem from earlier only surfaces if you’re actually checking response content, not just status codes. Track your success rate over time, and run a small regression check against a handful of stable, known-good URLs periodically to catch silent breakage early.
Favor APIs and Cached Sources When Possible
This is the step that would’ve saved me the most time over the years, and it’s the one people skip because parsing HTML feels more direct.
Check for an official public API first. Plenty of sites have one, and it’s not even always documented well, but a quick search usually turns it up. If there’s no public API, open DevTools, watch the Network tab, and look for the XHR or Fetch calls the page makes to load its own data. Hitting that JSON endpoint directly is almost always faster and more stable than rendering the page and parsing HTML.
Mobile apps are worth checking too. Intercepting traffic from a site’s mobile app through a proxy sometimes exposes a cleaner, less-protected API than the one the website uses.
And if you don’t need real-time data, you may not need to crawl the live site at all. The Wayback Machine and Common Crawl both hold historical snapshots that can cover a surprising amount of ground without sending a single request to the target.
Proxy Setup Examples for Crawling
Whichever proxy type you land on, the connection syntax is the same shape across HTTP, HTTPS, and SOCKS. Here’s what that looks like in Python with requests, since that’s what most crawlers are built on.
HTTP proxy:
import requests
proxies = {
"http": "http://proxy_host:port"
}
response = requests.get("https://example.com", proxies=proxies)
print(response.status_code)HTTPS proxy:
import requests
proxies = {
"https": "https://proxy_host:port"
}
response = requests.get("https://example.com", proxies=proxies)
print(response.status_code)SOCKS4 proxy:
import requests
proxies = {
"http": "socks4://proxy_host:port",
"https": "socks4://proxy_host:port"
}
response = requests.get("https://example.com", proxies=proxies)
print(response.status_code)SOCKS5 proxy:
import requests
proxies = {
"http": "socks5://proxy_host:port",
"https": "socks5://proxy_host:port"
}
response = requests.get("https://example.com", proxies=proxies)
print(response.status_code)SOCKS5 is worth reaching for over SOCKS4 if you also need UDP support or proxy-level authentication. Our guide to using a SOCKS5 proxy walks through the full setup outside of Python too.
Authenticated proxy:
import requests
proxies = {
"http": "http://username:password@proxy_host:port",
"https": "http://username:password@proxy_host:port"
}
response = requests.get("https://example.com", proxies=proxies)
print(response.status_code)Note the socks4/socks5 scheme needs the requests[socks] extra installed (pip install requests[socks]), otherwise you’ll get a silent MissingSchema error that has nothing to do with your proxy actually being wrong.
Troubleshooting Common Issues
Getting 403s even after rotating IPs. The proxy isn’t the problem anymore; your headers or TLS fingerprint probably are. Check that your User-Agent and TLS handshake actually agree with each other.
Success rate drops mid-run with no config change. Individual IPs in a rotation pool can get flagged independently of each other. Log per-IP outcomes so you can spot and drop the bad ones instead of assuming the whole pool went bad. Running your pool through a quick proxy testing pass before a big crawl catches dead or already-flagged IPs before they burn through your rate limit.
Empty or truncated content on a 200 response. That’s shadow-banning, not a bug in your parser. Validate actual page content against an expected pattern, not just the status code.
CAPTCHA on every single request. Something upstream is broken badly enough that solving the CAPTCHA won’t fix the underlying issue for long. Revisit your request pattern and fingerprint consistency before reaching for a solver.
Headless browser gets flagged instantly. Default Playwright and Puppeteer installs leave detectable automation fingerprints out of the box. You’ll need stealth patches, and even then, pair it with a clean rotating proxy rather than expecting the stealth patch to carry the whole job alone.
For the source-level detail, the official robots.txt specification covers exactly what a robots.txt file can and can’t enforce. The MDN reference on Fetch metadata headers is worth reading too if you want to understand what Sec-Fetch-* headers actually communicate to a server.
None of this works well without a proxy pool that can actually take the load, and it’s worth being specific about which pool for which job. Proxying splits that by exactly this kind of use case, residential proxies for targets that fingerprint aggressively, ISP proxies when you want speed without landing on a known datacenter range, and datacenter proxies for lower-sensitivity crawls where cost matters more than stealth.
