# Rate limits vs. bot detection: why throttling scrapers is not enough | LotWall

Source: https://lotwall.com/blog/2026-09-27-rate-limits-vs-bot-detection-throttling-scrapers.html

[Home](../index.md) / [Blog](index.md)


September 27, 2026 - LotWall

**Short answer:** Rate limits cap how fast one client can pull pages, but scrapers do not need speed. A bot pulling ten listings a minute from a thousand rotating IPs stays under every threshold and still walks off with your whole inventory by morning. Stopping scrapers takes detection: knowing which clients are automated, whatever rate they run at.

## What rate limiting is good at

Rate limits protect infrastructure: they stop one client from hammering your servers and make denial-of-service attempts expensive. Every marketplace needs them. The mistake is treating them as a security control against scraping, which is a different problem.

## Why scrapers do not care about your thresholds

* **They parallelize.** A scraper limited to 60 requests a minute per IP uses 100 cheap proxy IPs and gets 6,000 requests a minute.
* **They cache and diff.** Smart scrapers fetch each listing once and only re-check for changes, staying far below any threshold you could set without hurting real users.

## The threshold trap

When throttling fails, the instinct is to lower the limits. Tighter thresholds catch real users: shoppers comparing listings, dealer tools syncing inventory, your own app polling for updates. The scrapers just add more IPs. You cannot win a rate war against an adversary whose marginal cost per IP is near zero.

## What detection adds that throttling cannot

* **Intent over velocity.** Detection looks at what the client does, not how fast: systematic traversal, form-field probing, inhuman session patterns. A slow scraper still behaves like a scraper.
* **Identity that survives rotation.** Browser fingerprints and automation-framework traces identify the tool behind the request. Block the tool, not the IP.
* **Graduated responses.** Route suspected scrapers to degraded content: stale prices or watermarked images. The scraper's data becomes worthless while real users never notice.
* **Coverage of the quiet ones.** The most damaging scrapers are the polite ones your rate limiter never sees. Detection is the only layer that finds them.

## The right stack for a marketplace

* Keep rate limits as infrastructure protection, set generously enough that no legitimate user ever hits them.
* Add behavioral bot detection as the security layer, tuned for your vertical's scraping patterns: inventory harvesting, VIN collection, price monitoring.
* Feed confirmed scraper identities into your edge rules so repeat offenders get blocked before costing you another request.
* Measure listings scraped per day and fake leads blocked, not requests throttled. Those numbers tell you if you are winning.

## See your own numbers.

A free bot-traffic audit shows the human-automated split in your live traffic - no code changes, no commitment.

[Get a free bot-traffic audit](mailto:hello@lotwall.com?subject=Free%20free%20bot-traffic%20audit)