Collecting Data Politely: Rate Limits, robots.txt and Stopping When You Are Blocked

Whenever a program fetches data from someone else’s service, it takes something: bandwidth, processing time, and the goodwill of the people who run it. A person clicking around is a light load; a program can be a thousand times heavier without anyone intending harm. Good collection practice is mostly about keeping that burden small and respecting the answer when the source says no. This article describes a practical approach. It is general guidance and not legal advice; the rules that apply to a particular source are in its own terms.

Start with the question: is there a better route?

Before fetching web pages, check whether the data are offered in a form meant for programs:

An official interface (API) with documented limits and terms.
A licensed feed or a file export, possibly paid.
An open dataset under a licence that allows your use (Data License).
Only then, the pages themselves, as a fallback (Web Scraping).

The earlier options are more stable (a page redesign does not break them), clearer about permission, and kinder to the source. Collection from pages should cover only what the better routes cannot give.

Read the rules

Terms of use and licence. They state what you may do with the data: store it, for how long, derive from it, publish it. Having access does not grant these rights (Data License). Record your reading of each source, and “not verified” where you are unsure.
robots.txt. The site’s published request about which paths automated clients should not fetch (robots.txt). Read it before the first request, honour it, and log the paths you skipped. It is a convention, not a lock, and not a statement about permission to use the data.
Guidelines for crawlers. Many operators ask crawlers to identify themselves with a descriptive name and a way to reach the operator, or to stay within a stated rate. Follow any such request.

Spend a small budget, slowly

Decide in advance how much you will take, and make it small.

A daily cap on requests, set low and raised only if needed. Start with dozens, not thousands.
Spacing between requests, with random variation (jitter), so that your traffic is a trickle and not a burst (Rate Limiting).
Active hours rather than around the clock, if the source has a quiet period.
No repeats without reason. Cache what you have, and use conditional requests where the service supports them.

For a sense of scale: a cap of 200 pages a day with a gap of 20 to 60 seconds between requests averages 40 seconds, so one run takes about 200 × 40 = 8,000 seconds, a little over two hours, and the source sees under one request a minute. Raising the cap tenfold without changing the spacing would turn the run into a 22-hour load.

Listen to the signals

The server tells you how it feels, in status codes and headers. A polite collector reads them and responds.

Signal	Meaning	Response
200, normal	All is well	Continue at the planned pace
429 Too Many Requests, or a `Retry-After` header	You are going too fast	Stop for at least the time given, then slow down
5xx errors, timeouts	The service is struggling	Back off: wait longer after each failure, with jitter (Exponential Backoff)
403 Forbidden, a block page, a challenge	Access has been refused	Stop and look. Do not retry in a loop
A path disallowed in robots.txt	The owner asks you not to fetch it	Skip it and log the skip
A change of page layout	Your parser may now be wrong	Pause, inspect, update before continuing

The most important row is the fifth: when access is refused, the answer is to stop. Do not try to get around a block, a challenge or a login you are not entitled to. Escalating against the source is the opposite of polite collection, risks your standing with the operator and often breaks their terms, and a gap in your data is a far smaller problem than a dispute.

Build the stop into the system

A circuit breaker turns “stop when blocked” from a good intention into a property of the program (Circuit Breaker): after a few consecutive refusals or errors it opens, refuses to make further requests, alerts a person, and stays open until someone has looked at what changed. Resuming should be a human decision, because the cause (a block, new terms, a redesign) is exactly what a human has to judge.

Record the stop honestly in the data. A day with no collection is a gap, kept as a gap and not filled (Missing Data, Collecting a Time Series You Can Compare: Fixed Conditions, Snapshots and Honest Gaps).

Take only what you need

Collect the fields your question needs, not everything on the page.
Leave out personal data where you can: names, photos, profile links of private individuals. If you must hold any, have a purpose, a limit and a deletion date.
Keep raw copies only as long as your licence allows, and record when they expire (WARC).
Do not republish what you may not.

A checklist before the first request

Is there an official interface, a feed or an export? Use it first.
Have you read the terms and licence, and written down what they allow?
Have you read robots.txt, and does your collector obey and log it?
Is the daily cap small and the spacing generous, with jitter?
Does the collector honour 429s, Retry-After and backoff?
Does a circuit breaker stop collection on a block and alert a person?
Are gaps recorded as gaps?
Are you collecting the minimum, with personal data excluded or tightly limited?
When did you last re-read the source’s terms? Put a date in your calendar; terms and interfaces change.