Web Scraping
Also known as: screen scraping, web data extraction, crawling.
Collecting data by having a program request web pages meant for people and pull out the fields it needs. A fallback with real fragility.
Synthetic dataExamples on this page use synthetic data created for illustration.
Definition
Web scraping is collecting data by letting a program request web pages meant for people and extract the fields it needs from their markup. It is a fallback when no structured source exists, and it is fragile because page layouts change without notice.
In plain terms
A web page mixes content with layout. A scraper downloads the page, locates elements by their structure and reads the text. Nothing promises that the structure stays the same, so a redesign can break a scraper or, worse, make it read the wrong field without any error.
Because of that, an official interface, a licensed feed or a file export is almost always the better first choice: it is documented, stable and clearly permitted. When scraping is the only option, treat the scraper as a measuring instrument. Keep the raw page as evidence, record which parser version read it, check how many values it found, and follow the site’s published rules and the limits set out in robots.txt and Rate Limiting.
Example
A scraper reads the price from the element marked price, once a day, from 100 product pages.
| Day | What happened | Pages fetched | Pages with a price | Prices that look right |
|---|---|---|---|---|
| 1 | Normal | 100 | 100 | 100 |
| 2 | Site redesign renames the element | 100 | 0 | — (the alarm on empty fields fires) |
| 3 | Parser updated, but it now picks the struck-through list price | 100 | 100 | 0 (all prices about 25% too high, no error raised) |
Day 2 is a loud failure and easy to catch. Day 3 is a silent one: only a sample check against the real page, or a comparison with an earlier day, reveals it.
Common misunderstandings
- “Public means free to take.” Public access does not remove a site’s terms of use, copyright or data-protection rules. Check them for every source (this is not legal advice).
- “A scraper that runs without errors is correct.” The worst failures are silent: the wrong field, a stale page, a changed unit.
- “Scraping is the natural first step.” Look for an official interface, a licensed feed or an export first.
References
- Ryan Mitchell. Web Scraping with Python . O'Reilly Media, 2018. ISBN 978-1491985571.
- An overview of HTTP . MDN Web Docs.