Headless Browser

Definition

A headless browser is a real web browser run without a visible window and controlled by a program. Because it executes a page’s scripts like any browser, it can read content that is assembled on the client, at the cost of being slower and heavier than plain HTTP requests.

In plain terms

Many pages send very little HTML and build their content with JavaScript. A simple HTTP request then returns an empty shell. A headless browser, driven through a standard such as WebDriver or a browser’s debugging protocol, loads the page, runs the scripts and exposes the final document. Testers use the same tools to check sites and to take screenshots or print pages to PDF.

The costs are real: more memory and processing, slower runs and more moving parts. Everything that applies to polite collection still applies: following the site’s rules, limiting request rates, and stopping when access is refused.

Example

A page builds its list of 50 items with a script after it loads.

If the same data is offered through an interface the page itself calls, using that interface directly is cheaper and steadier than driving a browser.

Common misunderstandings

• “Headless means the site cannot tell and the rules do not apply.” Running without a window changes nothing about what is permitted.
• “It is always the better tool.” It is slower and more fragile; use plain requests or an official interface when they suffice.
• “A screenshot is enough evidence.” A screenshot loses the structure and headers; keep the page itself (see WARC).