2.16. HTTPX Template

The HTTPX Template implements the Web Scraping Template’s WebScrapingAsyncPort on top of HTTPX, a fast async HTTP client — no browser involved, for pages that don’t need JavaScript execution.

It depends on web_scraping (installed automatically) and is combinable with the other engines (selenium, playwright, curl_cffi) — install whichever this project needs side by side.

2.16.1. Choosing a parser

HTTPX itself only fetches pages — parsing HTML is handled by a separate, swappable parser: BeautifulSoup or selectolax. Two ready-made adapter classes cover both (HTTPXBeautifulSoupAdapter, HTTPXSelectolaxAdapter), sharing the exact same HTTP-fetching logic and differing only in which parser they compose.

In this project’s own benchmarks on a large page, selectolax beat BeautifulSoup by roughly 5.6× end to end — see the Web Scraping Template page for the numbers. selectolax is the better default unless a specific BeautifulSoup feature is needed.

2.16.2. Cookies and headers

Both work through HTTPX’s own client — cookies via its cookie jar, headers via its own mutable header dict. This is what makes session handoff from a browser engine to HTTPX possible.

2.16.3. Using HTTPX specifically

This template doesn’t add its own runners — the Web Scraping Template’s page_links runner already works once HTTPX is installed. To make HTTPX the one actually used, either set it as the default (config/web_scraping.toml’s default_reading_adapter, e.g. "httpx_selectolax") or inject HTTPXBeautifulSoupAdapter/HTTPXSelectolaxAdapter directly in your own code to pick the parser explicitly. See Available runners.