wowhead-scraper/wowhead_scraper

Wowhead Scraper

Scrapes quests and objects from Wowhead in two stages:

1. List: crawls the list pages and writes one CSV row per entity (output/quests.csv, output/objects.csv).
2. Detail: visits each row's page and appends one JSON object per line (output/quests.jsonl, output/objects.jsonl).

Setup

python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
playwright install chromium

Wowhead is behind an AWS WAF JavaScript challenge. Requests go through curl_cffi with a Chrome fingerprint; when the challenge appears, headless Chromium solves it once and its cookie is reused.

Usage

python runner.py list quest            # -> output/quests.csv
python runner.py detail quest          # -> output/quests.jsonl
python runner.py all object            # both stages
python runner.py detail quest --ids 166 98028
python runner.py detail quest --limit 50
python runner.py find-missing quest    # re-fetch quests skipped during a detail run
python runner.py find-missing quest --dry-run   # only list them

find-missing only touches rows below the highest ID already in the JSONL, i.e. ones a detail run skipped (usually because a temporary block outlasted the retries). Rows past that point haven't been reached yet; detail continues with those.

The detail stage skips IDs already in the JSONL file, so an interrupted run (or one with failures) resumes where it left off when rerun. Pages Wowhead doesn't have are written as {"id": ..., "not_found": true}.

Option	Default	Description
-v, --version	forever	Wowhead path prefix
--delay	1.0	Minimum seconds between requests (across all workers)
--jitter	0.5	Extra random delay, 0..N seconds
--concurrency	2	Parallel detail workers
--max-retries	5	Retries on 403/429/5xx/network errors
--backoff	2.0	First retry wait in seconds; doubles each retry, max 120

How list retrieval works

A list page holds at most 1000 rows (the #200 style pagination is client side only), so the list stage requests ID ranges with Wowhead's ID filter and splits any range whose result was truncated. A full quest or object list takes about 30 requests. The final count is checked against Wowhead's own "N found" total.

Layout

Path	Contents
config.py	All URLs, filter ids and crawl settings
fetcher.py	HTTP client: throttle, back-off, WAF handling
parsers/	Pure functions from page HTML to dicts, one module per type
spiders/	Spider base class with both stages; thin subclasses per type
tests/	Parser tests against example_pages/ and tests/fixtures/

Adding a type (e.g. items)

1. Add an EntityConfig to ENTITIES in config.py (the "ID" filter id is in Filter.init on the list page).
2. Write parsers/item.py with LISTVIEW_ID, CSV_FIELDS, list_row() and parse_page().
3. Add a Spider subclass in spiders/ and register it in SPIDERS.

Run the tests with pytest.