72 lines
3.5 KiB
Markdown
72 lines
3.5 KiB
Markdown
# Wowhead Scraper
|
|
|
|
Scrapes quests and objects from [Wowhead](https://www.wowhead.com/forever) in two stages:
|
|
|
|
1. **List**: crawls the list pages and writes one CSV row per entity (`output/quests.csv`, `output/objects.csv`).
|
|
2. **Detail**: visits each row's page and appends one JSON object per line (`output/quests.jsonl`, `output/objects.jsonl`).
|
|
|
|
## Setup
|
|
|
|
```
|
|
python -m venv venv
|
|
venv\Scripts\activate
|
|
pip install -r requirements.txt
|
|
playwright install chromium
|
|
```
|
|
|
|
Wowhead is behind an AWS WAF JavaScript challenge. Requests go through `curl_cffi` with a Chrome
|
|
fingerprint; when the challenge appears, headless Chromium solves it once and its cookie is reused.
|
|
|
|
## Usage
|
|
|
|
```
|
|
python runner.py list quest # -> output/quests.csv
|
|
python runner.py detail quest # -> output/quests.jsonl
|
|
python runner.py all object # both stages
|
|
python runner.py detail quest --ids 166 98028
|
|
python runner.py detail quest --limit 50
|
|
python runner.py find-missing quest # re-fetch quests skipped during a detail run
|
|
python runner.py find-missing quest --dry-run # only list them
|
|
```
|
|
|
|
`find-missing` only touches rows *below* the highest ID already in the JSONL, i.e. ones a detail run
|
|
skipped (usually because a temporary block outlasted the retries). Rows past that point haven't been
|
|
reached yet; `detail` continues with those.
|
|
|
|
The detail stage skips IDs already in the JSONL file, so an interrupted run (or one with failures)
|
|
resumes where it left off when rerun. Pages Wowhead doesn't have are written as `{"id": ..., "not_found": true}`.
|
|
|
|
| Option | Default | Description |
|
|
|-----------------|-----------|---------------------------------------------------------|
|
|
| `-v, --version` | `forever` | Wowhead path prefix |
|
|
| `--delay` | `1.0` | Minimum seconds between requests (across all workers) |
|
|
| `--jitter` | `0.5` | Extra random delay, 0..N seconds |
|
|
| `--concurrency` | `2` | Parallel detail workers |
|
|
| `--max-retries` | `5` | Retries on 403/429/5xx/network errors |
|
|
| `--backoff` | `2.0` | First retry wait in seconds; doubles each retry, max 120 |
|
|
|
|
## How list retrieval works
|
|
|
|
A list page holds at most 1000 rows (the `#200` style pagination is client side only), so the list
|
|
stage requests ID ranges with Wowhead's ID filter and splits any range whose result was truncated.
|
|
A full quest or object list takes about 30 requests. The final count is checked against Wowhead's
|
|
own "N found" total.
|
|
|
|
## Layout
|
|
|
|
| Path | Contents |
|
|
|-----------------|------------------------------------------------------------|
|
|
| `config.py` | All URLs, filter ids and crawl settings |
|
|
| `fetcher.py` | HTTP client: throttle, back-off, WAF handling |
|
|
| `parsers/` | Pure functions from page HTML to dicts, one module per type |
|
|
| `spiders/` | `Spider` base class with both stages; thin subclasses per type |
|
|
| `tests/` | Parser tests against `example_pages/` and `tests/fixtures/` |
|
|
|
|
### Adding a type (e.g. items)
|
|
|
|
1. Add an `EntityConfig` to `ENTITIES` in `config.py` (the "ID" filter id is in `Filter.init` on the list page).
|
|
2. Write `parsers/item.py` with `LISTVIEW_ID`, `CSV_FIELDS`, `list_row()` and `parse_page()`.
|
|
3. Add a `Spider` subclass in `spiders/` and register it in `SPIDERS`.
|
|
|
|
Run the tests with `pytest`.
|