# Wowhead Scraper

Scrapes quests and objects from [Wowhead](https://www.wowhead.com/forever) in two stages:

1. **List**: crawls the list pages and writes one CSV row per entity (`output/quests.csv`, `output/objects.csv`).
2. **Detail**: visits each row's page and appends one JSON object per line (`output/quests.jsonl`, `output/objects.jsonl`).

## Setup

```
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
playwright install chromium
```

Wowhead is behind an AWS WAF JavaScript challenge. Requests go through `curl_cffi` with a Chrome
fingerprint; when the challenge appears, headless Chromium solves it once and its cookie is reused.

## Usage

```
python runner.py list quest            # -> output/quests.csv
python runner.py detail quest          # -> output/quests.jsonl
python runner.py all object            # both stages
python runner.py detail quest --ids 166 98028
python runner.py detail quest --limit 50
python runner.py find-missing quest    # re-fetch quests skipped during a detail run
python runner.py find-missing quest --dry-run   # only list them
```

`find-missing` only touches rows *below* the highest ID already in the JSONL, i.e. ones a detail run
skipped (usually because a temporary block outlasted the retries). Rows past that point haven't been
reached yet; `detail` continues with those.

The detail stage skips IDs already in the JSONL file, so an interrupted run (or one with failures)
resumes where it left off when rerun. Pages Wowhead doesn't have are written as `{"id": ..., "not_found": true}`.

| Option          | Default   | Description                                             |
|-----------------|-----------|---------------------------------------------------------|
| `-v, --version` | `forever` | Wowhead path prefix                                     |
| `--delay`       | `1.0`     | Minimum seconds between requests (across all workers)   |
| `--jitter`      | `0.5`     | Extra random delay, 0..N seconds                        |
| `--concurrency` | `2`       | Parallel detail workers                                 |
| `--max-retries` | `5`       | Retries on 403/429/5xx/network errors                   |
| `--backoff`     | `2.0`     | First retry wait in seconds; doubles each retry, max 120 |

## How list retrieval works

A list page holds at most 1000 rows (the `#200` style pagination is client side only), so the list
stage requests ID ranges with Wowhead's ID filter and splits any range whose result was truncated.
A full quest or object list takes about 30 requests. The final count is checked against Wowhead's
own "N found" total.

## Layout

| Path            | Contents                                                   |
|-----------------|------------------------------------------------------------|
| `config.py`     | All URLs, filter ids and crawl settings                    |
| `fetcher.py`    | HTTP client: throttle, back-off, WAF handling              |
| `parsers/`      | Pure functions from page HTML to dicts, one module per type |
| `spiders/`      | `Spider` base class with both stages; thin subclasses per type |
| `tests/`        | Parser tests against `example_pages/` and `tests/fixtures/` |

### Adding a type (e.g. items)

1. Add an `EntityConfig` to `ENTITIES` in `config.py` (the "ID" filter id is in `Filter.init` on the list page).
2. Write `parsers/item.py` with `LISTVIEW_ID`, `CSV_FIELDS`, `list_row()` and `parse_page()`.
3. Add a `Spider` subclass in `spiders/` and register it in `SPIDERS`.

Run the tests with `pytest`.