Vibecode WooRank

You are building a lean indie version of WooRank.

Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt.

===== README.md =====
# WooRank · indie build

A self-hosted site auditor whose hard problems are crawling somebody's site politely and provably with permission, reporting findings with evidence instead of an invented score, and measuring performance in a way that does not pretend a single local run is what search engines actually see.

Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts.

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | A Python command line tool with a headless browser for rendering, output to files | an audit is a batch job that produces a document, not an application people log into |
| Crawl | One request in flight by default, a delay between requests, a page ceiling per run | the site you are auditing belongs to somebody with a hosting bill |
| Checks | Declarative check definitions, each carrying its own citation to published guidance | a check nobody can trace back to a source is an opinion in a table |
| Storage | SQLite, one row per finding, with the evidence stored beside it | a finding you cannot show the evidence for cannot be argued |
| Output | HTML and CSV reports generated from the findings table | the deliverable is the document, and it has to stand on its own |

## Before you start

Have every one of these ready. The plan assumes them from step one.

- [ ] **Python 3.12 or newer** · free
  - Why: Runs the crawler, the checks and the report generation.
  - Get it: python.org or your package manager.
  - Verify: python3 --version prints 3.12 or higher
- [ ] **A headless browser automation library with its browsers installed** · free, plus a few hundred megabytes of disk
  - Why: Half the checks and all the performance work need a rendered page, since a large share of sites assemble their content in the browser.
  - Get it: Install the library and run its browser download step.
  - Verify: A scripted page load returns rendered content and a performance entry
- [ ] **Written authorisation for every site you will crawl** · free
  - Why: This is rule zero and it is also the difference between an audit and an unsolicited crawl of somebody's server.
  - Get it: For your own sites, a note to yourself. For a client, an email saying which origin, at what rate, and until when.
  - Verify: Every site row has an authorised_by value and a date before its first crawl
- [ ] **The current published performance thresholds and metric names** · free
  - Why: The metric set has been changed before, with one metric replaced outright in 2024, so copying numbers from memory dates your tool immediately.
  - Get it: Read the current published guidance, record each metric, its threshold and the date you checked.
  - Verify: docs/thresholds.md lists each metric with its value and date, and the checks reference it
- [ ] **A small site you control, with deliberate faults** · free
  - Why: Every check, the diff and the redirect matching need a site whose answers you already know, and testing against a real client site is how you discover your crawler is impolite.
  - Get it: Publish a handful of static pages locally with missing alternative text, a duplicate title and a redirect.
  - Verify: A crawl of the fixture produces exactly the findings you planted

## Quick start

Follow Phase 1 in `BUILD_PLAN.md`; it creates the project.

Then copy `.env.example` to `.env` and fill in the values it documents.

## Honest limits

This build deliberately does not replace:

- Keyword ranking data.
- Backlink data.
- Competitor benchmarking crawls.
- A public site grading widget.
- historical competitor datasets and support
- fresh proprietary keyword, backlink, or ranking data
- proprietary web and keyword indexes
- fresh rank and clickstream data
- large-scale crawling infrastructure

If one of those is essential to you, that is the reason to keep paying for WooRank, and the README should say so rather than pretend.

===== BRIEF.md =====
# Build brief · WooRank

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a site auditor that reports findings with their evidence instead of a mark out of one hundred to replace WooRank. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
The headline of the product you are replacing is a score out of one hundred. Do not build one.
A score like that is a weighted checklist, and nobody, including the people selling it, can defend
the weights. Two auditors grading the same site produce different numbers, the number moves when
the checklist changes rather than when the site does, and once a client sees it the work becomes
raising the number instead of fixing the site. Report findings: what was checked, on which page,
the evidence, how bad it is, and a link to the published guidance the check came from. Counts by
severity are allowed. A single figure summarising a website is not.
Second: crawl only sites you own or have written permission to audit. Record who authorised it and
when, before the first request. Identify your crawler honestly in the user agent with a contact,
respect the site's stated crawl directives, keep one request in flight at a time by default, and
put a page ceiling on every crawl. An auditor that hammers a small shared host is a denial of
service with a report attached.

### Stack (fixed, do not substitute)
- Runtime: A Python command line tool with a headless browser for rendering, output to files · an audit is a batch job that produces a document, not an application people log into
- Crawl: One request in flight by default, a delay between requests, a page ceiling per run · the site you are auditing belongs to somebody with a hosting bill
- Checks: Declarative check definitions, each carrying its own citation to published guidance · a check nobody can trace back to a source is an opinion in a table
- Storage: SQLite, one row per finding, with the evidence stored beside it · a finding you cannot show the evidence for cannot be argued
- Output: HTML and CSV reports generated from the findings table · the deliverable is the document, and it has to stand on its own

### Data model (create this before Phase 1)
- sites(id, origin, authorised_by, authorised_at, contact_email, max_pages, delay_ms)
- crawls(id, site_id, started_at, finished_at, pages_fetched, pages_skipped, stopped_reason)
- pages(id, crawl_id, url, status, content_type, bytes, fetched_at, render_ms, redirect_chain_json)
- findings(id, crawl_id, page_id, check_id, severity, evidence_json, source_url) · source_url is the published guidance, and a check without one does not ship
- measurements(id, page_id, metric, value, run_ordinal, method) · method is always lab, because that is what a local run is
- diffs(from_crawl, to_crawl, check_id, page_url, change) · appeared, resolved or unchanged
- There is no score column. There is no grade column. The report's headline is a count of findings by severity, and that is deliberate.

### Phase 1 · A crawl you are allowed to run
Build: Register a site with who authorised the audit, a contact address and the ceilings. Fetch with an
  honest user agent including that contact, one request at a time with a configurable delay, obeying
  the site's stated crawl directives and stopping at the page ceiling. Log every request with its
  status and timing. Provide a stop control that halts within one request.
Trap: Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
Done when: A crawl refuses to start without a recorded authorisation and contact address, tested. Request concurrency is one by default and every request carries the configured user agent with a contact, verified in a request log. The page ceiling stops the crawl and records the reason, tested against a site with more pages than the ceiling. Stopping a crawl halts it within one request and leaves the partial results usable.
Do not build yet: No checks, findings, performance or reports yet.

### Phase 2 · Findings, each with evidence and a source
Build: Define checks declaratively: an identifier, a severity, the selector or condition, the evidence to
  capture, and the URL of the published guidance behind it. Store one finding per page per check with
  the evidence attached, for instance the actual title text found, or the three images missing
  alternative text with their paths. A check with no source URL fails the test suite.
Trap: Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
Done when: Every check definition carries a source URL and the suite fails if one does not, tested by adding a check without one. Each finding stores the evidence that produced it and the report shows it inline. No check emits a numeric grade, verified by review. Two runs over an unchanged fixture site produce identical findings, tested.
Do not build yet: No performance measurement, diffing or reports yet.

### Phase 3 · Performance measured honestly
Build: Render each page in a headless browser and record the loading, interaction responsiveness and
  layout stability metrics the published guidance defines. Run each page three times and report the
  median with the spread. Compare against the published thresholds rather than thresholds you chose.
  Label every number as a local run, because that is what it is.
Trap: Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
Done when: Each page is measured at least three times and the report shows median and spread rather than a single figure, tested. Thresholds used in the report are the currently published ones and are cited with the date checked, verified in the check definitions. Every performance figure in the output is labelled as a local measurement. A page that fails to render is recorded as a failure with the error rather than silently scoring zero.
Do not build yet: No diffing between crawls, reports or scheduling yet.

### Phase 4 · What changed since last time
Build: Store crawls as immutable snapshots. Diff two crawls by page URL and check identifier, producing
  appeared, resolved and unchanged rows. Handle pages that were added, removed or redirected without
  reporting them as regressions. Show the diff as a list of changes, never as a movement in a number.
Trap: Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.
Done when: A finding fixed between crawls appears as resolved and nothing else changes, tested with two fixture crawls. A page that moved with a redirect is matched to its new URL rather than reported as removed, tested. A crawl snapshot is never modified after it finishes, asserted in the suite. The diff output contains no aggregate number, verified by review.
Do not build yet: No report generation, scheduling or storage limits yet.

### Phase 5 · A report you can hand to somebody
Build: Generate an HTML report that opens with no server and a CSV of every finding. Order by severity
  and then by how many pages are affected. Include the evidence, the source link and the crawl
  metadata, including who authorised it and when. Add scheduled re-crawls, and a retention setting
  that prunes old crawls while keeping their summary rows.
Trap: A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.
Done when: The HTML report opens from the file system with no server and no network requests, tested offline. Every finding in the report carries its evidence and its source link. The report states the authorisation, the crawl date, the page count and the tool version on the first screen. Retention prunes crawl detail beyond the configured age while keeping the summary rows needed for diffs, tested by moving the clock.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Keyword rankings, because getting them means either scraping a search engine against its terms or renting a data provider, and both are somebody else's index rather than your tool.
- Backlink data, which is a crawl of the entire web maintained continuously and is the one part of the paid product that genuinely cannot be rebuilt.
- Competitor benchmarking, since it means crawling sites nobody authorised you to crawl, which rule zero already rules out.
- A lead generation widget that grades a stranger's site, which is the score you refused to build, pointed at people who did not ask.

### README must contain
- That this tool produces findings and counts by severity, never a score or a grade, and why a weighted checklist score cannot be defended
- That a crawl requires recorded authorisation, a contact in the user agent, one request in flight by default and a page ceiling
- That every check cites the published guidance it came from, and that a check without a source does not ship
- That performance figures are local laboratory runs reported as a median of three with the spread, while the published thresholds are assessed at a percentile of real visits
- What the retention setting prunes, how large a crawl of a given size is on disk, and where reports are written

===== AGENTS.md =====
# Agent instructions · WooRank indie build

- Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python command line tool with a headless browser for rendering, output to files, One request in flight by default, a delay between requests, a page ceiling per run, Declarative check definitions, each carrying its own citation to published guidance, SQLite, one row per finding, with the evidence stored beside it, HTML and CSV reports generated from the findings table. Do not substitute.
- Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes.
- Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name.
- Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced.
- Do not invent cryptography, security guarantees, APIs or compliance claims.
- Add a focused test for every destructive, security-sensitive or data-loss path the plan names.
- Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs".

## Known traps

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.

===== BUILD_PLAN.md =====
# Build plan · WooRank

A self-hosted site auditor whose hard problems are crawling somebody's site politely and provably with permission, reporting findings with evidence instead of an invented score, and measuring performance in a way that does not pretend a single local run is what search engines actually see.

Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note.

## Phase 1 · A crawl you are allowed to run

Auditing a site requires recorded authorisation, and the crawler behaves like a guest.

### Steps

1. Record the authorisation before any request
   If you cannot name who agreed, do not fetch the first page.
   Files: `src/sites.py`, `tests/test_authorisation.py`
2. One request in flight, with a delay and a ceiling
   Politeness is a default, not a flag.
   Files: `src/crawl/fetcher.py`
3. Log every request with status and timing
   This log is your answer when somebody asks what you did to their server.
   Files: `src/crawl/log.py`

### Done when

- [ ] No crawl starts without a recorded authorisation
- [ ] Concurrency is one and the user agent carries a contact
- [ ] The page ceiling stops the crawl with a reason
- [ ] Stopping halts within one request

### Watch out

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.

## Phase 2 · Findings, each with evidence and a source

Every check produces rows containing the page, what was found, and a link to the guidance it came from.

### Steps

1. Make checks data, not code branches
   A table of checks can be read, argued with and extended.
   Files: `src/checks/definitions.py`
2. Require a source URL on every check
   If it is not in published guidance, it is your preference.
   Files: `src/checks/loader.py`, `tests/test_sources.py`
3. Store the evidence, not just the verdict
   Nobody fixes a finding they cannot see.
   Files: `src/checks/run.py`

### Done when

- [ ] Every check has a source URL or the suite fails
- [ ] Findings carry the evidence that produced them
- [ ] No check emits a grade or a score
- [ ] Repeat runs on a fixture are identical

### Watch out

- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.

## Phase 3 · Performance measured honestly

Page performance is measured several times, reported against the published thresholds, and labelled as a local measurement.

### Steps

1. Measure three times and report the median and spread
   A single run on a laptop is a sample of one.
   Files: `src/perf/measure.py`, `tests/test_median.py`
2. Cite the thresholds and the date you checked them
   The names and the numbers have changed before and will again.
   Files: `src/checks/definitions.py`, `docs/thresholds.md`
3. Label every number as a laboratory measurement
   The published assessment uses real visits at a percentile, not your machine.
   Files: `src/report/render.py`

### Done when

- [ ] Three runs per page, median and spread reported
- [ ] Thresholds are cited with a checked date
- [ ] Every figure is labelled as a local run
- [ ] Render failures are recorded, not scored

### Watch out

- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.

## Phase 4 · What changed since last time

Two crawls of the same site can be compared, and the comparison is about findings appearing and being resolved.

### Steps

1. Keep crawls immutable
   A comparison against something that changed underneath is not a comparison.
   Files: `src/crawl/snapshot.py`
2. Match across redirects before diffing
   Otherwise a URL change looks like the site fell over.
   Files: `src/diff/match.py`, `tests/test_redirect_match.py`
3. Report changes, not movement in a figure
   The whole point of having no score is not to reintroduce one here.
   Files: `src/diff/report.py`

### Done when

- [ ] A fixed finding shows as resolved and nothing else moves
- [ ] Redirected pages are matched, not reported as removed
- [ ] Finished crawls are immutable
- [ ] The diff contains no aggregate number

### Watch out

- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.

## Not in this build

- Keyword ranking data.
- Backlink data.
- Competitor benchmarking crawls.
- A public site grading widget.

## After v1, if you want it

- Search console data for sites you own, which is the only field data available to you without renting an index
- A check that compares two renders of the same page to catch content that only appears with a browser
- An accessibility check set, kept separate from the search findings because the audience and the urgency are different

===== .env.example =====
# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sites, crawls, pages, findings and measurements.
DATABASE_PATH=./data/audits.db

# Required. Sent on every request. The contact address in it is not optional.
USER_AGENT=SiteAudit/1.0 (+mailto:you@example.com)

# Required. Between requests to the same origin. Lower it only with the owner's agreement.
REQUEST_DELAY_MS=1000

# Required. Ceiling per crawl unless the site record overrides it.
MAX_PAGES_DEFAULT=500

# Required. Runs per page. The report shows the median and the spread of these.
PERF_RUNS=3

# Required. Generated HTML and CSV reports. These get emailed, so keep them self-contained.
REPORT_DIR=./reports

# Required. Crawl detail is pruned after this; summary rows are kept for diffs.
CRAWL_RETENTION_DAYS=180

You are building a lean indie version of WooRank.

Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt.

===== README.md =====
# WooRank · indie build

A self-hosted site auditor whose hard problems are crawling somebody's site politely and provably with permission, reporting findings with evidence instead of an invented score, and measuring performance in a way that does not pretend a single local run is what search engines actually see.

Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts.

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | A Python command line tool with a headless browser for rendering, output to files | an audit is a batch job that produces a document, not an application people log into |
| Crawl | One request in flight by default, a delay between requests, a page ceiling per run | the site you are auditing belongs to somebody with a hosting bill |
| Checks | Declarative check definitions, each carrying its own citation to published guidance | a check nobody can trace back to a source is an opinion in a table |
| Storage | SQLite, one row per finding, with the evidence stored beside it | a finding you cannot show the evidence for cannot be argued |
| Output | HTML and CSV reports generated from the findings table | the deliverable is the document, and it has to stand on its own |

## Before you start

Have every one of these ready. The plan assumes them from step one.

- [ ] **Python 3.12 or newer** · free
  - Why: Runs the crawler, the checks and the report generation.
  - Get it: python.org or your package manager.
  - Verify: python3 --version prints 3.12 or higher
- [ ] **A headless browser automation library with its browsers installed** · free, plus a few hundred megabytes of disk
  - Why: Half the checks and all the performance work need a rendered page, since a large share of sites assemble their content in the browser.
  - Get it: Install the library and run its browser download step.
  - Verify: A scripted page load returns rendered content and a performance entry
- [ ] **Written authorisation for every site you will crawl** · free
  - Why: This is rule zero and it is also the difference between an audit and an unsolicited crawl of somebody's server.
  - Get it: For your own sites, a note to yourself. For a client, an email saying which origin, at what rate, and until when.
  - Verify: Every site row has an authorised_by value and a date before its first crawl
- [ ] **The current published performance thresholds and metric names** · free
  - Why: The metric set has been changed before, with one metric replaced outright in 2024, so copying numbers from memory dates your tool immediately.
  - Get it: Read the current published guidance, record each metric, its threshold and the date you checked.
  - Verify: docs/thresholds.md lists each metric with its value and date, and the checks reference it
- [ ] **A small site you control, with deliberate faults** · free
  - Why: Every check, the diff and the redirect matching need a site whose answers you already know, and testing against a real client site is how you discover your crawler is impolite.
  - Get it: Publish a handful of static pages locally with missing alternative text, a duplicate title and a redirect.
  - Verify: A crawl of the fixture produces exactly the findings you planted

## Quick start

Follow Phase 1 in `BUILD_PLAN.md`; it creates the project.

Then copy `.env.example` to `.env` and fill in the values it documents.

## Honest limits

This build deliberately does not replace:

- Keyword ranking data.
- Backlink data.
- Competitor benchmarking crawls.
- A public site grading widget.
- historical competitor datasets and support
- fresh proprietary keyword, backlink, or ranking data
- proprietary web and keyword indexes
- fresh rank and clickstream data
- large-scale crawling infrastructure

If one of those is essential to you, that is the reason to keep paying for WooRank, and the README should say so rather than pretend.

===== BRIEF.md =====
# Build brief · WooRank

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a site auditor that reports findings with their evidence instead of a mark out of one hundred to replace WooRank. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
The headline of the product you are replacing is a score out of one hundred. Do not build one.
A score like that is a weighted checklist, and nobody, including the people selling it, can defend
the weights. Two auditors grading the same site produce different numbers, the number moves when
the checklist changes rather than when the site does, and once a client sees it the work becomes
raising the number instead of fixing the site. Report findings: what was checked, on which page,
the evidence, how bad it is, and a link to the published guidance the check came from. Counts by
severity are allowed. A single figure summarising a website is not.
Second: crawl only sites you own or have written permission to audit. Record who authorised it and
when, before the first request. Identify your crawler honestly in the user agent with a contact,
respect the site's stated crawl directives, keep one request in flight at a time by default, and
put a page ceiling on every crawl. An auditor that hammers a small shared host is a denial of
service with a report attached.

### Stack (fixed, do not substitute)
- Runtime: A Python command line tool with a headless browser for rendering, output to files · an audit is a batch job that produces a document, not an application people log into
- Crawl: One request in flight by default, a delay between requests, a page ceiling per run · the site you are auditing belongs to somebody with a hosting bill
- Checks: Declarative check definitions, each carrying its own citation to published guidance · a check nobody can trace back to a source is an opinion in a table
- Storage: SQLite, one row per finding, with the evidence stored beside it · a finding you cannot show the evidence for cannot be argued
- Output: HTML and CSV reports generated from the findings table · the deliverable is the document, and it has to stand on its own

### Data model (create this before Phase 1)
- sites(id, origin, authorised_by, authorised_at, contact_email, max_pages, delay_ms)
- crawls(id, site_id, started_at, finished_at, pages_fetched, pages_skipped, stopped_reason)
- pages(id, crawl_id, url, status, content_type, bytes, fetched_at, render_ms, redirect_chain_json)
- findings(id, crawl_id, page_id, check_id, severity, evidence_json, source_url) · source_url is the published guidance, and a check without one does not ship
- measurements(id, page_id, metric, value, run_ordinal, method) · method is always lab, because that is what a local run is
- diffs(from_crawl, to_crawl, check_id, page_url, change) · appeared, resolved or unchanged
- There is no score column. There is no grade column. The report's headline is a count of findings by severity, and that is deliberate.

### Phase 1 · A crawl you are allowed to run
Build: Register a site with who authorised the audit, a contact address and the ceilings. Fetch with an
  honest user agent including that contact, one request at a time with a configurable delay, obeying
  the site's stated crawl directives and stopping at the page ceiling. Log every request with its
  status and timing. Provide a stop control that halts within one request.
Trap: Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
Done when: A crawl refuses to start without a recorded authorisation and contact address, tested. Request concurrency is one by default and every request carries the configured user agent with a contact, verified in a request log. The page ceiling stops the crawl and records the reason, tested against a site with more pages than the ceiling. Stopping a crawl halts it within one request and leaves the partial results usable.
Do not build yet: No checks, findings, performance or reports yet.

### Phase 2 · Findings, each with evidence and a source
Build: Define checks declaratively: an identifier, a severity, the selector or condition, the evidence to
  capture, and the URL of the published guidance behind it. Store one finding per page per check with
  the evidence attached, for instance the actual title text found, or the three images missing
  alternative text with their paths. A check with no source URL fails the test suite.
Trap: Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
Done when: Every check definition carries a source URL and the suite fails if one does not, tested by adding a check without one. Each finding stores the evidence that produced it and the report shows it inline. No check emits a numeric grade, verified by review. Two runs over an unchanged fixture site produce identical findings, tested.
Do not build yet: No performance measurement, diffing or reports yet.

### Phase 3 · Performance measured honestly
Build: Render each page in a headless browser and record the loading, interaction responsiveness and
  layout stability metrics the published guidance defines. Run each page three times and report the
  median with the spread. Compare against the published thresholds rather than thresholds you chose.
  Label every number as a local run, because that is what it is.
Trap: Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
Done when: Each page is measured at least three times and the report shows median and spread rather than a single figure, tested. Thresholds used in the report are the currently published ones and are cited with the date checked, verified in the check definitions. Every performance figure in the output is labelled as a local measurement. A page that fails to render is recorded as a failure with the error rather than silently scoring zero.
Do not build yet: No diffing between crawls, reports or scheduling yet.

### Phase 4 · What changed since last time
Build: Store crawls as immutable snapshots. Diff two crawls by page URL and check identifier, producing
  appeared, resolved and unchanged rows. Handle pages that were added, removed or redirected without
  reporting them as regressions. Show the diff as a list of changes, never as a movement in a number.
Trap: Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.
Done when: A finding fixed between crawls appears as resolved and nothing else changes, tested with two fixture crawls. A page that moved with a redirect is matched to its new URL rather than reported as removed, tested. A crawl snapshot is never modified after it finishes, asserted in the suite. The diff output contains no aggregate number, verified by review.
Do not build yet: No report generation, scheduling or storage limits yet.

### Phase 5 · A report you can hand to somebody
Build: Generate an HTML report that opens with no server and a CSV of every finding. Order by severity
  and then by how many pages are affected. Include the evidence, the source link and the crawl
  metadata, including who authorised it and when. Add scheduled re-crawls, and a retention setting
  that prunes old crawls while keeping their summary rows.
Trap: A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.
Done when: The HTML report opens from the file system with no server and no network requests, tested offline. Every finding in the report carries its evidence and its source link. The report states the authorisation, the crawl date, the page count and the tool version on the first screen. Retention prunes crawl detail beyond the configured age while keeping the summary rows needed for diffs, tested by moving the clock.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Keyword rankings, because getting them means either scraping a search engine against its terms or renting a data provider, and both are somebody else's index rather than your tool.
- Backlink data, which is a crawl of the entire web maintained continuously and is the one part of the paid product that genuinely cannot be rebuilt.
- Competitor benchmarking, since it means crawling sites nobody authorised you to crawl, which rule zero already rules out.
- A lead generation widget that grades a stranger's site, which is the score you refused to build, pointed at people who did not ask.

### README must contain
- That this tool produces findings and counts by severity, never a score or a grade, and why a weighted checklist score cannot be defended
- That a crawl requires recorded authorisation, a contact in the user agent, one request in flight by default and a page ceiling
- That every check cites the published guidance it came from, and that a check without a source does not ship
- That performance figures are local laboratory runs reported as a median of three with the spread, while the published thresholds are assessed at a percentile of real visits
- What the retention setting prunes, how large a crawl of a given size is on disk, and where reports are written

===== AGENTS.md =====
# Agent instructions · WooRank indie build

- Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python command line tool with a headless browser for rendering, output to files, One request in flight by default, a delay between requests, a page ceiling per run, Declarative check definitions, each carrying its own citation to published guidance, SQLite, one row per finding, with the evidence stored beside it, HTML and CSV reports generated from the findings table. Do not substitute.
- Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes.
- Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name.
- Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced.
- Do not invent cryptography, security guarantees, APIs or compliance claims.
- Add a focused test for every destructive, security-sensitive or data-loss path the plan names.
- Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs".

## Known traps

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.

===== BUILD_PLAN.md =====
# Build plan · WooRank

A self-hosted site auditor whose hard problems are crawling somebody's site politely and provably with permission, reporting findings with evidence instead of an invented score, and measuring performance in a way that does not pretend a single local run is what search engines actually see.

Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note.

## Phase 1 · A crawl you are allowed to run

Auditing a site requires recorded authorisation, and the crawler behaves like a guest.

### Steps

1. Record the authorisation before any request
   If you cannot name who agreed, do not fetch the first page.
   Files: `src/sites.py`, `tests/test_authorisation.py`
2. One request in flight, with a delay and a ceiling
   Politeness is a default, not a flag.
   Files: `src/crawl/fetcher.py`
3. Log every request with status and timing
   This log is your answer when somebody asks what you did to their server.
   Files: `src/crawl/log.py`

### Done when

- [ ] No crawl starts without a recorded authorisation
- [ ] Concurrency is one and the user agent carries a contact
- [ ] The page ceiling stops the crawl with a reason
- [ ] Stopping halts within one request

### Watch out

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.

## Phase 2 · Findings, each with evidence and a source

Every check produces rows containing the page, what was found, and a link to the guidance it came from.

### Steps

1. Make checks data, not code branches
   A table of checks can be read, argued with and extended.
   Files: `src/checks/definitions.py`
2. Require a source URL on every check
   If it is not in published guidance, it is your preference.
   Files: `src/checks/loader.py`, `tests/test_sources.py`
3. Store the evidence, not just the verdict
   Nobody fixes a finding they cannot see.
   Files: `src/checks/run.py`

### Done when

- [ ] Every check has a source URL or the suite fails
- [ ] Findings carry the evidence that produced them
- [ ] No check emits a grade or a score
- [ ] Repeat runs on a fixture are identical

### Watch out

- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.

## Phase 3 · Performance measured honestly

Page performance is measured several times, reported against the published thresholds, and labelled as a local measurement.

### Steps

1. Measure three times and report the median and spread
   A single run on a laptop is a sample of one.
   Files: `src/perf/measure.py`, `tests/test_median.py`
2. Cite the thresholds and the date you checked them
   The names and the numbers have changed before and will again.
   Files: `src/checks/definitions.py`, `docs/thresholds.md`
3. Label every number as a laboratory measurement
   The published assessment uses real visits at a percentile, not your machine.
   Files: `src/report/render.py`

### Done when

- [ ] Three runs per page, median and spread reported
- [ ] Thresholds are cited with a checked date
- [ ] Every figure is labelled as a local run
- [ ] Render failures are recorded, not scored

### Watch out

- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.

## Phase 4 · What changed since last time

Two crawls of the same site can be compared, and the comparison is about findings appearing and being resolved.

### Steps

1. Keep crawls immutable
   A comparison against something that changed underneath is not a comparison.
   Files: `src/crawl/snapshot.py`
2. Match across redirects before diffing
   Otherwise a URL change looks like the site fell over.
   Files: `src/diff/match.py`, `tests/test_redirect_match.py`
3. Report changes, not movement in a figure
   The whole point of having no score is not to reintroduce one here.
   Files: `src/diff/report.py`

### Done when

- [ ] A fixed finding shows as resolved and nothing else moves
- [ ] Redirected pages are matched, not reported as removed
- [ ] Finished crawls are immutable
- [ ] The diff contains no aggregate number

### Watch out

- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.

## Not in this build

- Keyword ranking data.
- Backlink data.
- Competitor benchmarking crawls.
- A public site grading widget.

## After v1, if you want it

- Search console data for sites you own, which is the only field data available to you without renting an index
- A check that compares two renders of the same page to catch content that only appears with a browser
- An accessibility check set, kept separate from the search findings because the audience and the urgency are different

===== .env.example =====
# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sites, crawls, pages, findings and measurements.
DATABASE_PATH=./data/audits.db

# Required. Sent on every request. The contact address in it is not optional.
USER_AGENT=SiteAudit/1.0 (+mailto:you@example.com)

# Required. Between requests to the same origin. Lower it only with the owner's agreement.
REQUEST_DELAY_MS=1000

# Required. Ceiling per crawl unless the site record overrides it.
MAX_PAGES_DEFAULT=500

# Required. Runs per page. The report shows the median and the spread of these.
PERF_RUNS=3

# Required. Generated HTML and CSV reports. These get emailed, so keep them self-contained.
REPORT_DIR=./reports

# Required. Crawl detail is pruned after this; summary rows are kept for diffs.
CRAWL_RETENTION_DAYS=180

You are building a production product version of WooRank.

Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt.

===== PRODUCT.md =====
# WooRank · product brief

## Problem

The visible SEO auditing loop is buildable, but a credible replacement needs more than the first screen. WooRank earns its keep through proprietary data, index, crawl scale, so expect a weekend or multi-day build and a narrower personal scope.

## Product outcome

An auditor that can prove it was allowed to crawl, reports findings with evidence and a citation rather than a number out of one hundred, treats its own performance figures as laboratory runs, and produces a document somebody can act on.

## Target user

A builder who needs a maintainable product foundation, not a one-off demo.

## Required capabilities

- Python 3.12
- Playwright browsers
- permission to crawl the target site

## Explicit non-goals for v1

- Keyword ranking data.
- Backlink data.
- Competitor benchmarking crawls.
- A public site grading widget.
- historical competitor datasets and support
- fresh proprietary keyword, backlink, or ranking data
- proprietary web and keyword indexes
- fresh rank and clickstream data
- large-scale crawling infrastructure

## Success criteria

- No crawl is confirmed to start without a recorded authorisation
- Request concurrency is confirmed to be one by default with a contact in the user agent
- Every check is confirmed to carry a source URL
- No score or grade is confirmed to appear anywhere in the output
- Performance figures are confirmed to be medians of repeated runs, labelled as local

===== BRIEF.md =====
# Build brief · WooRank

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a site auditor that reports findings with their evidence instead of a mark out of one hundred to replace WooRank. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
The headline of the product you are replacing is a score out of one hundred. Do not build one.
A score like that is a weighted checklist, and nobody, including the people selling it, can defend
the weights. Two auditors grading the same site produce different numbers, the number moves when
the checklist changes rather than when the site does, and once a client sees it the work becomes
raising the number instead of fixing the site. Report findings: what was checked, on which page,
the evidence, how bad it is, and a link to the published guidance the check came from. Counts by
severity are allowed. A single figure summarising a website is not.
Second: crawl only sites you own or have written permission to audit. Record who authorised it and
when, before the first request. Identify your crawler honestly in the user agent with a contact,
respect the site's stated crawl directives, keep one request in flight at a time by default, and
put a page ceiling on every crawl. An auditor that hammers a small shared host is a denial of
service with a report attached.

### Stack (fixed, do not substitute)
- Runtime: A Python command line tool with a headless browser for rendering, output to files · an audit is a batch job that produces a document, not an application people log into
- Crawl: One request in flight by default, a delay between requests, a page ceiling per run · the site you are auditing belongs to somebody with a hosting bill
- Checks: Declarative check definitions, each carrying its own citation to published guidance · a check nobody can trace back to a source is an opinion in a table
- Storage: SQLite, one row per finding, with the evidence stored beside it · a finding you cannot show the evidence for cannot be argued
- Output: HTML and CSV reports generated from the findings table · the deliverable is the document, and it has to stand on its own

### Data model (create this before Phase 1)
- sites(id, origin, authorised_by, authorised_at, contact_email, max_pages, delay_ms)
- crawls(id, site_id, started_at, finished_at, pages_fetched, pages_skipped, stopped_reason)
- pages(id, crawl_id, url, status, content_type, bytes, fetched_at, render_ms, redirect_chain_json)
- findings(id, crawl_id, page_id, check_id, severity, evidence_json, source_url) · source_url is the published guidance, and a check without one does not ship
- measurements(id, page_id, metric, value, run_ordinal, method) · method is always lab, because that is what a local run is
- diffs(from_crawl, to_crawl, check_id, page_url, change) · appeared, resolved or unchanged
- There is no score column. There is no grade column. The report's headline is a count of findings by severity, and that is deliberate.

### Phase 1 · A crawl you are allowed to run
Build: Register a site with who authorised the audit, a contact address and the ceilings. Fetch with an
  honest user agent including that contact, one request at a time with a configurable delay, obeying
  the site's stated crawl directives and stopping at the page ceiling. Log every request with its
  status and timing. Provide a stop control that halts within one request.
Trap: Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
Done when: A crawl refuses to start without a recorded authorisation and contact address, tested. Request concurrency is one by default and every request carries the configured user agent with a contact, verified in a request log. The page ceiling stops the crawl and records the reason, tested against a site with more pages than the ceiling. Stopping a crawl halts it within one request and leaves the partial results usable.
Do not build yet: No checks, findings, performance or reports yet.

### Phase 2 · Findings, each with evidence and a source
Build: Define checks declaratively: an identifier, a severity, the selector or condition, the evidence to
  capture, and the URL of the published guidance behind it. Store one finding per page per check with
  the evidence attached, for instance the actual title text found, or the three images missing
  alternative text with their paths. A check with no source URL fails the test suite.
Trap: Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
Done when: Every check definition carries a source URL and the suite fails if one does not, tested by adding a check without one. Each finding stores the evidence that produced it and the report shows it inline. No check emits a numeric grade, verified by review. Two runs over an unchanged fixture site produce identical findings, tested.
Do not build yet: No performance measurement, diffing or reports yet.

### Phase 3 · Performance measured honestly
Build: Render each page in a headless browser and record the loading, interaction responsiveness and
  layout stability metrics the published guidance defines. Run each page three times and report the
  median with the spread. Compare against the published thresholds rather than thresholds you chose.
  Label every number as a local run, because that is what it is.
Trap: Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
Done when: Each page is measured at least three times and the report shows median and spread rather than a single figure, tested. Thresholds used in the report are the currently published ones and are cited with the date checked, verified in the check definitions. Every performance figure in the output is labelled as a local measurement. A page that fails to render is recorded as a failure with the error rather than silently scoring zero.
Do not build yet: No diffing between crawls, reports or scheduling yet.

### Phase 4 · What changed since last time
Build: Store crawls as immutable snapshots. Diff two crawls by page URL and check identifier, producing
  appeared, resolved and unchanged rows. Handle pages that were added, removed or redirected without
  reporting them as regressions. Show the diff as a list of changes, never as a movement in a number.
Trap: Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.
Done when: A finding fixed between crawls appears as resolved and nothing else changes, tested with two fixture crawls. A page that moved with a redirect is matched to its new URL rather than reported as removed, tested. A crawl snapshot is never modified after it finishes, asserted in the suite. The diff output contains no aggregate number, verified by review.
Do not build yet: No report generation, scheduling or storage limits yet.

### Phase 5 · A report you can hand to somebody
Build: Generate an HTML report that opens with no server and a CSV of every finding. Order by severity
  and then by how many pages are affected. Include the evidence, the source link and the crawl
  metadata, including who authorised it and when. Add scheduled re-crawls, and a retention setting
  that prunes old crawls while keeping their summary rows.
Trap: A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.
Done when: The HTML report opens from the file system with no server and no network requests, tested offline. Every finding in the report carries its evidence and its source link. The report states the authorisation, the crawl date, the page count and the tool version on the first screen. Retention prunes crawl detail beyond the configured age while keeping the summary rows needed for diffs, tested by moving the clock.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Keyword rankings, because getting them means either scraping a search engine against its terms or renting a data provider, and both are somebody else's index rather than your tool.
- Backlink data, which is a crawl of the entire web maintained continuously and is the one part of the paid product that genuinely cannot be rebuilt.
- Competitor benchmarking, since it means crawling sites nobody authorised you to crawl, which rule zero already rules out.
- A lead generation widget that grades a stranger's site, which is the score you refused to build, pointed at people who did not ask.

### README must contain
- That this tool produces findings and counts by severity, never a score or a grade, and why a weighted checklist score cannot be defended
- That a crawl requires recorded authorisation, a contact in the user agent, one request in flight by default and a page ceiling
- That every check cites the published guidance it came from, and that a check without a source does not ship
- That performance figures are local laboratory runs reported as a median of three with the spread, while the published thresholds are assessed at a percentile of real visits
- What the retention setting prunes, how large a crawl of a given size is on disk, and where reports are written

===== ARCHITECTURE.md =====
# Architecture · WooRank

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | A Python command line tool with a headless browser for rendering, output to files | an audit is a batch job that produces a document, not an application people log into |
| Crawl | One request in flight by default, a delay between requests, a page ceiling per run | the site you are auditing belongs to somebody with a hosting bill |
| Checks | Declarative check definitions, each carrying its own citation to published guidance | a check nobody can trace back to a source is an opinion in a table |
| Storage | SQLite, one row per finding, with the evidence stored beside it | a finding you cannot show the evidence for cannot be argued |
| Output | HTML and CSV reports generated from the findings table | the deliverable is the document, and it has to stand on its own |

## Modules

Each module has one owner concern and a documented way to replace it.

| Module | Owns | How to replace it |
| --- | --- | --- |
| Sites | Authorisation, contact and ceilings. | Not swappable for crawling first and asking later. |
| Crawl | Fetching, rendering and request logging. | Any fetcher, provided the politeness defaults survive. |
| Checks | Declarative definitions with source links. | Any check set, provided each one cites published guidance. |
| Performance | Repeated measurement and medians. | Any harness, provided the output stays labelled as lab data. |
| Report | Offline HTML and CSV, ordered by severity. | Any format, provided no aggregate score appears. |

## Configuration

Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists.

- `DATABASE_PATH` · required · Sites, crawls, pages, findings and measurements.
- `USER_AGENT` · required · Sent on every request. The contact address in it is not optional.
- `REQUEST_DELAY_MS` · required · Between requests to the same origin. Lower it only with the owner's agreement.
- `MAX_PAGES_DEFAULT` · required · Ceiling per crawl unless the site record overrides it.
- `PERF_RUNS` · required · Runs per page. The report shows the median and the spread of these.
- `REPORT_DIR` · required · Generated HTML and CSV reports. These get emailed, so keep them self-contained.
- `CRAWL_RETENTION_DAYS` · required · Crawl detail is pruned after this; summary rows are kept for diffs.

## Production baseline

- Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives.
- Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised.
- Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts.
- Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists.
- Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path.

## Decision records

For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.

===== AGENTS.md =====
# Agent instructions · WooRank product build

- Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A Python command line tool with a headless browser for rendering, output to files, One request in flight by default, a delay between requests, a page ceiling per run, Declarative check definitions, each carrying its own citation to published guidance, SQLite, one row per finding, with the evidence stored beside it, HTML and CSV reports generated from the findings table.
- Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit.
- Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present.
- Never invent cryptography or silently weaken a requirement to make a check pass.
- Put every external service behind an interface with a deterministic fake for tests.
- Add migrations and rollback or recovery notes for every persistent data change.
- Log useful operational context without credentials, tokens, passwords or personal data.
- Update documentation and run every check before completing a milestone.

## Known traps

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.
- A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.

===== MILESTONES.md =====
# Delivery milestones · WooRank

Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer.

## M1 · A crawl you are allowed to run

Auditing a site requires recorded authorisation, and the crawler behaves like a guest.

### Steps

1. Record the authorisation before any request
   If you cannot name who agreed, do not fetch the first page.
   Files: `src/sites.py`, `tests/test_authorisation.py`
2. One request in flight, with a delay and a ceiling
   Politeness is a default, not a flag.
   Files: `src/crawl/fetcher.py`
3. Log every request with status and timing
   This log is your answer when somebody asks what you did to their server.
   Files: `src/crawl/log.py`

### Done when

- [ ] No crawl starts without a recorded authorisation
- [ ] Concurrency is one and the user agent carries a contact
- [ ] The page ceiling stops the crawl with a reason
- [ ] Stopping halts within one request

### Watch out

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.

## M2 · Findings, each with evidence and a source

Every check produces rows containing the page, what was found, and a link to the guidance it came from.

### Steps

1. Make checks data, not code branches
   A table of checks can be read, argued with and extended.
   Files: `src/checks/definitions.py`
2. Require a source URL on every check
   If it is not in published guidance, it is your preference.
   Files: `src/checks/loader.py`, `tests/test_sources.py`
3. Store the evidence, not just the verdict
   Nobody fixes a finding they cannot see.
   Files: `src/checks/run.py`

### Done when

- [ ] Every check has a source URL or the suite fails
- [ ] Findings carry the evidence that produced them
- [ ] No check emits a grade or a score
- [ ] Repeat runs on a fixture are identical

### Watch out

- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.

## M3 · Performance measured honestly

Page performance is measured several times, reported against the published thresholds, and labelled as a local measurement.

### Steps

1. Measure three times and report the median and spread
   A single run on a laptop is a sample of one.
   Files: `src/perf/measure.py`, `tests/test_median.py`
2. Cite the thresholds and the date you checked them
   The names and the numbers have changed before and will again.
   Files: `src/checks/definitions.py`, `docs/thresholds.md`
3. Label every number as a laboratory measurement
   The published assessment uses real visits at a percentile, not your machine.
   Files: `src/report/render.py`

### Done when

- [ ] Three runs per page, median and spread reported
- [ ] Thresholds are cited with a checked date
- [ ] Every figure is labelled as a local run
- [ ] Render failures are recorded, not scored

### Watch out

- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.

## M4 · What changed since last time

Two crawls of the same site can be compared, and the comparison is about findings appearing and being resolved.

### Steps

1. Keep crawls immutable
   A comparison against something that changed underneath is not a comparison.
   Files: `src/crawl/snapshot.py`
2. Match across redirects before diffing
   Otherwise a URL change looks like the site fell over.
   Files: `src/diff/match.py`, `tests/test_redirect_match.py`
3. Report changes, not movement in a figure
   The whole point of having no score is not to reintroduce one here.
   Files: `src/diff/report.py`

### Done when

- [ ] A fixed finding shows as resolved and nothing else moves
- [ ] Redirected pages are matched, not reported as removed
- [ ] Finished crawls are immutable
- [ ] The diff contains no aggregate number

### Watch out

- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.

## M5 · A report you can hand to somebody (production only)

The deliverable is a self-contained document with evidence and sources, and the storage it consumes has a ceiling.

### Steps

1. Make the report a single file that works offline
   It will be emailed, and it should still work.
   Files: `src/report/html.py`
2. Group by check, order by severity and page count
   Readable is what makes it get acted on.
   Files: `src/report/order.py`
3. Prune old crawl detail, keep the summaries
   Rendered pages and evidence are most of the disk use here.
   Files: `src/retention.py`, `tests/test_retention.py`

### Done when

- [ ] The report opens offline with no requests
- [ ] Every finding shows evidence and a source
- [ ] Authorisation and crawl metadata lead the report
- [ ] Retention prunes detail and keeps summaries

### Watch out

- A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.

===== OPERATIONS.md =====
# Operations · WooRank

## Backup

The database and the reports directory. Rendered page copies are the bulk of the disk and are regenerable, so exclude them and say so in the README.

## Restore

Restore the database, run a crawl against the fixture site, and confirm the planted findings appear exactly as before. If they do not, a check definition changed and the historical diffs are no longer comparable.

Do a restore drill before the first real user, and write the date here when it passes.

## Monitoring

Disk used per crawl, the failure rate of page renders, and the time per page against the delay setting. A render failure rate that climbs usually means the browser was updated, not that the site broke.

## Incident checklist

The incident to plan for is a site owner asking why your crawler was on their server. The request log, the user agent with its contact, and the authorisation record on the site row are the whole answer, which is why all three exist before the first fetch.

1. Contain the issue without destroying evidence or user data.
2. Record the timeline and affected scope.
3. Rotate exposed secrets and revoke compromised sessions or credentials.
4. Restore from a verified backup when needed.
5. Document the root cause, the remediation and the regression test.

## Release gate

- [ ] No crawl is confirmed to start without a recorded authorisation
- [ ] Request concurrency is confirmed to be one by default with a contact in the user agent
- [ ] Every check is confirmed to carry a source URL
- [ ] No score or grade is confirmed to appear anywhere in the output
- [ ] Performance figures are confirmed to be medians of repeated runs, labelled as local

## Launch constraint

Do not market omitted WooRank capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.

===== .env.example =====
# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sites, crawls, pages, findings and measurements.
DATABASE_PATH=./data/audits.db

# Required. Sent on every request. The contact address in it is not optional.
USER_AGENT=SiteAudit/1.0 (+mailto:you@example.com)

# Required. Between requests to the same origin. Lower it only with the owner's agreement.
REQUEST_DELAY_MS=1000

# Required. Ceiling per crawl unless the site record overrides it.
MAX_PAGES_DEFAULT=500

# Required. Runs per page. The report shows the median and the spread of these.
PERF_RUNS=3

# Required. Generated HTML and CSV reports. These get emailed, so keep them self-contained.
REPORT_DIR=./reports

# Required. Crawl detail is pruned after this; summary rows are kept for diffs.
CRAWL_RETENTION_DAYS=180

# WooRank · indie build

A self-hosted site auditor whose hard problems are crawling somebody's site politely and provably with permission, reporting findings with evidence instead of an invented score, and measuring performance in a way that does not pretend a single local run is what search engines actually see.

Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts.

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | A Python command line tool with a headless browser for rendering, output to files | an audit is a batch job that produces a document, not an application people log into |
| Crawl | One request in flight by default, a delay between requests, a page ceiling per run | the site you are auditing belongs to somebody with a hosting bill |
| Checks | Declarative check definitions, each carrying its own citation to published guidance | a check nobody can trace back to a source is an opinion in a table |
| Storage | SQLite, one row per finding, with the evidence stored beside it | a finding you cannot show the evidence for cannot be argued |
| Output | HTML and CSV reports generated from the findings table | the deliverable is the document, and it has to stand on its own |

## Before you start

Have every one of these ready. The plan assumes them from step one.

- [ ] **Python 3.12 or newer** · free
  - Why: Runs the crawler, the checks and the report generation.
  - Get it: python.org or your package manager.
  - Verify: python3 --version prints 3.12 or higher
- [ ] **A headless browser automation library with its browsers installed** · free, plus a few hundred megabytes of disk
  - Why: Half the checks and all the performance work need a rendered page, since a large share of sites assemble their content in the browser.
  - Get it: Install the library and run its browser download step.
  - Verify: A scripted page load returns rendered content and a performance entry
- [ ] **Written authorisation for every site you will crawl** · free
  - Why: This is rule zero and it is also the difference between an audit and an unsolicited crawl of somebody's server.
  - Get it: For your own sites, a note to yourself. For a client, an email saying which origin, at what rate, and until when.
  - Verify: Every site row has an authorised_by value and a date before its first crawl
- [ ] **The current published performance thresholds and metric names** · free
  - Why: The metric set has been changed before, with one metric replaced outright in 2024, so copying numbers from memory dates your tool immediately.
  - Get it: Read the current published guidance, record each metric, its threshold and the date you checked.
  - Verify: docs/thresholds.md lists each metric with its value and date, and the checks reference it
- [ ] **A small site you control, with deliberate faults** · free
  - Why: Every check, the diff and the redirect matching need a site whose answers you already know, and testing against a real client site is how you discover your crawler is impolite.
  - Get it: Publish a handful of static pages locally with missing alternative text, a duplicate title and a redirect.
  - Verify: A crawl of the fixture produces exactly the findings you planted

## Quick start

Follow Phase 1 in `BUILD_PLAN.md`; it creates the project.

Then copy `.env.example` to `.env` and fill in the values it documents.

## Honest limits

This build deliberately does not replace:

- Keyword ranking data.
- Backlink data.
- Competitor benchmarking crawls.
- A public site grading widget.
- historical competitor datasets and support
- fresh proprietary keyword, backlink, or ranking data
- proprietary web and keyword indexes
- fresh rank and clickstream data
- large-scale crawling infrastructure

If one of those is essential to you, that is the reason to keep paying for WooRank, and the README should say so rather than pretend.

# Build brief · WooRank

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a site auditor that reports findings with their evidence instead of a mark out of one hundred to replace WooRank. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
The headline of the product you are replacing is a score out of one hundred. Do not build one.
A score like that is a weighted checklist, and nobody, including the people selling it, can defend
the weights. Two auditors grading the same site produce different numbers, the number moves when
the checklist changes rather than when the site does, and once a client sees it the work becomes
raising the number instead of fixing the site. Report findings: what was checked, on which page,
the evidence, how bad it is, and a link to the published guidance the check came from. Counts by
severity are allowed. A single figure summarising a website is not.
Second: crawl only sites you own or have written permission to audit. Record who authorised it and
when, before the first request. Identify your crawler honestly in the user agent with a contact,
respect the site's stated crawl directives, keep one request in flight at a time by default, and
put a page ceiling on every crawl. An auditor that hammers a small shared host is a denial of
service with a report attached.

### Stack (fixed, do not substitute)
- Runtime: A Python command line tool with a headless browser for rendering, output to files · an audit is a batch job that produces a document, not an application people log into
- Crawl: One request in flight by default, a delay between requests, a page ceiling per run · the site you are auditing belongs to somebody with a hosting bill
- Checks: Declarative check definitions, each carrying its own citation to published guidance · a check nobody can trace back to a source is an opinion in a table
- Storage: SQLite, one row per finding, with the evidence stored beside it · a finding you cannot show the evidence for cannot be argued
- Output: HTML and CSV reports generated from the findings table · the deliverable is the document, and it has to stand on its own

### Data model (create this before Phase 1)
- sites(id, origin, authorised_by, authorised_at, contact_email, max_pages, delay_ms)
- crawls(id, site_id, started_at, finished_at, pages_fetched, pages_skipped, stopped_reason)
- pages(id, crawl_id, url, status, content_type, bytes, fetched_at, render_ms, redirect_chain_json)
- findings(id, crawl_id, page_id, check_id, severity, evidence_json, source_url) · source_url is the published guidance, and a check without one does not ship
- measurements(id, page_id, metric, value, run_ordinal, method) · method is always lab, because that is what a local run is
- diffs(from_crawl, to_crawl, check_id, page_url, change) · appeared, resolved or unchanged
- There is no score column. There is no grade column. The report's headline is a count of findings by severity, and that is deliberate.

### Phase 1 · A crawl you are allowed to run
Build: Register a site with who authorised the audit, a contact address and the ceilings. Fetch with an
  honest user agent including that contact, one request at a time with a configurable delay, obeying
  the site's stated crawl directives and stopping at the page ceiling. Log every request with its
  status and timing. Provide a stop control that halts within one request.
Trap: Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
Done when: A crawl refuses to start without a recorded authorisation and contact address, tested. Request concurrency is one by default and every request carries the configured user agent with a contact, verified in a request log. The page ceiling stops the crawl and records the reason, tested against a site with more pages than the ceiling. Stopping a crawl halts it within one request and leaves the partial results usable.
Do not build yet: No checks, findings, performance or reports yet.

### Phase 2 · Findings, each with evidence and a source
Build: Define checks declaratively: an identifier, a severity, the selector or condition, the evidence to
  capture, and the URL of the published guidance behind it. Store one finding per page per check with
  the evidence attached, for instance the actual title text found, or the three images missing
  alternative text with their paths. A check with no source URL fails the test suite.
Trap: Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
Done when: Every check definition carries a source URL and the suite fails if one does not, tested by adding a check without one. Each finding stores the evidence that produced it and the report shows it inline. No check emits a numeric grade, verified by review. Two runs over an unchanged fixture site produce identical findings, tested.
Do not build yet: No performance measurement, diffing or reports yet.

### Phase 3 · Performance measured honestly
Build: Render each page in a headless browser and record the loading, interaction responsiveness and
  layout stability metrics the published guidance defines. Run each page three times and report the
  median with the spread. Compare against the published thresholds rather than thresholds you chose.
  Label every number as a local run, because that is what it is.
Trap: Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
Done when: Each page is measured at least three times and the report shows median and spread rather than a single figure, tested. Thresholds used in the report are the currently published ones and are cited with the date checked, verified in the check definitions. Every performance figure in the output is labelled as a local measurement. A page that fails to render is recorded as a failure with the error rather than silently scoring zero.
Do not build yet: No diffing between crawls, reports or scheduling yet.

### Phase 4 · What changed since last time
Build: Store crawls as immutable snapshots. Diff two crawls by page URL and check identifier, producing
  appeared, resolved and unchanged rows. Handle pages that were added, removed or redirected without
  reporting them as regressions. Show the diff as a list of changes, never as a movement in a number.
Trap: Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.
Done when: A finding fixed between crawls appears as resolved and nothing else changes, tested with two fixture crawls. A page that moved with a redirect is matched to its new URL rather than reported as removed, tested. A crawl snapshot is never modified after it finishes, asserted in the suite. The diff output contains no aggregate number, verified by review.
Do not build yet: No report generation, scheduling or storage limits yet.

### Phase 5 · A report you can hand to somebody
Build: Generate an HTML report that opens with no server and a CSV of every finding. Order by severity
  and then by how many pages are affected. Include the evidence, the source link and the crawl
  metadata, including who authorised it and when. Add scheduled re-crawls, and a retention setting
  that prunes old crawls while keeping their summary rows.
Trap: A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.
Done when: The HTML report opens from the file system with no server and no network requests, tested offline. Every finding in the report carries its evidence and its source link. The report states the authorisation, the crawl date, the page count and the tool version on the first screen. Retention prunes crawl detail beyond the configured age while keeping the summary rows needed for diffs, tested by moving the clock.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Keyword rankings, because getting them means either scraping a search engine against its terms or renting a data provider, and both are somebody else's index rather than your tool.
- Backlink data, which is a crawl of the entire web maintained continuously and is the one part of the paid product that genuinely cannot be rebuilt.
- Competitor benchmarking, since it means crawling sites nobody authorised you to crawl, which rule zero already rules out.
- A lead generation widget that grades a stranger's site, which is the score you refused to build, pointed at people who did not ask.

### README must contain
- That this tool produces findings and counts by severity, never a score or a grade, and why a weighted checklist score cannot be defended
- That a crawl requires recorded authorisation, a contact in the user agent, one request in flight by default and a page ceiling
- That every check cites the published guidance it came from, and that a check without a source does not ship
- That performance figures are local laboratory runs reported as a median of three with the spread, while the published thresholds are assessed at a percentile of real visits
- What the retention setting prunes, how large a crawl of a given size is on disk, and where reports are written

# Agent instructions · WooRank indie build

- Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python command line tool with a headless browser for rendering, output to files, One request in flight by default, a delay between requests, a page ceiling per run, Declarative check definitions, each carrying its own citation to published guidance, SQLite, one row per finding, with the evidence stored beside it, HTML and CSV reports generated from the findings table. Do not substitute.
- Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes.
- Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name.
- Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced.
- Do not invent cryptography, security guarantees, APIs or compliance claims.
- Add a focused test for every destructive, security-sensitive or data-loss path the plan names.
- Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs".

## Known traps

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.

# Build plan · WooRank

A self-hosted site auditor whose hard problems are crawling somebody's site politely and provably with permission, reporting findings with evidence instead of an invented score, and measuring performance in a way that does not pretend a single local run is what search engines actually see.

Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note.

## Phase 1 · A crawl you are allowed to run

Auditing a site requires recorded authorisation, and the crawler behaves like a guest.

### Steps

1. Record the authorisation before any request
   If you cannot name who agreed, do not fetch the first page.
   Files: `src/sites.py`, `tests/test_authorisation.py`
2. One request in flight, with a delay and a ceiling
   Politeness is a default, not a flag.
   Files: `src/crawl/fetcher.py`
3. Log every request with status and timing
   This log is your answer when somebody asks what you did to their server.
   Files: `src/crawl/log.py`

### Done when

- [ ] No crawl starts without a recorded authorisation
- [ ] Concurrency is one and the user agent carries a contact
- [ ] The page ceiling stops the crawl with a reason
- [ ] Stopping halts within one request

### Watch out

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.

## Phase 2 · Findings, each with evidence and a source

Every check produces rows containing the page, what was found, and a link to the guidance it came from.

### Steps

1. Make checks data, not code branches
   A table of checks can be read, argued with and extended.
   Files: `src/checks/definitions.py`
2. Require a source URL on every check
   If it is not in published guidance, it is your preference.
   Files: `src/checks/loader.py`, `tests/test_sources.py`
3. Store the evidence, not just the verdict
   Nobody fixes a finding they cannot see.
   Files: `src/checks/run.py`

### Done when

- [ ] Every check has a source URL or the suite fails
- [ ] Findings carry the evidence that produced them
- [ ] No check emits a grade or a score
- [ ] Repeat runs on a fixture are identical

### Watch out

- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.

## Phase 3 · Performance measured honestly

Page performance is measured several times, reported against the published thresholds, and labelled as a local measurement.

### Steps

1. Measure three times and report the median and spread
   A single run on a laptop is a sample of one.
   Files: `src/perf/measure.py`, `tests/test_median.py`
2. Cite the thresholds and the date you checked them
   The names and the numbers have changed before and will again.
   Files: `src/checks/definitions.py`, `docs/thresholds.md`
3. Label every number as a laboratory measurement
   The published assessment uses real visits at a percentile, not your machine.
   Files: `src/report/render.py`

### Done when

- [ ] Three runs per page, median and spread reported
- [ ] Thresholds are cited with a checked date
- [ ] Every figure is labelled as a local run
- [ ] Render failures are recorded, not scored

### Watch out

- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.

## Phase 4 · What changed since last time

Two crawls of the same site can be compared, and the comparison is about findings appearing and being resolved.

### Steps

1. Keep crawls immutable
   A comparison against something that changed underneath is not a comparison.
   Files: `src/crawl/snapshot.py`
2. Match across redirects before diffing
   Otherwise a URL change looks like the site fell over.
   Files: `src/diff/match.py`, `tests/test_redirect_match.py`
3. Report changes, not movement in a figure
   The whole point of having no score is not to reintroduce one here.
   Files: `src/diff/report.py`

### Done when

- [ ] A fixed finding shows as resolved and nothing else moves
- [ ] Redirected pages are matched, not reported as removed
- [ ] Finished crawls are immutable
- [ ] The diff contains no aggregate number

### Watch out

- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.

## Not in this build

- Keyword ranking data.
- Backlink data.
- Competitor benchmarking crawls.
- A public site grading widget.

## After v1, if you want it

- Search console data for sites you own, which is the only field data available to you without renting an index
- A check that compares two renders of the same page to catch content that only appears with a browser
- An accessibility check set, kept separate from the search findings because the audience and the urgency are different

# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sites, crawls, pages, findings and measurements.
DATABASE_PATH=./data/audits.db

# Required. Sent on every request. The contact address in it is not optional.
USER_AGENT=SiteAudit/1.0 (+mailto:you@example.com)

# Required. Between requests to the same origin. Lower it only with the owner's agreement.
REQUEST_DELAY_MS=1000

# Required. Ceiling per crawl unless the site record overrides it.
MAX_PAGES_DEFAULT=500

# Required. Runs per page. The report shows the median and the spread of these.
PERF_RUNS=3

# Required. Generated HTML and CSV reports. These get emailed, so keep them self-contained.
REPORT_DIR=./reports

# Required. Crawl detail is pruned after this; summary rows are kept for diffs.
CRAWL_RETENTION_DAYS=180

# WooRank · product brief

## Problem

The visible SEO auditing loop is buildable, but a credible replacement needs more than the first screen. WooRank earns its keep through proprietary data, index, crawl scale, so expect a weekend or multi-day build and a narrower personal scope.

## Product outcome

An auditor that can prove it was allowed to crawl, reports findings with evidence and a citation rather than a number out of one hundred, treats its own performance figures as laboratory runs, and produces a document somebody can act on.

## Target user

A builder who needs a maintainable product foundation, not a one-off demo.

## Required capabilities

- Python 3.12
- Playwright browsers
- permission to crawl the target site

## Explicit non-goals for v1

- Keyword ranking data.
- Backlink data.
- Competitor benchmarking crawls.
- A public site grading widget.
- historical competitor datasets and support
- fresh proprietary keyword, backlink, or ranking data
- proprietary web and keyword indexes
- fresh rank and clickstream data
- large-scale crawling infrastructure

## Success criteria

- No crawl is confirmed to start without a recorded authorisation
- Request concurrency is confirmed to be one by default with a contact in the user agent
- Every check is confirmed to carry a source URL
- No score or grade is confirmed to appear anywhere in the output
- Performance figures are confirmed to be medians of repeated runs, labelled as local

# Build brief · WooRank

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a site auditor that reports findings with their evidence instead of a mark out of one hundred to replace WooRank. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
The headline of the product you are replacing is a score out of one hundred. Do not build one.
A score like that is a weighted checklist, and nobody, including the people selling it, can defend
the weights. Two auditors grading the same site produce different numbers, the number moves when
the checklist changes rather than when the site does, and once a client sees it the work becomes
raising the number instead of fixing the site. Report findings: what was checked, on which page,
the evidence, how bad it is, and a link to the published guidance the check came from. Counts by
severity are allowed. A single figure summarising a website is not.
Second: crawl only sites you own or have written permission to audit. Record who authorised it and
when, before the first request. Identify your crawler honestly in the user agent with a contact,
respect the site's stated crawl directives, keep one request in flight at a time by default, and
put a page ceiling on every crawl. An auditor that hammers a small shared host is a denial of
service with a report attached.

### Stack (fixed, do not substitute)
- Runtime: A Python command line tool with a headless browser for rendering, output to files · an audit is a batch job that produces a document, not an application people log into
- Crawl: One request in flight by default, a delay between requests, a page ceiling per run · the site you are auditing belongs to somebody with a hosting bill
- Checks: Declarative check definitions, each carrying its own citation to published guidance · a check nobody can trace back to a source is an opinion in a table
- Storage: SQLite, one row per finding, with the evidence stored beside it · a finding you cannot show the evidence for cannot be argued
- Output: HTML and CSV reports generated from the findings table · the deliverable is the document, and it has to stand on its own

### Data model (create this before Phase 1)
- sites(id, origin, authorised_by, authorised_at, contact_email, max_pages, delay_ms)
- crawls(id, site_id, started_at, finished_at, pages_fetched, pages_skipped, stopped_reason)
- pages(id, crawl_id, url, status, content_type, bytes, fetched_at, render_ms, redirect_chain_json)
- findings(id, crawl_id, page_id, check_id, severity, evidence_json, source_url) · source_url is the published guidance, and a check without one does not ship
- measurements(id, page_id, metric, value, run_ordinal, method) · method is always lab, because that is what a local run is
- diffs(from_crawl, to_crawl, check_id, page_url, change) · appeared, resolved or unchanged
- There is no score column. There is no grade column. The report's headline is a count of findings by severity, and that is deliberate.

### Phase 1 · A crawl you are allowed to run
Build: Register a site with who authorised the audit, a contact address and the ceilings. Fetch with an
  honest user agent including that contact, one request at a time with a configurable delay, obeying
  the site's stated crawl directives and stopping at the page ceiling. Log every request with its
  status and timing. Provide a stop control that halts within one request.
Trap: Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
Done when: A crawl refuses to start without a recorded authorisation and contact address, tested. Request concurrency is one by default and every request carries the configured user agent with a contact, verified in a request log. The page ceiling stops the crawl and records the reason, tested against a site with more pages than the ceiling. Stopping a crawl halts it within one request and leaves the partial results usable.
Do not build yet: No checks, findings, performance or reports yet.

### Phase 2 · Findings, each with evidence and a source
Build: Define checks declaratively: an identifier, a severity, the selector or condition, the evidence to
  capture, and the URL of the published guidance behind it. Store one finding per page per check with
  the evidence attached, for instance the actual title text found, or the three images missing
  alternative text with their paths. A check with no source URL fails the test suite.
Trap: Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
Done when: Every check definition carries a source URL and the suite fails if one does not, tested by adding a check without one. Each finding stores the evidence that produced it and the report shows it inline. No check emits a numeric grade, verified by review. Two runs over an unchanged fixture site produce identical findings, tested.
Do not build yet: No performance measurement, diffing or reports yet.

### Phase 3 · Performance measured honestly
Build: Render each page in a headless browser and record the loading, interaction responsiveness and
  layout stability metrics the published guidance defines. Run each page three times and report the
  median with the spread. Compare against the published thresholds rather than thresholds you chose.
  Label every number as a local run, because that is what it is.
Trap: Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
Done when: Each page is measured at least three times and the report shows median and spread rather than a single figure, tested. Thresholds used in the report are the currently published ones and are cited with the date checked, verified in the check definitions. Every performance figure in the output is labelled as a local measurement. A page that fails to render is recorded as a failure with the error rather than silently scoring zero.
Do not build yet: No diffing between crawls, reports or scheduling yet.

### Phase 4 · What changed since last time
Build: Store crawls as immutable snapshots. Diff two crawls by page URL and check identifier, producing
  appeared, resolved and unchanged rows. Handle pages that were added, removed or redirected without
  reporting them as regressions. Show the diff as a list of changes, never as a movement in a number.
Trap: Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.
Done when: A finding fixed between crawls appears as resolved and nothing else changes, tested with two fixture crawls. A page that moved with a redirect is matched to its new URL rather than reported as removed, tested. A crawl snapshot is never modified after it finishes, asserted in the suite. The diff output contains no aggregate number, verified by review.
Do not build yet: No report generation, scheduling or storage limits yet.

### Phase 5 · A report you can hand to somebody
Build: Generate an HTML report that opens with no server and a CSV of every finding. Order by severity
  and then by how many pages are affected. Include the evidence, the source link and the crawl
  metadata, including who authorised it and when. Add scheduled re-crawls, and a retention setting
  that prunes old crawls while keeping their summary rows.
Trap: A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.
Done when: The HTML report opens from the file system with no server and no network requests, tested offline. Every finding in the report carries its evidence and its source link. The report states the authorisation, the crawl date, the page count and the tool version on the first screen. Retention prunes crawl detail beyond the configured age while keeping the summary rows needed for diffs, tested by moving the clock.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Keyword rankings, because getting them means either scraping a search engine against its terms or renting a data provider, and both are somebody else's index rather than your tool.
- Backlink data, which is a crawl of the entire web maintained continuously and is the one part of the paid product that genuinely cannot be rebuilt.
- Competitor benchmarking, since it means crawling sites nobody authorised you to crawl, which rule zero already rules out.
- A lead generation widget that grades a stranger's site, which is the score you refused to build, pointed at people who did not ask.

### README must contain
- That this tool produces findings and counts by severity, never a score or a grade, and why a weighted checklist score cannot be defended
- That a crawl requires recorded authorisation, a contact in the user agent, one request in flight by default and a page ceiling
- That every check cites the published guidance it came from, and that a check without a source does not ship
- That performance figures are local laboratory runs reported as a median of three with the spread, while the published thresholds are assessed at a percentile of real visits
- What the retention setting prunes, how large a crawl of a given size is on disk, and where reports are written

# Architecture · WooRank

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | A Python command line tool with a headless browser for rendering, output to files | an audit is a batch job that produces a document, not an application people log into |
| Crawl | One request in flight by default, a delay between requests, a page ceiling per run | the site you are auditing belongs to somebody with a hosting bill |
| Checks | Declarative check definitions, each carrying its own citation to published guidance | a check nobody can trace back to a source is an opinion in a table |
| Storage | SQLite, one row per finding, with the evidence stored beside it | a finding you cannot show the evidence for cannot be argued |
| Output | HTML and CSV reports generated from the findings table | the deliverable is the document, and it has to stand on its own |

## Modules

Each module has one owner concern and a documented way to replace it.

| Module | Owns | How to replace it |
| --- | --- | --- |
| Sites | Authorisation, contact and ceilings. | Not swappable for crawling first and asking later. |
| Crawl | Fetching, rendering and request logging. | Any fetcher, provided the politeness defaults survive. |
| Checks | Declarative definitions with source links. | Any check set, provided each one cites published guidance. |
| Performance | Repeated measurement and medians. | Any harness, provided the output stays labelled as lab data. |
| Report | Offline HTML and CSV, ordered by severity. | Any format, provided no aggregate score appears. |

## Configuration

Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists.

- `DATABASE_PATH` · required · Sites, crawls, pages, findings and measurements.
- `USER_AGENT` · required · Sent on every request. The contact address in it is not optional.
- `REQUEST_DELAY_MS` · required · Between requests to the same origin. Lower it only with the owner's agreement.
- `MAX_PAGES_DEFAULT` · required · Ceiling per crawl unless the site record overrides it.
- `PERF_RUNS` · required · Runs per page. The report shows the median and the spread of these.
- `REPORT_DIR` · required · Generated HTML and CSV reports. These get emailed, so keep them self-contained.
- `CRAWL_RETENTION_DAYS` · required · Crawl detail is pruned after this; summary rows are kept for diffs.

## Production baseline

- Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives.
- Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised.
- Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts.
- Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists.
- Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path.

## Decision records

For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.

# Agent instructions · WooRank product build

- Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A Python command line tool with a headless browser for rendering, output to files, One request in flight by default, a delay between requests, a page ceiling per run, Declarative check definitions, each carrying its own citation to published guidance, SQLite, one row per finding, with the evidence stored beside it, HTML and CSV reports generated from the findings table.
- Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit.
- Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present.
- Never invent cryptography or silently weaken a requirement to make a check pass.
- Put every external service behind an interface with a deterministic fake for tests.
- Add migrations and rollback or recovery notes for every persistent data change.
- Log useful operational context without credentials, tokens, passwords or personal data.
- Update documentation and run every check before completing a milestone.

## Known traps

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.
- A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.

# Delivery milestones · WooRank

Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer.

## M1 · A crawl you are allowed to run

Auditing a site requires recorded authorisation, and the crawler behaves like a guest.

### Steps

1. Record the authorisation before any request
   If you cannot name who agreed, do not fetch the first page.
   Files: `src/sites.py`, `tests/test_authorisation.py`
2. One request in flight, with a delay and a ceiling
   Politeness is a default, not a flag.
   Files: `src/crawl/fetcher.py`
3. Log every request with status and timing
   This log is your answer when somebody asks what you did to their server.
   Files: `src/crawl/log.py`

### Done when

- [ ] No crawl starts without a recorded authorisation
- [ ] Concurrency is one and the user agent carries a contact
- [ ] The page ceiling stops the crawl with a reason
- [ ] Stopping halts within one request

### Watch out

- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.

## M2 · Findings, each with evidence and a source

Every check produces rows containing the page, what was found, and a link to the guidance it came from.

### Steps

1. Make checks data, not code branches
   A table of checks can be read, argued with and extended.
   Files: `src/checks/definitions.py`
2. Require a source URL on every check
   If it is not in published guidance, it is your preference.
   Files: `src/checks/loader.py`, `tests/test_sources.py`
3. Store the evidence, not just the verdict
   Nobody fixes a finding they cannot see.
   Files: `src/checks/run.py`

### Done when

- [ ] Every check has a source URL or the suite fails
- [ ] Findings carry the evidence that produced them
- [ ] No check emits a grade or a score
- [ ] Repeat runs on a fixture are identical

### Watch out

- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.

## M3 · Performance measured honestly

Page performance is measured several times, reported against the published thresholds, and labelled as a local measurement.

### Steps

1. Measure three times and report the median and spread
   A single run on a laptop is a sample of one.
   Files: `src/perf/measure.py`, `tests/test_median.py`
2. Cite the thresholds and the date you checked them
   The names and the numbers have changed before and will again.
   Files: `src/checks/definitions.py`, `docs/thresholds.md`
3. Label every number as a laboratory measurement
   The published assessment uses real visits at a percentile, not your machine.
   Files: `src/report/render.py`

### Done when

- [ ] Three runs per page, median and spread reported
- [ ] Thresholds are cited with a checked date
- [ ] Every figure is labelled as a local run
- [ ] Render failures are recorded, not scored

### Watch out

- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.

## M4 · What changed since last time

Two crawls of the same site can be compared, and the comparison is about findings appearing and being resolved.

### Steps

1. Keep crawls immutable
   A comparison against something that changed underneath is not a comparison.
   Files: `src/crawl/snapshot.py`
2. Match across redirects before diffing
   Otherwise a URL change looks like the site fell over.
   Files: `src/diff/match.py`, `tests/test_redirect_match.py`
3. Report changes, not movement in a figure
   The whole point of having no score is not to reintroduce one here.
   Files: `src/diff/report.py`

### Done when

- [ ] A fixed finding shows as resolved and nothing else moves
- [ ] Redirected pages are matched, not reported as removed
- [ ] Finished crawls are immutable
- [ ] The diff contains no aggregate number

### Watch out

- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.

## M5 · A report you can hand to somebody (production only)

The deliverable is a self-contained document with evidence and sources, and the storage it consumes has a ceiling.

### Steps

1. Make the report a single file that works offline
   It will be emailed, and it should still work.
   Files: `src/report/html.py`
2. Group by check, order by severity and page count
   Readable is what makes it get acted on.
   Files: `src/report/order.py`
3. Prune old crawl detail, keep the summaries
   Rendered pages and evidence are most of the disk use here.
   Files: `src/retention.py`, `tests/test_retention.py`

### Done when

- [ ] The report opens offline with no requests
- [ ] Every finding shows evidence and a source
- [ ] Authorisation and crawl metadata lead the report
- [ ] Retention prunes detail and keeps summaries

### Watch out

- A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.

# Operations · WooRank

## Backup

The database and the reports directory. Rendered page copies are the bulk of the disk and are regenerable, so exclude them and say so in the README.

## Restore

Restore the database, run a crawl against the fixture site, and confirm the planted findings appear exactly as before. If they do not, a check definition changed and the historical diffs are no longer comparable.

Do a restore drill before the first real user, and write the date here when it passes.

## Monitoring

Disk used per crawl, the failure rate of page renders, and the time per page against the delay setting. A render failure rate that climbs usually means the browser was updated, not that the site broke.

## Incident checklist

The incident to plan for is a site owner asking why your crawler was on their server. The request log, the user agent with its contact, and the authorisation record on the site row are the whole answer, which is why all three exist before the first fetch.

1. Contain the issue without destroying evidence or user data.
2. Record the timeline and affected scope.
3. Rotate exposed secrets and revoke compromised sessions or credentials.
4. Restore from a verified backup when needed.
5. Document the root cause, the remediation and the regression test.

## Release gate

- [ ] No crawl is confirmed to start without a recorded authorisation
- [ ] Request concurrency is confirmed to be one by default with a contact in the user agent
- [ ] Every check is confirmed to carry a source URL
- [ ] No score or grade is confirmed to appear anywhere in the output
- [ ] Performance figures are confirmed to be medians of repeated runs, labelled as local

## Launch constraint

Do not market omitted WooRank capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.

# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sites, crawls, pages, findings and measurements.
DATABASE_PATH=./data/audits.db

# Required. Sent on every request. The contact address in it is not optional.
USER_AGENT=SiteAudit/1.0 (+mailto:you@example.com)

# Required. Between requests to the same origin. Lower it only with the owner's agreement.
REQUEST_DELAY_MS=1000

# Required. Ceiling per crawl unless the site record overrides it.
MAX_PAGES_DEFAULT=500

# Required. Runs per page. The report shows the median and the spread of these.
PERF_RUNS=3

# Required. Generated HTML and CSV reports. These get emailed, so keep them self-contained.
REPORT_DIR=./reports

# Required. Crawl detail is pruned after this; summary rows are kept for diffs.
CRAWL_RETENTION_DAYS=180

$ choose a build depth, inspect the files, then open the complete pack in your agent