Build WooRank

KINDAreplaces $89.99/mosaves $1,079.88/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A self-hosted site auditor whose hard problems are crawling somebody's site politely and provably with permission, reporting findings with evidence instead of an invented score, and measuring performance in a way that does not pretend a single local run is what search engines actually see.

estimated effort multi-daythe files for this build are in the project pack

RuntimeA Python command line tool with a headless browser for rendering, output to filesCrawlOne request in flight by default, a delay between requests, a page ceiling per runChecksDeclarative check definitions, each carrying its own citation to published guidanceStorageSQLite, one row per finding, with the evidence stored beside itOutputHTML and CSV reports generated from the findings table

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the crawler, the checks and the report generation.
Get it python.org or your package manager. open ↗
Verify python3 --version prints 3.12 or higher
installfree, plus a few hundred megabytes of disk
Why Half the checks and all the performance work need a rendered page, since a large share of sites assemble their content in the browser.
Get it Install the library and run its browser download step.
Verify A scripted page load returns rendered content and a performance entry
decidefree
Why This is rule zero and it is also the difference between an audit and an unsolicited crawl of somebody's server.
Get it For your own sites, a note to yourself. For a client, an email saying which origin, at what rate, and until when.
Verify Every site row has an authorised_by value and a date before its first crawl
have readyfree
Why The metric set has been changed before, with one metric replaced outright in 2024, so copying numbers from memory dates your tool immediately.
Get it Read the current published guidance, record each metric, its threshold and the date you checked. open ↗
Verify docs/thresholds.md lists each metric with its value and date, and the checks reference it
have readyfree
Why Every check, the diff and the redirect matching need a site whose answers you already know, and testing against a real client site is how you discover your crawler is impolite.
Get it Publish a handful of static pages locally with missing alternative text, a duplicate title and a redirect.
Verify A crawl of the fixture produces exactly the findings you planted

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- sites(id, origin, authorised_by, authorised_at, contact_email, max_pages, delay_ms)
- crawls(id, site_id, started_at, finished_at, pages_fetched, pages_skipped, stopped_reason)
- pages(id, crawl_id, url, status, content_type, bytes, fetched_at, render_ms, redirect_chain_json)
- findings(id, crawl_id, page_id, check_id, severity, evidence_json, source_url) · source_url is the published guidance, and a check without one does not ship
- measurements(id, page_id, metric, value, run_ordinal, method) · method is always lab, because that is what a local run is
- diffs(from_crawl, to_crawl, check_id, page_url, change) · appeared, resolved or unchanged
- There is no score column. There is no grade column. The report's headline is a count of findings by severity, and that is deliberate.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_PATH`	required	`./data/audits.db`	Sites, crawls, pages, findings and measurements.
`USER_AGENT`	required	`SiteAudit/1.0 (+mailto:you@example.com)`	Sent on every request. The contact address in it is not optional.
`REQUEST_DELAY_MS`	required	`1000`	Between requests to the same origin. Lower it only with the owner's agreement.
`MAX_PAGES_DEFAULT`	required	`500`	Ceiling per crawl unless the site record overrides it.
`PERF_RUNS`	required	`3`	Runs per page. The report shows the median and the spread of these.
`REPORT_DIR`	required	`./reports`	Generated HTML and CSV reports. These get emailed, so keep them self-contained.
`CRAWL_RETENTION_DAYS`	required	`180`	Crawl detail is pruned after this; summary rows are kept for diffs.

The build, in order

1
A crawl you are allowed to run
Auditing a site requires recorded authorisation, and the crawler behaves like a guest.
1. If you cannot name who agreed, do not fetch the first page.
  Files src/sites.pytests/test_authorisation.py
2. Politeness is a default, not a flag.
  Files src/crawl/fetcher.py
3. This log is your answer when somebody asks what you did to their server.
  Files src/crawl/log.py
done when · tick each as it passes
watch out
- Running a fast parallel crawler against a small shared host takes the site down, and the first person to notice is the owner who did not know you were auditing. Default to one request in flight with a delay, put the ceiling in the site record rather than in a flag somebody forgets, and make going faster an explicit decision recorded against the site.
next phase
2
Findings, each with evidence and a source
Every check produces rows containing the page, what was found, and a link to the guidance it came from.
1. A table of checks can be read, argued with and extended.
  Files src/checks/definitions.py
2. If it is not in published guidance, it is your preference.
  Files src/checks/loader.pytests/test_sources.py
3. Nobody fixes a finding they cannot see.
  Files src/checks/run.py
done when · tick each as it passes
watch out
- Grading a page's title by character count reports failures on pages that are completely fine, because the documented behaviour is that there is no length limit on the title element and the search engine may rewrite the title link it displays anyway, from the page's own text or from links to it. Report what the title is, flag the ones that are empty, duplicated or missing, and leave length as information rather than a fault.
next phase
3
Performance measured honestly
Page performance is measured several times, reported against the published thresholds, and labelled as a local measurement.
1. A single run on a laptop is a sample of one.
  Files src/perf/measure.pytests/test_median.py
2. The names and the numbers have changed before and will again.
  Files src/checks/definitions.pydocs/thresholds.md
3. The published assessment uses real visits at a percentile, not your machine.
  Files src/report/render.py
done when · tick each as it passes
watch out
- Reporting one local run as the page's performance produces numbers that move by a large fraction between runs on the same machine and bear no fixed relationship to what visitors experience, since the published guidance assesses these metrics at the seventy fifth percentile of real page loads split by device type. Run three times, report the median and the spread, and say on the page that this is a laboratory measurement and not field data.
next phase
4
What changed since last time
Two crawls of the same site can be compared, and the comparison is about findings appearing and being resolved.
1. A comparison against something that changed underneath is not a comparison.
  Files src/crawl/snapshot.py
2. Otherwise a URL change looks like the site fell over.
  Files src/diff/match.pytests/test_redirect_match.py
3. The whole point of having no score is not to reintroduce one here.
  Files src/diff/report.py
done when · tick each as it passes
watch out
- Diffing by page URL alone reports a site-wide catastrophe the day somebody changes a URL structure, drowning the real regressions in a few hundred false ones. Follow the redirect chain recorded during the crawl and match the old URL to its destination before deciding a finding disappeared.
5
A report you can hand to somebodyproduct builder
The deliverable is a self-contained document with evidence and sources, and the storage it consumes has a ceiling.
1. It will be emailed, and it should still work.
  Files src/report/html.py
2. Readable is what makes it get acted on.
  Files src/report/order.py
3. Rendered pages and evidence are most of the disk use here.
  Files src/retention.pytests/test_retention.py
done when · tick each as it passes
watch out
- A report that lists every finding on every page in crawl order is unreadable at three hundred pages, so the recipient reads the first screen and stops, and the thing that actually mattered was on page eleven. Group by check, order by severity and affected page count, and put the full page list behind the group rather than in front of it.

what this build does not replace

Keyword ranking data.
Backlink data.
Competitor benchmarking crawls.
A public site grading widget.
historical competitor datasets and support
fresh proprietary keyword, backlink, or ranking data
proprietary web and keyword indexes
fresh rank and clickstream data
large-scale crawling infrastructure

after v1, if you want it

Search console data for sites you own, which is the only field data available to you without renting an index
A check that compares two renders of the same page to catch content that only appears with a browser
An accessibility check set, kept separate from the search findings because the audience and the urgency are different

Need the files? The project pack on the verdict page hands your agent the whole brief · more seo & marketing.