Build Debriefing

KINDAreplaces $80/mosaves $960/yrback to the verdict

0%0 of 18 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A local competitor watch whose hard problems are telling a real change from a rotated testimonial, keeping every claim traceable to a stored snapshot, and keeping enough history that a single diff eventually reads as a trend.

estimated effort weekendthe files for this build are in the project pack

RuntimeNode 22 with TypeScript, a command line only, run from a schedulerFetchingA headless browser for pages that render client-side, one request per URL per runStorageSQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshotsExtractionA versioned recipe that turns a page into plain textBriefOne model call over the surviving diffs, every bullet required to cite a snapshot

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the fetcher, the differ and the brief generator.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
installfree
Why A pricing page that renders client-side is empty in the HTML, and that is the page you most want to watch.
Get it Install a browser automation library and let it download its browser binary.
Verify A client-rendered page yields the same text a human sees
API keypennies per brief at a handful of competitors
Why The brief is one call per window and it needs a ceiling before it needs anything else.
Get it Create a key with its own spend limit.
Verify One brief generates and the token cost is recorded
decidefree
Why Watching the wrong four pages produces a brief about nothing, and the choice is judgement rather than code.
Get it For each rival, pick the homepage, the pricing page, the changelog and the careers page, and check each one is public.
Verify Every URL is permitted by that host's robots.txt and returns content
API keyfree at one message a week
Why A brief you have to remember to read is a brief you stop reading in week three.
Get it Any transactional mail provider, or an incoming webhook on a channel you actually watch.
Verify A test brief arrives
decidefree
Why Each is asserted by a test, and tuning them after a noisy month means tuning to whatever you have already grown used to.
Get it Write down the minimum changed characters, how long raw snapshots are kept, and the maximum tokens per brief.
Verify All three are in the configuration and named in tests

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- competitors(id, name, homepage, enabled, added_at)
- watched_urls(id, competitor_id, url, kind, robots_allowed, robots_checked_at, last_status, last_fetch_at) · kind is homepage, pricing, changelog or careers
- snapshots(id, url_id, fetched_at, http_status, content_hash, extractor_version, text) · stored only when the hash changed
- diffs(id, url_id, from_snapshot_id, to_snapshot_id, changed_chars, unified_text, suppressed_reason, labelled_as) · suppressed rows are kept, not deleted
- noise_rules(id, pattern, note, added_at, hits) · an editable regular expression list with a hit count, so a useless rule is visible
- briefs(id, window_start, window_end, path, model, tokens_in, tokens_out, sent_at, state)
- brief_claims(brief_id, competitor_id, text, url_id, snapshot_id) · a claim with no snapshot id is never written
- Suppressed diffs are kept with the reason. The false positive rate is a number you can compute only if you did not throw the evidence away.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_PATH`	required	`./data/watch.db`	Snapshots, diffs, briefs and claims. The history is the product.
`USER_AGENT`	required	`competitor-watch/0.1 (contact: you@example.com)`	Identifies the fetcher on every request, as the standard expects.
`MIN_CHANGED_CHARS`	required	`40`	Diffs smaller than this are suppressed with a reason, not deleted.
`SNAPSHOT_RETENTION_DAYS`	required	`365`	Raw text only. Diffs and briefs are kept for ever.
`LLM_API_KEY`secret	required	`sk-...redacted`	One call per brief. Give it its own spend limit.
`BRIEF_TOKEN_CEILING`	required	`60000`	Checked before the call. Refusing beats a surprise invoice.
`BRIEF_DIR`	required	`~/CompetitorBriefs`	Dated Markdown files. This is the record; email is delivery.
`SMTP_URL`secret	required	`smtps://user:redacted@smtp.example:465`	Delivery only. A failure alerts and the file stays.

The build, in order

1
Fetching politely, storing only what changed
Every fetch is permitted, identified, and produces a snapshot only when the content actually differs.
1. Daily at most, and a server error means stop.
  Files src/fetch/robots.tstests/robots.test.ts
2. A pricing page that is empty in the HTML is the normal case.
  Files src/fetch/browser.ts
3. An unchanged page should cost one request and nothing else.
  Files src/snapshot.ts
done when · tick each as it passes
watch out
- Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for.
next phase
2
Extraction stable enough to diff
A change to your own extraction never looks like a change on the competitor's site.
1. The version travels with every snapshot it produced.
  Files src/extract/recipe.tssrc/extract/version.ts
2. So an extractor change is a failing test rather than a surprise brief.
  Files tests/fixtures/tests/extract.test.ts
3. Your refactor is not their announcement.
  Files src/diff.ts
done when · tick each as it passes
watch out
- Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news.
next phase
3
Noise, measured rather than felt
The false positive rate is a number you can look at, and it falls.
1. The evidence is the only way to tune anything.
  Files src/noise.tssrc/diff.ts
2. A rule that never fires is a rule that might be hiding something.
  Files src/noise.ts
3. Month one is tuning, so make tuning measurable.
  Files src/label.tssrc/report/falsepositive.ts
done when · tick each as it passes
watch out
- Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete.
next phase
4
A brief where every claim has a receipt
No sentence reaches the file unless it names a URL and a snapshot you stored.
1. The model cannot cite what you did not give it.
  Files src/brief/prompt.ts
2. The count is the health metric for this phase.
  Files src/brief/validate.tstests/citations.test.ts
3. The Markdown file is the record; email can fail.
  Files src/brief/write.ts
done when · tick each as it passes
watch out
- Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive.
5
History is the only thing you ownproduct builder
A single change reads as a trend, and the archive survives a disk decision.
1. So pruning snapshots never costs you the trend.
  Files src/history.tstests/history-after-prune.test.ts
2. And say so in the README, because it is a real trade-off.
  Files src/retention.ts
3. The file is the brief. Email is a convenience.
  Files src/deliver.ts
done when · tick each as it passes
watch out
- Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate.

what this build does not replace

Anything behind a login or a paywall.
Hiring, funding and traffic data.
Screenshots as the stored artefact.
A dashboard.
noise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves
horizon watch, meaning the substitutes and new entrants you did not think to add to the list
the non-page signals stitched into the same brief: hiring, funding, traffic and LinkedIn movement
the analytical step from what changed to why it matters to what to do next
an archive going back far enough that a change reads as a trend rather than an event

after v1, if you want it

Sitemap watching to catch pages you did not think to add, still subject to the same rules
A monthly rollup built from the same diffs, which needs no new fetching
Per-competitor thresholds, once the labelled sample shows which ones are noisy

Need the files? The project pack on the verdict page hands your agent the whole brief · more rss & research.