Vibecode Debriefing
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
Diffing a competitor's pricing page on a cron and asking a model what changed is genuinely an afternoon. changedetection.io will do the watching for you before you write a line. The gap is everything between a diff and a brief. Most page changes are noise: a rotated testimonial, a reordered nav, a CDN hash. Deciding which changes are real, tying them to hiring and funding signals, and turning that into three sentences a founder acts on is judgement encoded over many iterations. Build it and your first month is mostly you deleting alerts about nothing.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Debriefing. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Debriefing · indie build A local competitor watch whose hard problems are telling a real change from a rotated testimonial, keeping every claim traceable to a stored snapshot, and keeping enough history that a single diff eventually reads as a trend. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a command line only, run from a scheduler | there is no interface worth building before the noise is under control | | Fetching | A headless browser for pages that render client-side, one request per URL per run | half of a modern pricing page does not exist in the HTML | | Storage | SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots | text diffs, compresses and searches; a screenshot does none of those | | Extraction | A versioned recipe that turns a page into plain text | a change to your own extractor must not look like five competitors relaunching at once | | Brief | One model call over the surviving diffs, every bullet required to cite a snapshot | an uncited sentence is a guess with good grammar | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the fetcher, the differ and the brief generator. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A headless browser with its runtime dependencies** · free - Why: A pricing page that renders client-side is empty in the HTML, and that is the page you most want to watch. - Get it: Install a browser automation library and let it download its browser binary. - Verify: A client-rendered page yields the same text a human sees - [ ] **An API key for a language model** · pennies per brief at a handful of competitors - Why: The brief is one call per window and it needs a ceiling before it needs anything else. - Get it: Create a key with its own spend limit. - Verify: One brief generates and the token cost is recorded - [ ] **Your competitor list and the URLs that matter** · free - Why: Watching the wrong four pages produces a brief about nothing, and the choice is judgement rather than code. - Get it: For each rival, pick the homepage, the pricing page, the changelog and the careers page, and check each one is public. - Verify: Every URL is permitted by that host's robots.txt and returns content - [ ] **SMTP credentials or a webhook you control** · free at one message a week - Why: A brief you have to remember to read is a brief you stop reading in week three. - Get it: Any transactional mail provider, or an incoming webhook on a channel you actually watch. - Verify: A test brief arrives - [ ] **Your change threshold, retention and cost ceiling** · free - Why: Each is asserted by a test, and tuning them after a noisy month means tuning to whatever you have already grown used to. - Get it: Write down the minimum changed characters, how long raw snapshots are kept, and the maximum tokens per brief. - Verify: All three are in the configuration and named in tests ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Anything behind a login or a paywall. - Hiring, funding and traffic data. - Screenshots as the stored artefact. - A dashboard. - noise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves - horizon watch, meaning the substitutes and new entrants you did not think to add to the list - the non-page signals stitched into the same brief: hiring, funding, traffic and LinkedIn movement - the analytical step from what changed to why it matters to what to do next - an archive going back far enough that a change reads as a trend rather than an event If one of those is essential to you, that is the reason to keep paying for Debriefing, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Debriefing The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a competitor watch that files a weekly brief and drops any claim it cannot cite to replace Debriefing. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules. Public pages only, fetched the way a well-behaved crawler fetches them. Read each site's robots.txt and honour it. The standard is specific: a robots.txt answered with a server error means assume complete disallow, a cached copy should not be used for more than 24 hours, the most specific matching rule wins and allow beats disallow on an equal-length match, and a parser must handle at least 500 kibibytes (checked September 2026). Identify yourself in the User-Agent, fetch each URL at most once per run, and never touch anything behind a login. Second, every sentence in the brief cites a URL and a stored snapshot, or it does not appear. A brief that quietly invents a price change is worse than no brief, because you will act on it and the meeting where that unravels is expensive. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a command line only, run from a scheduler · there is no interface worth building before the noise is under control - Fetching: A headless browser for pages that render client-side, one request per URL per run · half of a modern pricing page does not exist in the HTML - Storage: SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots · text diffs, compresses and searches; a screenshot does none of those - Extraction: A versioned recipe that turns a page into plain text · a change to your own extractor must not look like five competitors relaunching at once - Brief: One model call over the surviving diffs, every bullet required to cite a snapshot · an uncited sentence is a guess with good grammar ### Data model (create this before Phase 1) - competitors(id, name, homepage, enabled, added_at) - watched_urls(id, competitor_id, url, kind, robots_allowed, robots_checked_at, last_status, last_fetch_at) · kind is homepage, pricing, changelog or careers - snapshots(id, url_id, fetched_at, http_status, content_hash, extractor_version, text) · stored only when the hash changed - diffs(id, url_id, from_snapshot_id, to_snapshot_id, changed_chars, unified_text, suppressed_reason, labelled_as) · suppressed rows are kept, not deleted - noise_rules(id, pattern, note, added_at, hits) · an editable regular expression list with a hit count, so a useless rule is visible - briefs(id, window_start, window_end, path, model, tokens_in, tokens_out, sent_at, state) - brief_claims(brief_id, competitor_id, text, url_id, snapshot_id) · a claim with no snapshot id is never written - Suppressed diffs are kept with the reason. The false positive rate is a number you can compute only if you did not throw the evidence away. ### Phase 1 · Fetching politely, storing only what changed Build: Fetch and cache robots.txt per host, re-reading it at least daily, and treat a server error or an unreachable file as a refusal rather than a licence. Render client-side pages in a headless browser. Hash the extracted text and store a snapshot only when the hash changed. Seed the configuration with three competitors and four URLs each. Trap: Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. Done when: A disallowed path is never fetched, tested against a local server serving a robots.txt that forbids it. A robots.txt answered with a server error stops fetching for that host, tested. A page whose content has not changed adds no snapshot row, verified by fetching twice. Every request carries a User-Agent naming the tool and a contact, and each URL is requested at most once per run, asserted by counting. Do not build yet: No diffing, noise filtering, briefs or delivery. ### Phase 2 · Extraction stable enough to diff Build: Turn a page into plain text with a recipe that strips scripts, styles, navigation and footers and keeps the main content. Version the recipe and store the version with every snapshot. A diff spanning two extractor versions is suppressed and the page is re-baselined instead. Trap: Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. Done when: Two snapshots taken with different extractor versions produce a suppressed diff with the reason recorded, tested. Changing a class name or an asset hash on a fixture page produces no diff. The extracted text for a fixture page matches a stored expectation, so extractor changes are deliberate. Every snapshot records the extractor version that produced it. Do not build yet: No noise rules, briefs or delivery yet. ### Phase 3 · Noise, measured rather than felt Build: Drop diffs under a stated changed-character threshold, and drop lines matching an editable list of patterns for dates, cache-busting hashes, view counts and rotating testimonials. Keep suppressed diffs with their reason. Label each surfaced diff as signal or noise for a fortnight and compute the rate per competitor and per URL kind. Trap: Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. Done when: Suppressed diffs are stored with their reason and are visible in a command, not deleted. Each noise rule carries a hit count and a rule with no hits is flagged. A labelled fortnight produces a false positive rate per competitor and per URL kind, printed by a command. Raising the changed-character threshold visibly moves that rate, demonstrated with a stored sample. Do not build yet: No brief generation or delivery yet. ### Phase 4 · A brief where every claim has a receipt Build: Send the surviving diffs for the window in one call, with each diff labelled by competitor, URL and snapshot id. Require every output bullet to carry a snapshot id from that set. Drop any bullet whose id is missing or unknown, and record how many were dropped. Write the brief to a dated Markdown file; that file is the record and email is only delivery. Trap: Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. Done when: A bullet without a snapshot id from the request set is dropped before the file is written, tested by feeding a fabricated response. Every claim in the file links to a URL and a snapshot that exists in the database, asserted over a generated brief. The number of dropped bullets is printed and stored with the brief. The cost of each brief in tokens is recorded and a per-brief ceiling refuses rather than exceeding. Do not build yet: No history view, retention or delivery failure handling yet. ### Phase 5 · History is the only thing you own Build: A history command per competitor printing every change over time with dates and links. Compress stored text and prune raw snapshots past a stated age while keeping every diff and every brief forever. Deliver the brief by mail or webhook with retries, and treat a delivery failure as an alert rather than a lost brief. Trap: Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate. Done when: The history command prints a competitor's changes over time with dates and source links, from stored diffs alone. Pruning removes only raw snapshot text past the stated age and leaves every diff and brief intact, tested by pruning and re-running history. A delivery failure leaves the brief file written and raises an alert, tested by breaking the mail configuration. Database growth per week is printed so the retention decision is informed. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, a paywall or a trial account, which is where the terms you agreed to start applying and where this stops being a public-page watcher. - Hiring feeds, funding data and traffic estimates, which are the analyst layer the paid product sells and each one a separate data licence. - Screenshots as the stored artefact. They cannot be diffed usefully, they cost far more disk, and text is what the brief is built from. - A dashboard. Until the false positive rate is low, an interface just makes it easier to ignore more alerts faster. ### README must contain - That only public pages are fetched, robots.txt is honoured, and a robots.txt server error stops that host entirely - That the extractor is versioned, and a diff across a version boundary is suppressed rather than reported as news - That suppressed diffs are kept with their reason, because the false positive rate is a number you compute from them - That every brief claim cites a stored snapshot and uncited bullets are dropped and counted - The retention policy, the weekly disk growth, and the per-brief token cost, plus a warning that month one is mostly tuning the noise rules ===== AGENTS.md ===== # Agent instructions · Debriefing indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript, a command line only, run from a scheduler, A headless browser for pages that render client-side, one request per URL per run, SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots, A versioned recipe that turns a page into plain text, One model call over the surviving diffs, every bullet required to cite a snapshot. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. ===== BUILD_PLAN.md ===== # Build plan · Debriefing A local competitor watch whose hard problems are telling a real change from a rotated testimonial, keeping every claim traceable to a stored snapshot, and keeping enough history that a single diff eventually reads as a trend. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Fetching politely, storing only what changed Every fetch is permitted, identified, and produces a snapshot only when the content actually differs. ### Steps 1. Cache robots.txt per host with an expiry Daily at most, and a server error means stop. Files: `src/fetch/robots.ts`, `tests/robots.test.ts` 2. Render client-side pages properly A pricing page that is empty in the HTML is the normal case. Files: `src/fetch/browser.ts` 3. Hash first, store second An unchanged page should cost one request and nothing else. Files: `src/snapshot.ts` ### Done when - [ ] A disallowed path is never fetched - [ ] A robots.txt server error stops that host - [ ] An unchanged page stores no snapshot - [ ] One request per URL per run, with a named User-Agent ### Watch out - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. ## Phase 2 · Extraction stable enough to diff A change to your own extraction never looks like a change on the competitor's site. ### Steps 1. Write the extraction recipe once, and version it The version travels with every snapshot it produced. Files: `src/extract/recipe.ts`, `src/extract/version.ts` 2. Pin fixture pages with expected text So an extractor change is a failing test rather than a surprise brief. Files: `tests/fixtures/`, `tests/extract.test.ts` 3. Suppress and re-baseline across a version change Your refactor is not their announcement. Files: `src/diff.ts` ### Done when - [ ] A cross-version diff is suppressed with a reason - [ ] Markup churn produces no diff - [ ] Fixture extraction matches stored expectations - [ ] Every snapshot records its extractor version ### Watch out - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. ## Phase 3 · Noise, measured rather than felt The false positive rate is a number you can look at, and it falls. ### Steps 1. Keep suppressed diffs with reasons The evidence is the only way to tune anything. Files: `src/noise.ts`, `src/diff.ts` 2. Count hits per noise rule A rule that never fires is a rule that might be hiding something. Files: `src/noise.ts` 3. Label a fortnight and compute the rate Month one is tuning, so make tuning measurable. Files: `src/label.ts`, `src/report/falsepositive.ts` ### Done when - [ ] Suppressed diffs are kept with reasons - [ ] Each noise rule shows a hit count - [ ] A labelled sample yields a false positive rate - [ ] Changing the threshold moves the rate visibly ### Watch out - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. ## Phase 4 · A brief where every claim has a receipt No sentence reaches the file unless it names a URL and a snapshot you stored. ### Steps 1. Label every diff with a snapshot id in the prompt The model cannot cite what you did not give it. Files: `src/brief/prompt.ts` 2. Drop uncited bullets and count them The count is the health metric for this phase. Files: `src/brief/validate.ts`, `tests/citations.test.ts` 3. Write the file first, deliver second The Markdown file is the record; email can fail. Files: `src/brief/write.ts` ### Done when - [ ] Uncited bullets never reach the file - [ ] Every claim resolves to a stored snapshot - [ ] The drop count is printed and stored - [ ] A per-brief cost ceiling refuses rather than overspends ### Watch out - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. ## Not in this build - Anything behind a login or a paywall. - Hiring, funding and traffic data. - Screenshots as the stored artefact. - A dashboard. ## After v1, if you want it - Sitemap watching to catch pages you did not think to add, still subject to the same rules - A monthly rollup built from the same diffs, which needs no new fetching - Per-competitor thresholds, once the labelled sample shows which ones are noisy ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Snapshots, diffs, briefs and claims. The history is the product. DATABASE_PATH=./data/watch.db # Required. Identifies the fetcher on every request, as the standard expects. USER_AGENT=competitor-watch/0.1 (contact: you@example.com) # Required. Diffs smaller than this are suppressed with a reason, not deleted. MIN_CHANGED_CHARS=40 # Required. Raw text only. Diffs and briefs are kept for ever. SNAPSHOT_RETENTION_DAYS=365 # Required · secret. One call per brief. Give it its own spend limit. LLM_API_KEY=sk-...redacted # Required. Checked before the call. Refusing beats a surprise invoice. BRIEF_TOKEN_CEILING=60000 # Required. Dated Markdown files. This is the record; email is delivery. BRIEF_DIR=~/CompetitorBriefs # Required · secret. Delivery only. A failure alerts and the file stays. SMTP_URL=smtps://user:redacted@smtp.example:465
You are building a lean indie version of Debriefing. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Debriefing · indie build A local competitor watch whose hard problems are telling a real change from a rotated testimonial, keeping every claim traceable to a stored snapshot, and keeping enough history that a single diff eventually reads as a trend. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a command line only, run from a scheduler | there is no interface worth building before the noise is under control | | Fetching | A headless browser for pages that render client-side, one request per URL per run | half of a modern pricing page does not exist in the HTML | | Storage | SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots | text diffs, compresses and searches; a screenshot does none of those | | Extraction | A versioned recipe that turns a page into plain text | a change to your own extractor must not look like five competitors relaunching at once | | Brief | One model call over the surviving diffs, every bullet required to cite a snapshot | an uncited sentence is a guess with good grammar | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the fetcher, the differ and the brief generator. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A headless browser with its runtime dependencies** · free - Why: A pricing page that renders client-side is empty in the HTML, and that is the page you most want to watch. - Get it: Install a browser automation library and let it download its browser binary. - Verify: A client-rendered page yields the same text a human sees - [ ] **An API key for a language model** · pennies per brief at a handful of competitors - Why: The brief is one call per window and it needs a ceiling before it needs anything else. - Get it: Create a key with its own spend limit. - Verify: One brief generates and the token cost is recorded - [ ] **Your competitor list and the URLs that matter** · free - Why: Watching the wrong four pages produces a brief about nothing, and the choice is judgement rather than code. - Get it: For each rival, pick the homepage, the pricing page, the changelog and the careers page, and check each one is public. - Verify: Every URL is permitted by that host's robots.txt and returns content - [ ] **SMTP credentials or a webhook you control** · free at one message a week - Why: A brief you have to remember to read is a brief you stop reading in week three. - Get it: Any transactional mail provider, or an incoming webhook on a channel you actually watch. - Verify: A test brief arrives - [ ] **Your change threshold, retention and cost ceiling** · free - Why: Each is asserted by a test, and tuning them after a noisy month means tuning to whatever you have already grown used to. - Get it: Write down the minimum changed characters, how long raw snapshots are kept, and the maximum tokens per brief. - Verify: All three are in the configuration and named in tests ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Anything behind a login or a paywall. - Hiring, funding and traffic data. - Screenshots as the stored artefact. - A dashboard. - noise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves - horizon watch, meaning the substitutes and new entrants you did not think to add to the list - the non-page signals stitched into the same brief: hiring, funding, traffic and LinkedIn movement - the analytical step from what changed to why it matters to what to do next - an archive going back far enough that a change reads as a trend rather than an event If one of those is essential to you, that is the reason to keep paying for Debriefing, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Debriefing The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a competitor watch that files a weekly brief and drops any claim it cannot cite to replace Debriefing. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules. Public pages only, fetched the way a well-behaved crawler fetches them. Read each site's robots.txt and honour it. The standard is specific: a robots.txt answered with a server error means assume complete disallow, a cached copy should not be used for more than 24 hours, the most specific matching rule wins and allow beats disallow on an equal-length match, and a parser must handle at least 500 kibibytes (checked September 2026). Identify yourself in the User-Agent, fetch each URL at most once per run, and never touch anything behind a login. Second, every sentence in the brief cites a URL and a stored snapshot, or it does not appear. A brief that quietly invents a price change is worse than no brief, because you will act on it and the meeting where that unravels is expensive. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a command line only, run from a scheduler · there is no interface worth building before the noise is under control - Fetching: A headless browser for pages that render client-side, one request per URL per run · half of a modern pricing page does not exist in the HTML - Storage: SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots · text diffs, compresses and searches; a screenshot does none of those - Extraction: A versioned recipe that turns a page into plain text · a change to your own extractor must not look like five competitors relaunching at once - Brief: One model call over the surviving diffs, every bullet required to cite a snapshot · an uncited sentence is a guess with good grammar ### Data model (create this before Phase 1) - competitors(id, name, homepage, enabled, added_at) - watched_urls(id, competitor_id, url, kind, robots_allowed, robots_checked_at, last_status, last_fetch_at) · kind is homepage, pricing, changelog or careers - snapshots(id, url_id, fetched_at, http_status, content_hash, extractor_version, text) · stored only when the hash changed - diffs(id, url_id, from_snapshot_id, to_snapshot_id, changed_chars, unified_text, suppressed_reason, labelled_as) · suppressed rows are kept, not deleted - noise_rules(id, pattern, note, added_at, hits) · an editable regular expression list with a hit count, so a useless rule is visible - briefs(id, window_start, window_end, path, model, tokens_in, tokens_out, sent_at, state) - brief_claims(brief_id, competitor_id, text, url_id, snapshot_id) · a claim with no snapshot id is never written - Suppressed diffs are kept with the reason. The false positive rate is a number you can compute only if you did not throw the evidence away. ### Phase 1 · Fetching politely, storing only what changed Build: Fetch and cache robots.txt per host, re-reading it at least daily, and treat a server error or an unreachable file as a refusal rather than a licence. Render client-side pages in a headless browser. Hash the extracted text and store a snapshot only when the hash changed. Seed the configuration with three competitors and four URLs each. Trap: Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. Done when: A disallowed path is never fetched, tested against a local server serving a robots.txt that forbids it. A robots.txt answered with a server error stops fetching for that host, tested. A page whose content has not changed adds no snapshot row, verified by fetching twice. Every request carries a User-Agent naming the tool and a contact, and each URL is requested at most once per run, asserted by counting. Do not build yet: No diffing, noise filtering, briefs or delivery. ### Phase 2 · Extraction stable enough to diff Build: Turn a page into plain text with a recipe that strips scripts, styles, navigation and footers and keeps the main content. Version the recipe and store the version with every snapshot. A diff spanning two extractor versions is suppressed and the page is re-baselined instead. Trap: Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. Done when: Two snapshots taken with different extractor versions produce a suppressed diff with the reason recorded, tested. Changing a class name or an asset hash on a fixture page produces no diff. The extracted text for a fixture page matches a stored expectation, so extractor changes are deliberate. Every snapshot records the extractor version that produced it. Do not build yet: No noise rules, briefs or delivery yet. ### Phase 3 · Noise, measured rather than felt Build: Drop diffs under a stated changed-character threshold, and drop lines matching an editable list of patterns for dates, cache-busting hashes, view counts and rotating testimonials. Keep suppressed diffs with their reason. Label each surfaced diff as signal or noise for a fortnight and compute the rate per competitor and per URL kind. Trap: Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. Done when: Suppressed diffs are stored with their reason and are visible in a command, not deleted. Each noise rule carries a hit count and a rule with no hits is flagged. A labelled fortnight produces a false positive rate per competitor and per URL kind, printed by a command. Raising the changed-character threshold visibly moves that rate, demonstrated with a stored sample. Do not build yet: No brief generation or delivery yet. ### Phase 4 · A brief where every claim has a receipt Build: Send the surviving diffs for the window in one call, with each diff labelled by competitor, URL and snapshot id. Require every output bullet to carry a snapshot id from that set. Drop any bullet whose id is missing or unknown, and record how many were dropped. Write the brief to a dated Markdown file; that file is the record and email is only delivery. Trap: Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. Done when: A bullet without a snapshot id from the request set is dropped before the file is written, tested by feeding a fabricated response. Every claim in the file links to a URL and a snapshot that exists in the database, asserted over a generated brief. The number of dropped bullets is printed and stored with the brief. The cost of each brief in tokens is recorded and a per-brief ceiling refuses rather than exceeding. Do not build yet: No history view, retention or delivery failure handling yet. ### Phase 5 · History is the only thing you own Build: A history command per competitor printing every change over time with dates and links. Compress stored text and prune raw snapshots past a stated age while keeping every diff and every brief forever. Deliver the brief by mail or webhook with retries, and treat a delivery failure as an alert rather than a lost brief. Trap: Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate. Done when: The history command prints a competitor's changes over time with dates and source links, from stored diffs alone. Pruning removes only raw snapshot text past the stated age and leaves every diff and brief intact, tested by pruning and re-running history. A delivery failure leaves the brief file written and raises an alert, tested by breaking the mail configuration. Database growth per week is printed so the retention decision is informed. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, a paywall or a trial account, which is where the terms you agreed to start applying and where this stops being a public-page watcher. - Hiring feeds, funding data and traffic estimates, which are the analyst layer the paid product sells and each one a separate data licence. - Screenshots as the stored artefact. They cannot be diffed usefully, they cost far more disk, and text is what the brief is built from. - A dashboard. Until the false positive rate is low, an interface just makes it easier to ignore more alerts faster. ### README must contain - That only public pages are fetched, robots.txt is honoured, and a robots.txt server error stops that host entirely - That the extractor is versioned, and a diff across a version boundary is suppressed rather than reported as news - That suppressed diffs are kept with their reason, because the false positive rate is a number you compute from them - That every brief claim cites a stored snapshot and uncited bullets are dropped and counted - The retention policy, the weekly disk growth, and the per-brief token cost, plus a warning that month one is mostly tuning the noise rules ===== AGENTS.md ===== # Agent instructions · Debriefing indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript, a command line only, run from a scheduler, A headless browser for pages that render client-side, one request per URL per run, SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots, A versioned recipe that turns a page into plain text, One model call over the surviving diffs, every bullet required to cite a snapshot. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. ===== BUILD_PLAN.md ===== # Build plan · Debriefing A local competitor watch whose hard problems are telling a real change from a rotated testimonial, keeping every claim traceable to a stored snapshot, and keeping enough history that a single diff eventually reads as a trend. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Fetching politely, storing only what changed Every fetch is permitted, identified, and produces a snapshot only when the content actually differs. ### Steps 1. Cache robots.txt per host with an expiry Daily at most, and a server error means stop. Files: `src/fetch/robots.ts`, `tests/robots.test.ts` 2. Render client-side pages properly A pricing page that is empty in the HTML is the normal case. Files: `src/fetch/browser.ts` 3. Hash first, store second An unchanged page should cost one request and nothing else. Files: `src/snapshot.ts` ### Done when - [ ] A disallowed path is never fetched - [ ] A robots.txt server error stops that host - [ ] An unchanged page stores no snapshot - [ ] One request per URL per run, with a named User-Agent ### Watch out - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. ## Phase 2 · Extraction stable enough to diff A change to your own extraction never looks like a change on the competitor's site. ### Steps 1. Write the extraction recipe once, and version it The version travels with every snapshot it produced. Files: `src/extract/recipe.ts`, `src/extract/version.ts` 2. Pin fixture pages with expected text So an extractor change is a failing test rather than a surprise brief. Files: `tests/fixtures/`, `tests/extract.test.ts` 3. Suppress and re-baseline across a version change Your refactor is not their announcement. Files: `src/diff.ts` ### Done when - [ ] A cross-version diff is suppressed with a reason - [ ] Markup churn produces no diff - [ ] Fixture extraction matches stored expectations - [ ] Every snapshot records its extractor version ### Watch out - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. ## Phase 3 · Noise, measured rather than felt The false positive rate is a number you can look at, and it falls. ### Steps 1. Keep suppressed diffs with reasons The evidence is the only way to tune anything. Files: `src/noise.ts`, `src/diff.ts` 2. Count hits per noise rule A rule that never fires is a rule that might be hiding something. Files: `src/noise.ts` 3. Label a fortnight and compute the rate Month one is tuning, so make tuning measurable. Files: `src/label.ts`, `src/report/falsepositive.ts` ### Done when - [ ] Suppressed diffs are kept with reasons - [ ] Each noise rule shows a hit count - [ ] A labelled sample yields a false positive rate - [ ] Changing the threshold moves the rate visibly ### Watch out - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. ## Phase 4 · A brief where every claim has a receipt No sentence reaches the file unless it names a URL and a snapshot you stored. ### Steps 1. Label every diff with a snapshot id in the prompt The model cannot cite what you did not give it. Files: `src/brief/prompt.ts` 2. Drop uncited bullets and count them The count is the health metric for this phase. Files: `src/brief/validate.ts`, `tests/citations.test.ts` 3. Write the file first, deliver second The Markdown file is the record; email can fail. Files: `src/brief/write.ts` ### Done when - [ ] Uncited bullets never reach the file - [ ] Every claim resolves to a stored snapshot - [ ] The drop count is printed and stored - [ ] A per-brief cost ceiling refuses rather than overspends ### Watch out - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. ## Not in this build - Anything behind a login or a paywall. - Hiring, funding and traffic data. - Screenshots as the stored artefact. - A dashboard. ## After v1, if you want it - Sitemap watching to catch pages you did not think to add, still subject to the same rules - A monthly rollup built from the same diffs, which needs no new fetching - Per-competitor thresholds, once the labelled sample shows which ones are noisy ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Snapshots, diffs, briefs and claims. The history is the product. DATABASE_PATH=./data/watch.db # Required. Identifies the fetcher on every request, as the standard expects. USER_AGENT=competitor-watch/0.1 (contact: you@example.com) # Required. Diffs smaller than this are suppressed with a reason, not deleted. MIN_CHANGED_CHARS=40 # Required. Raw text only. Diffs and briefs are kept for ever. SNAPSHOT_RETENTION_DAYS=365 # Required · secret. One call per brief. Give it its own spend limit. LLM_API_KEY=sk-...redacted # Required. Checked before the call. Refusing beats a surprise invoice. BRIEF_TOKEN_CEILING=60000 # Required. Dated Markdown files. This is the record; email is delivery. BRIEF_DIR=~/CompetitorBriefs # Required · secret. Delivery only. A failure alerts and the file stays. SMTP_URL=smtps://user:redacted@smtp.example:465
You are building a production product version of Debriefing. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Debriefing · product brief ## Problem Diffing a competitor's pricing page on a cron and asking a model what changed is genuinely an afternoon. changedetection.io will do the watching for you before you write a line. The gap is everything between a diff and a brief. Most page changes are noise: a rotated testimonial, a reordered nav, a CDN hash. Deciding which changes are real, tying them to hiring and funding signals, and turning that into three sentences a founder acts on is judgement encoded over many iterations. Build it and your first month is mostly you deleting alerts about nothing. ## Product outcome A weekly brief whose every claim links to a page snapshot you stored, whose noise rate you can measure and reduce, and whose accumulated history lets a single change be read as part of a pattern. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - an OpenAI or Anthropic API key - a scheduler and somewhere to keep page snapshots - a headless browser for pages that render client-side - SMTP or a Slack webhook for delivery - patience for the first month of false positives ## Explicit non-goals for v1 - Anything behind a login or a paywall. - Hiring, funding and traffic data. - Screenshots as the stored artefact. - A dashboard. - noise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves - horizon watch, meaning the substitutes and new entrants you did not think to add to the list - the non-page signals stitched into the same brief: hiring, funding, traffic and LinkedIn movement - the analytical step from what changed to why it matters to what to do next - an archive going back far enough that a change reads as a trend rather than an event ## Success criteria - A disallowed path is confirmed never fetched - A robots.txt server error is confirmed to stop that host - A cross-version diff is confirmed suppressed - Uncited bullets are confirmed dropped and counted - History is confirmed to survive a snapshot prune ===== BRIEF.md ===== # Build brief · Debriefing The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a competitor watch that files a weekly brief and drops any claim it cannot cite to replace Debriefing. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules. Public pages only, fetched the way a well-behaved crawler fetches them. Read each site's robots.txt and honour it. The standard is specific: a robots.txt answered with a server error means assume complete disallow, a cached copy should not be used for more than 24 hours, the most specific matching rule wins and allow beats disallow on an equal-length match, and a parser must handle at least 500 kibibytes (checked September 2026). Identify yourself in the User-Agent, fetch each URL at most once per run, and never touch anything behind a login. Second, every sentence in the brief cites a URL and a stored snapshot, or it does not appear. A brief that quietly invents a price change is worse than no brief, because you will act on it and the meeting where that unravels is expensive. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a command line only, run from a scheduler · there is no interface worth building before the noise is under control - Fetching: A headless browser for pages that render client-side, one request per URL per run · half of a modern pricing page does not exist in the HTML - Storage: SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots · text diffs, compresses and searches; a screenshot does none of those - Extraction: A versioned recipe that turns a page into plain text · a change to your own extractor must not look like five competitors relaunching at once - Brief: One model call over the surviving diffs, every bullet required to cite a snapshot · an uncited sentence is a guess with good grammar ### Data model (create this before Phase 1) - competitors(id, name, homepage, enabled, added_at) - watched_urls(id, competitor_id, url, kind, robots_allowed, robots_checked_at, last_status, last_fetch_at) · kind is homepage, pricing, changelog or careers - snapshots(id, url_id, fetched_at, http_status, content_hash, extractor_version, text) · stored only when the hash changed - diffs(id, url_id, from_snapshot_id, to_snapshot_id, changed_chars, unified_text, suppressed_reason, labelled_as) · suppressed rows are kept, not deleted - noise_rules(id, pattern, note, added_at, hits) · an editable regular expression list with a hit count, so a useless rule is visible - briefs(id, window_start, window_end, path, model, tokens_in, tokens_out, sent_at, state) - brief_claims(brief_id, competitor_id, text, url_id, snapshot_id) · a claim with no snapshot id is never written - Suppressed diffs are kept with the reason. The false positive rate is a number you can compute only if you did not throw the evidence away. ### Phase 1 · Fetching politely, storing only what changed Build: Fetch and cache robots.txt per host, re-reading it at least daily, and treat a server error or an unreachable file as a refusal rather than a licence. Render client-side pages in a headless browser. Hash the extracted text and store a snapshot only when the hash changed. Seed the configuration with three competitors and four URLs each. Trap: Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. Done when: A disallowed path is never fetched, tested against a local server serving a robots.txt that forbids it. A robots.txt answered with a server error stops fetching for that host, tested. A page whose content has not changed adds no snapshot row, verified by fetching twice. Every request carries a User-Agent naming the tool and a contact, and each URL is requested at most once per run, asserted by counting. Do not build yet: No diffing, noise filtering, briefs or delivery. ### Phase 2 · Extraction stable enough to diff Build: Turn a page into plain text with a recipe that strips scripts, styles, navigation and footers and keeps the main content. Version the recipe and store the version with every snapshot. A diff spanning two extractor versions is suppressed and the page is re-baselined instead. Trap: Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. Done when: Two snapshots taken with different extractor versions produce a suppressed diff with the reason recorded, tested. Changing a class name or an asset hash on a fixture page produces no diff. The extracted text for a fixture page matches a stored expectation, so extractor changes are deliberate. Every snapshot records the extractor version that produced it. Do not build yet: No noise rules, briefs or delivery yet. ### Phase 3 · Noise, measured rather than felt Build: Drop diffs under a stated changed-character threshold, and drop lines matching an editable list of patterns for dates, cache-busting hashes, view counts and rotating testimonials. Keep suppressed diffs with their reason. Label each surfaced diff as signal or noise for a fortnight and compute the rate per competitor and per URL kind. Trap: Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. Done when: Suppressed diffs are stored with their reason and are visible in a command, not deleted. Each noise rule carries a hit count and a rule with no hits is flagged. A labelled fortnight produces a false positive rate per competitor and per URL kind, printed by a command. Raising the changed-character threshold visibly moves that rate, demonstrated with a stored sample. Do not build yet: No brief generation or delivery yet. ### Phase 4 · A brief where every claim has a receipt Build: Send the surviving diffs for the window in one call, with each diff labelled by competitor, URL and snapshot id. Require every output bullet to carry a snapshot id from that set. Drop any bullet whose id is missing or unknown, and record how many were dropped. Write the brief to a dated Markdown file; that file is the record and email is only delivery. Trap: Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. Done when: A bullet without a snapshot id from the request set is dropped before the file is written, tested by feeding a fabricated response. Every claim in the file links to a URL and a snapshot that exists in the database, asserted over a generated brief. The number of dropped bullets is printed and stored with the brief. The cost of each brief in tokens is recorded and a per-brief ceiling refuses rather than exceeding. Do not build yet: No history view, retention or delivery failure handling yet. ### Phase 5 · History is the only thing you own Build: A history command per competitor printing every change over time with dates and links. Compress stored text and prune raw snapshots past a stated age while keeping every diff and every brief forever. Deliver the brief by mail or webhook with retries, and treat a delivery failure as an alert rather than a lost brief. Trap: Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate. Done when: The history command prints a competitor's changes over time with dates and source links, from stored diffs alone. Pruning removes only raw snapshot text past the stated age and leaves every diff and brief intact, tested by pruning and re-running history. A delivery failure leaves the brief file written and raises an alert, tested by breaking the mail configuration. Database growth per week is printed so the retention decision is informed. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, a paywall or a trial account, which is where the terms you agreed to start applying and where this stops being a public-page watcher. - Hiring feeds, funding data and traffic estimates, which are the analyst layer the paid product sells and each one a separate data licence. - Screenshots as the stored artefact. They cannot be diffed usefully, they cost far more disk, and text is what the brief is built from. - A dashboard. Until the false positive rate is low, an interface just makes it easier to ignore more alerts faster. ### README must contain - That only public pages are fetched, robots.txt is honoured, and a robots.txt server error stops that host entirely - That the extractor is versioned, and a diff across a version boundary is suppressed rather than reported as news - That suppressed diffs are kept with their reason, because the false positive rate is a number you compute from them - That every brief claim cites a stored snapshot and uncited bullets are dropped and counted - The retention policy, the weekly disk growth, and the per-brief token cost, plus a warning that month one is mostly tuning the noise rules ===== ARCHITECTURE.md ===== # Architecture · Debriefing ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a command line only, run from a scheduler | there is no interface worth building before the noise is under control | | Fetching | A headless browser for pages that render client-side, one request per URL per run | half of a modern pricing page does not exist in the HTML | | Storage | SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots | text diffs, compresses and searches; a screenshot does none of those | | Extraction | A versioned recipe that turns a page into plain text | a change to your own extractor must not look like five competitors relaunching at once | | Brief | One model call over the surviving diffs, every bullet required to cite a snapshot | an uncited sentence is a guess with good grammar | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Fetch | robots.txt, rendering and one request per URL per run. | Any client that honours the same rules and identifies itself. | | Extract | The versioned page-to-text recipe. | Any recipe, provided the version travels with the snapshot. | | Noise | Thresholds, patterns and suppressed evidence. | Any filter, provided suppressed diffs are kept and counted. | | Brief | One call with citations enforced. | Any model, provided uncited bullets are still dropped. | | History | Trends built from diffs alone. | Not swappable for anything that needs raw snapshots to exist. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Snapshots, diffs, briefs and claims. The history is the product. - `USER_AGENT` · required · Identifies the fetcher on every request, as the standard expects. - `MIN_CHANGED_CHARS` · required · Diffs smaller than this are suppressed with a reason, not deleted. - `SNAPSHOT_RETENTION_DAYS` · required · Raw text only. Diffs and briefs are kept for ever. - `LLM_API_KEY` · required, secret · One call per brief. Give it its own spend limit. - `BRIEF_TOKEN_CEILING` · required · Checked before the call. Refusing beats a surprise invoice. - `BRIEF_DIR` · required · Dated Markdown files. This is the record; email is delivery. - `SMTP_URL` · required, secret · Delivery only. A failure alerts and the file stays. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Debriefing product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with TypeScript, a command line only, run from a scheduler, A headless browser for pages that render client-side, one request per URL per run, SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots, A versioned recipe that turns a page into plain text, One model call over the surviving diffs, every bullet required to cite a snapshot. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. - Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate. ===== MILESTONES.md ===== # Delivery milestones · Debriefing Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Fetching politely, storing only what changed Every fetch is permitted, identified, and produces a snapshot only when the content actually differs. ### Steps 1. Cache robots.txt per host with an expiry Daily at most, and a server error means stop. Files: `src/fetch/robots.ts`, `tests/robots.test.ts` 2. Render client-side pages properly A pricing page that is empty in the HTML is the normal case. Files: `src/fetch/browser.ts` 3. Hash first, store second An unchanged page should cost one request and nothing else. Files: `src/snapshot.ts` ### Done when - [ ] A disallowed path is never fetched - [ ] A robots.txt server error stops that host - [ ] An unchanged page stores no snapshot - [ ] One request per URL per run, with a named User-Agent ### Watch out - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. ## M2 · Extraction stable enough to diff A change to your own extraction never looks like a change on the competitor's site. ### Steps 1. Write the extraction recipe once, and version it The version travels with every snapshot it produced. Files: `src/extract/recipe.ts`, `src/extract/version.ts` 2. Pin fixture pages with expected text So an extractor change is a failing test rather than a surprise brief. Files: `tests/fixtures/`, `tests/extract.test.ts` 3. Suppress and re-baseline across a version change Your refactor is not their announcement. Files: `src/diff.ts` ### Done when - [ ] A cross-version diff is suppressed with a reason - [ ] Markup churn produces no diff - [ ] Fixture extraction matches stored expectations - [ ] Every snapshot records its extractor version ### Watch out - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. ## M3 · Noise, measured rather than felt The false positive rate is a number you can look at, and it falls. ### Steps 1. Keep suppressed diffs with reasons The evidence is the only way to tune anything. Files: `src/noise.ts`, `src/diff.ts` 2. Count hits per noise rule A rule that never fires is a rule that might be hiding something. Files: `src/noise.ts` 3. Label a fortnight and compute the rate Month one is tuning, so make tuning measurable. Files: `src/label.ts`, `src/report/falsepositive.ts` ### Done when - [ ] Suppressed diffs are kept with reasons - [ ] Each noise rule shows a hit count - [ ] A labelled sample yields a false positive rate - [ ] Changing the threshold moves the rate visibly ### Watch out - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. ## M4 · A brief where every claim has a receipt No sentence reaches the file unless it names a URL and a snapshot you stored. ### Steps 1. Label every diff with a snapshot id in the prompt The model cannot cite what you did not give it. Files: `src/brief/prompt.ts` 2. Drop uncited bullets and count them The count is the health metric for this phase. Files: `src/brief/validate.ts`, `tests/citations.test.ts` 3. Write the file first, deliver second The Markdown file is the record; email can fail. Files: `src/brief/write.ts` ### Done when - [ ] Uncited bullets never reach the file - [ ] Every claim resolves to a stored snapshot - [ ] The drop count is printed and stored - [ ] A per-brief cost ceiling refuses rather than overspends ### Watch out - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. ## M5 · History is the only thing you own (production only) A single change reads as a trend, and the archive survives a disk decision. ### Steps 1. Build history from diffs, not snapshots So pruning snapshots never costs you the trend. Files: `src/history.ts`, `tests/history-after-prune.test.ts` 2. Prune raw text on a stated age only And say so in the README, because it is a real trade-off. Files: `src/retention.ts` 3. Alert on delivery failure, keep the file The file is the brief. Email is a convenience. Files: `src/deliver.ts` ### Done when - [ ] History prints from diffs alone - [ ] Pruning leaves diffs and briefs intact - [ ] A delivery failure alerts and keeps the file - [ ] Weekly growth is printed ### Watch out - Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate. ===== OPERATIONS.md ===== # Operations · Debriefing ## Backup The database and the brief folder. The database holds the diffs, which are the accumulated asset; the folder holds the briefs, which are what you actually read. Raw snapshot text is the largest part and the least valuable, which is why retention applies to it alone. ## Restore Restore both, run one fetch cycle, and confirm it produces no diffs for pages that have not changed. A restore that produces a flood of diffs means snapshots came back but hashes did not. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Suppressed diff counts per rule, the false positive rate from your labelled sample, dropped brief bullets, weekly database growth, and hosts currently disallowed by robots.txt. A host that silently became disallowed is the one that looks like a competitor going quiet. ## Incident checklist A brief full of nothing usually means the extractor changed or a site rebuilt, and the extractor version on the snapshots will say which. A brief with a claim you cannot verify should not exist; if one does, the citation enforcement has a hole and that is the bug to fix first. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A disallowed path is confirmed never fetched - [ ] A robots.txt server error is confirmed to stop that host - [ ] A cross-version diff is confirmed suppressed - [ ] Uncited bullets are confirmed dropped and counted - [ ] History is confirmed to survive a snapshot prune ## Launch constraint Do not market omitted Debriefing capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Snapshots, diffs, briefs and claims. The history is the product. DATABASE_PATH=./data/watch.db # Required. Identifies the fetcher on every request, as the standard expects. USER_AGENT=competitor-watch/0.1 (contact: you@example.com) # Required. Diffs smaller than this are suppressed with a reason, not deleted. MIN_CHANGED_CHARS=40 # Required. Raw text only. Diffs and briefs are kept for ever. SNAPSHOT_RETENTION_DAYS=365 # Required · secret. One call per brief. Give it its own spend limit. LLM_API_KEY=sk-...redacted # Required. Checked before the call. Refusing beats a surprise invoice. BRIEF_TOKEN_CEILING=60000 # Required. Dated Markdown files. This is the record; email is delivery. BRIEF_DIR=~/CompetitorBriefs # Required · secret. Delivery only. A failure alerts and the file stays. SMTP_URL=smtps://user:redacted@smtp.example:465
README.md
# Debriefing · indie build A local competitor watch whose hard problems are telling a real change from a rotated testimonial, keeping every claim traceable to a stored snapshot, and keeping enough history that a single diff eventually reads as a trend. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a command line only, run from a scheduler | there is no interface worth building before the noise is under control | | Fetching | A headless browser for pages that render client-side, one request per URL per run | half of a modern pricing page does not exist in the HTML | | Storage | SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots | text diffs, compresses and searches; a screenshot does none of those | | Extraction | A versioned recipe that turns a page into plain text | a change to your own extractor must not look like five competitors relaunching at once | | Brief | One model call over the surviving diffs, every bullet required to cite a snapshot | an uncited sentence is a guess with good grammar | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the fetcher, the differ and the brief generator. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A headless browser with its runtime dependencies** · free - Why: A pricing page that renders client-side is empty in the HTML, and that is the page you most want to watch. - Get it: Install a browser automation library and let it download its browser binary. - Verify: A client-rendered page yields the same text a human sees - [ ] **An API key for a language model** · pennies per brief at a handful of competitors - Why: The brief is one call per window and it needs a ceiling before it needs anything else. - Get it: Create a key with its own spend limit. - Verify: One brief generates and the token cost is recorded - [ ] **Your competitor list and the URLs that matter** · free - Why: Watching the wrong four pages produces a brief about nothing, and the choice is judgement rather than code. - Get it: For each rival, pick the homepage, the pricing page, the changelog and the careers page, and check each one is public. - Verify: Every URL is permitted by that host's robots.txt and returns content - [ ] **SMTP credentials or a webhook you control** · free at one message a week - Why: A brief you have to remember to read is a brief you stop reading in week three. - Get it: Any transactional mail provider, or an incoming webhook on a channel you actually watch. - Verify: A test brief arrives - [ ] **Your change threshold, retention and cost ceiling** · free - Why: Each is asserted by a test, and tuning them after a noisy month means tuning to whatever you have already grown used to. - Get it: Write down the minimum changed characters, how long raw snapshots are kept, and the maximum tokens per brief. - Verify: All three are in the configuration and named in tests ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Anything behind a login or a paywall. - Hiring, funding and traffic data. - Screenshots as the stored artefact. - A dashboard. - noise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves - horizon watch, meaning the substitutes and new entrants you did not think to add to the list - the non-page signals stitched into the same brief: hiring, funding, traffic and LinkedIn movement - the analytical step from what changed to why it matters to what to do next - an archive going back far enough that a change reads as a trend rather than an event If one of those is essential to you, that is the reason to keep paying for Debriefing, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Debriefing The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a competitor watch that files a weekly brief and drops any claim it cannot cite to replace Debriefing. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules. Public pages only, fetched the way a well-behaved crawler fetches them. Read each site's robots.txt and honour it. The standard is specific: a robots.txt answered with a server error means assume complete disallow, a cached copy should not be used for more than 24 hours, the most specific matching rule wins and allow beats disallow on an equal-length match, and a parser must handle at least 500 kibibytes (checked September 2026). Identify yourself in the User-Agent, fetch each URL at most once per run, and never touch anything behind a login. Second, every sentence in the brief cites a URL and a stored snapshot, or it does not appear. A brief that quietly invents a price change is worse than no brief, because you will act on it and the meeting where that unravels is expensive. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a command line only, run from a scheduler · there is no interface worth building before the noise is under control - Fetching: A headless browser for pages that render client-side, one request per URL per run · half of a modern pricing page does not exist in the HTML - Storage: SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots · text diffs, compresses and searches; a screenshot does none of those - Extraction: A versioned recipe that turns a page into plain text · a change to your own extractor must not look like five competitors relaunching at once - Brief: One model call over the surviving diffs, every bullet required to cite a snapshot · an uncited sentence is a guess with good grammar ### Data model (create this before Phase 1) - competitors(id, name, homepage, enabled, added_at) - watched_urls(id, competitor_id, url, kind, robots_allowed, robots_checked_at, last_status, last_fetch_at) · kind is homepage, pricing, changelog or careers - snapshots(id, url_id, fetched_at, http_status, content_hash, extractor_version, text) · stored only when the hash changed - diffs(id, url_id, from_snapshot_id, to_snapshot_id, changed_chars, unified_text, suppressed_reason, labelled_as) · suppressed rows are kept, not deleted - noise_rules(id, pattern, note, added_at, hits) · an editable regular expression list with a hit count, so a useless rule is visible - briefs(id, window_start, window_end, path, model, tokens_in, tokens_out, sent_at, state) - brief_claims(brief_id, competitor_id, text, url_id, snapshot_id) · a claim with no snapshot id is never written - Suppressed diffs are kept with the reason. The false positive rate is a number you can compute only if you did not throw the evidence away. ### Phase 1 · Fetching politely, storing only what changed Build: Fetch and cache robots.txt per host, re-reading it at least daily, and treat a server error or an unreachable file as a refusal rather than a licence. Render client-side pages in a headless browser. Hash the extracted text and store a snapshot only when the hash changed. Seed the configuration with three competitors and four URLs each. Trap: Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. Done when: A disallowed path is never fetched, tested against a local server serving a robots.txt that forbids it. A robots.txt answered with a server error stops fetching for that host, tested. A page whose content has not changed adds no snapshot row, verified by fetching twice. Every request carries a User-Agent naming the tool and a contact, and each URL is requested at most once per run, asserted by counting. Do not build yet: No diffing, noise filtering, briefs or delivery. ### Phase 2 · Extraction stable enough to diff Build: Turn a page into plain text with a recipe that strips scripts, styles, navigation and footers and keeps the main content. Version the recipe and store the version with every snapshot. A diff spanning two extractor versions is suppressed and the page is re-baselined instead. Trap: Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. Done when: Two snapshots taken with different extractor versions produce a suppressed diff with the reason recorded, tested. Changing a class name or an asset hash on a fixture page produces no diff. The extracted text for a fixture page matches a stored expectation, so extractor changes are deliberate. Every snapshot records the extractor version that produced it. Do not build yet: No noise rules, briefs or delivery yet. ### Phase 3 · Noise, measured rather than felt Build: Drop diffs under a stated changed-character threshold, and drop lines matching an editable list of patterns for dates, cache-busting hashes, view counts and rotating testimonials. Keep suppressed diffs with their reason. Label each surfaced diff as signal or noise for a fortnight and compute the rate per competitor and per URL kind. Trap: Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. Done when: Suppressed diffs are stored with their reason and are visible in a command, not deleted. Each noise rule carries a hit count and a rule with no hits is flagged. A labelled fortnight produces a false positive rate per competitor and per URL kind, printed by a command. Raising the changed-character threshold visibly moves that rate, demonstrated with a stored sample. Do not build yet: No brief generation or delivery yet. ### Phase 4 · A brief where every claim has a receipt Build: Send the surviving diffs for the window in one call, with each diff labelled by competitor, URL and snapshot id. Require every output bullet to carry a snapshot id from that set. Drop any bullet whose id is missing or unknown, and record how many were dropped. Write the brief to a dated Markdown file; that file is the record and email is only delivery. Trap: Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. Done when: A bullet without a snapshot id from the request set is dropped before the file is written, tested by feeding a fabricated response. Every claim in the file links to a URL and a snapshot that exists in the database, asserted over a generated brief. The number of dropped bullets is printed and stored with the brief. The cost of each brief in tokens is recorded and a per-brief ceiling refuses rather than exceeding. Do not build yet: No history view, retention or delivery failure handling yet. ### Phase 5 · History is the only thing you own Build: A history command per competitor printing every change over time with dates and links. Compress stored text and prune raw snapshots past a stated age while keeping every diff and every brief forever. Deliver the brief by mail or webhook with retries, and treat a delivery failure as an alert rather than a lost brief. Trap: Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate. Done when: The history command prints a competitor's changes over time with dates and source links, from stored diffs alone. Pruning removes only raw snapshot text past the stated age and leaves every diff and brief intact, tested by pruning and re-running history. A delivery failure leaves the brief file written and raises an alert, tested by breaking the mail configuration. Database growth per week is printed so the retention decision is informed. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, a paywall or a trial account, which is where the terms you agreed to start applying and where this stops being a public-page watcher. - Hiring feeds, funding data and traffic estimates, which are the analyst layer the paid product sells and each one a separate data licence. - Screenshots as the stored artefact. They cannot be diffed usefully, they cost far more disk, and text is what the brief is built from. - A dashboard. Until the false positive rate is low, an interface just makes it easier to ignore more alerts faster. ### README must contain - That only public pages are fetched, robots.txt is honoured, and a robots.txt server error stops that host entirely - That the extractor is versioned, and a diff across a version boundary is suppressed rather than reported as news - That suppressed diffs are kept with their reason, because the false positive rate is a number you compute from them - That every brief claim cites a stored snapshot and uncited bullets are dropped and counted - The retention policy, the weekly disk growth, and the per-brief token cost, plus a warning that month one is mostly tuning the noise rules
AGENTS.md
# Agent instructions · Debriefing indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript, a command line only, run from a scheduler, A headless browser for pages that render client-side, one request per URL per run, SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots, A versioned recipe that turns a page into plain text, One model call over the surviving diffs, every bullet required to cite a snapshot. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive.
BUILD_PLAN.md
# Build plan · Debriefing A local competitor watch whose hard problems are telling a real change from a rotated testimonial, keeping every claim traceable to a stored snapshot, and keeping enough history that a single diff eventually reads as a trend. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Fetching politely, storing only what changed Every fetch is permitted, identified, and produces a snapshot only when the content actually differs. ### Steps 1. Cache robots.txt per host with an expiry Daily at most, and a server error means stop. Files: `src/fetch/robots.ts`, `tests/robots.test.ts` 2. Render client-side pages properly A pricing page that is empty in the HTML is the normal case. Files: `src/fetch/browser.ts` 3. Hash first, store second An unchanged page should cost one request and nothing else. Files: `src/snapshot.ts` ### Done when - [ ] A disallowed path is never fetched - [ ] A robots.txt server error stops that host - [ ] An unchanged page stores no snapshot - [ ] One request per URL per run, with a named User-Agent ### Watch out - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. ## Phase 2 · Extraction stable enough to diff A change to your own extraction never looks like a change on the competitor's site. ### Steps 1. Write the extraction recipe once, and version it The version travels with every snapshot it produced. Files: `src/extract/recipe.ts`, `src/extract/version.ts` 2. Pin fixture pages with expected text So an extractor change is a failing test rather than a surprise brief. Files: `tests/fixtures/`, `tests/extract.test.ts` 3. Suppress and re-baseline across a version change Your refactor is not their announcement. Files: `src/diff.ts` ### Done when - [ ] A cross-version diff is suppressed with a reason - [ ] Markup churn produces no diff - [ ] Fixture extraction matches stored expectations - [ ] Every snapshot records its extractor version ### Watch out - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. ## Phase 3 · Noise, measured rather than felt The false positive rate is a number you can look at, and it falls. ### Steps 1. Keep suppressed diffs with reasons The evidence is the only way to tune anything. Files: `src/noise.ts`, `src/diff.ts` 2. Count hits per noise rule A rule that never fires is a rule that might be hiding something. Files: `src/noise.ts` 3. Label a fortnight and compute the rate Month one is tuning, so make tuning measurable. Files: `src/label.ts`, `src/report/falsepositive.ts` ### Done when - [ ] Suppressed diffs are kept with reasons - [ ] Each noise rule shows a hit count - [ ] A labelled sample yields a false positive rate - [ ] Changing the threshold moves the rate visibly ### Watch out - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. ## Phase 4 · A brief where every claim has a receipt No sentence reaches the file unless it names a URL and a snapshot you stored. ### Steps 1. Label every diff with a snapshot id in the prompt The model cannot cite what you did not give it. Files: `src/brief/prompt.ts` 2. Drop uncited bullets and count them The count is the health metric for this phase. Files: `src/brief/validate.ts`, `tests/citations.test.ts` 3. Write the file first, deliver second The Markdown file is the record; email can fail. Files: `src/brief/write.ts` ### Done when - [ ] Uncited bullets never reach the file - [ ] Every claim resolves to a stored snapshot - [ ] The drop count is printed and stored - [ ] A per-brief cost ceiling refuses rather than overspends ### Watch out - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. ## Not in this build - Anything behind a login or a paywall. - Hiring, funding and traffic data. - Screenshots as the stored artefact. - A dashboard. ## After v1, if you want it - Sitemap watching to catch pages you did not think to add, still subject to the same rules - A monthly rollup built from the same diffs, which needs no new fetching - Per-competitor thresholds, once the labelled sample shows which ones are noisy
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Snapshots, diffs, briefs and claims. The history is the product. DATABASE_PATH=./data/watch.db # Required. Identifies the fetcher on every request, as the standard expects. USER_AGENT=competitor-watch/0.1 (contact: you@example.com) # Required. Diffs smaller than this are suppressed with a reason, not deleted. MIN_CHANGED_CHARS=40 # Required. Raw text only. Diffs and briefs are kept for ever. SNAPSHOT_RETENTION_DAYS=365 # Required · secret. One call per brief. Give it its own spend limit. LLM_API_KEY=sk-...redacted # Required. Checked before the call. Refusing beats a surprise invoice. BRIEF_TOKEN_CEILING=60000 # Required. Dated Markdown files. This is the record; email is delivery. BRIEF_DIR=~/CompetitorBriefs # Required · secret. Delivery only. A failure alerts and the file stays. SMTP_URL=smtps://user:redacted@smtp.example:465
PRODUCT.md
# Debriefing · product brief ## Problem Diffing a competitor's pricing page on a cron and asking a model what changed is genuinely an afternoon. changedetection.io will do the watching for you before you write a line. The gap is everything between a diff and a brief. Most page changes are noise: a rotated testimonial, a reordered nav, a CDN hash. Deciding which changes are real, tying them to hiring and funding signals, and turning that into three sentences a founder acts on is judgement encoded over many iterations. Build it and your first month is mostly you deleting alerts about nothing. ## Product outcome A weekly brief whose every claim links to a page snapshot you stored, whose noise rate you can measure and reduce, and whose accumulated history lets a single change be read as part of a pattern. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - an OpenAI or Anthropic API key - a scheduler and somewhere to keep page snapshots - a headless browser for pages that render client-side - SMTP or a Slack webhook for delivery - patience for the first month of false positives ## Explicit non-goals for v1 - Anything behind a login or a paywall. - Hiring, funding and traffic data. - Screenshots as the stored artefact. - A dashboard. - noise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves - horizon watch, meaning the substitutes and new entrants you did not think to add to the list - the non-page signals stitched into the same brief: hiring, funding, traffic and LinkedIn movement - the analytical step from what changed to why it matters to what to do next - an archive going back far enough that a change reads as a trend rather than an event ## Success criteria - A disallowed path is confirmed never fetched - A robots.txt server error is confirmed to stop that host - A cross-version diff is confirmed suppressed - Uncited bullets are confirmed dropped and counted - History is confirmed to survive a snapshot prune
BRIEF.md
# Build brief · Debriefing The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a competitor watch that files a weekly brief and drops any claim it cannot cite to replace Debriefing. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules. Public pages only, fetched the way a well-behaved crawler fetches them. Read each site's robots.txt and honour it. The standard is specific: a robots.txt answered with a server error means assume complete disallow, a cached copy should not be used for more than 24 hours, the most specific matching rule wins and allow beats disallow on an equal-length match, and a parser must handle at least 500 kibibytes (checked September 2026). Identify yourself in the User-Agent, fetch each URL at most once per run, and never touch anything behind a login. Second, every sentence in the brief cites a URL and a stored snapshot, or it does not appear. A brief that quietly invents a price change is worse than no brief, because you will act on it and the meeting where that unravels is expensive. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a command line only, run from a scheduler · there is no interface worth building before the noise is under control - Fetching: A headless browser for pages that render client-side, one request per URL per run · half of a modern pricing page does not exist in the HTML - Storage: SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots · text diffs, compresses and searches; a screenshot does none of those - Extraction: A versioned recipe that turns a page into plain text · a change to your own extractor must not look like five competitors relaunching at once - Brief: One model call over the surviving diffs, every bullet required to cite a snapshot · an uncited sentence is a guess with good grammar ### Data model (create this before Phase 1) - competitors(id, name, homepage, enabled, added_at) - watched_urls(id, competitor_id, url, kind, robots_allowed, robots_checked_at, last_status, last_fetch_at) · kind is homepage, pricing, changelog or careers - snapshots(id, url_id, fetched_at, http_status, content_hash, extractor_version, text) · stored only when the hash changed - diffs(id, url_id, from_snapshot_id, to_snapshot_id, changed_chars, unified_text, suppressed_reason, labelled_as) · suppressed rows are kept, not deleted - noise_rules(id, pattern, note, added_at, hits) · an editable regular expression list with a hit count, so a useless rule is visible - briefs(id, window_start, window_end, path, model, tokens_in, tokens_out, sent_at, state) - brief_claims(brief_id, competitor_id, text, url_id, snapshot_id) · a claim with no snapshot id is never written - Suppressed diffs are kept with the reason. The false positive rate is a number you can compute only if you did not throw the evidence away. ### Phase 1 · Fetching politely, storing only what changed Build: Fetch and cache robots.txt per host, re-reading it at least daily, and treat a server error or an unreachable file as a refusal rather than a licence. Render client-side pages in a headless browser. Hash the extracted text and store a snapshot only when the hash changed. Seed the configuration with three competitors and four URLs each. Trap: Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. Done when: A disallowed path is never fetched, tested against a local server serving a robots.txt that forbids it. A robots.txt answered with a server error stops fetching for that host, tested. A page whose content has not changed adds no snapshot row, verified by fetching twice. Every request carries a User-Agent naming the tool and a contact, and each URL is requested at most once per run, asserted by counting. Do not build yet: No diffing, noise filtering, briefs or delivery. ### Phase 2 · Extraction stable enough to diff Build: Turn a page into plain text with a recipe that strips scripts, styles, navigation and footers and keeps the main content. Version the recipe and store the version with every snapshot. A diff spanning two extractor versions is suppressed and the page is re-baselined instead. Trap: Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. Done when: Two snapshots taken with different extractor versions produce a suppressed diff with the reason recorded, tested. Changing a class name or an asset hash on a fixture page produces no diff. The extracted text for a fixture page matches a stored expectation, so extractor changes are deliberate. Every snapshot records the extractor version that produced it. Do not build yet: No noise rules, briefs or delivery yet. ### Phase 3 · Noise, measured rather than felt Build: Drop diffs under a stated changed-character threshold, and drop lines matching an editable list of patterns for dates, cache-busting hashes, view counts and rotating testimonials. Keep suppressed diffs with their reason. Label each surfaced diff as signal or noise for a fortnight and compute the rate per competitor and per URL kind. Trap: Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. Done when: Suppressed diffs are stored with their reason and are visible in a command, not deleted. Each noise rule carries a hit count and a rule with no hits is flagged. A labelled fortnight produces a false positive rate per competitor and per URL kind, printed by a command. Raising the changed-character threshold visibly moves that rate, demonstrated with a stored sample. Do not build yet: No brief generation or delivery yet. ### Phase 4 · A brief where every claim has a receipt Build: Send the surviving diffs for the window in one call, with each diff labelled by competitor, URL and snapshot id. Require every output bullet to carry a snapshot id from that set. Drop any bullet whose id is missing or unknown, and record how many were dropped. Write the brief to a dated Markdown file; that file is the record and email is only delivery. Trap: Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. Done when: A bullet without a snapshot id from the request set is dropped before the file is written, tested by feeding a fabricated response. Every claim in the file links to a URL and a snapshot that exists in the database, asserted over a generated brief. The number of dropped bullets is printed and stored with the brief. The cost of each brief in tokens is recorded and a per-brief ceiling refuses rather than exceeding. Do not build yet: No history view, retention or delivery failure handling yet. ### Phase 5 · History is the only thing you own Build: A history command per competitor printing every change over time with dates and links. Compress stored text and prune raw snapshots past a stated age while keeping every diff and every brief forever. Deliver the brief by mail or webhook with retries, and treat a delivery failure as an alert rather than a lost brief. Trap: Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate. Done when: The history command prints a competitor's changes over time with dates and source links, from stored diffs alone. Pruning removes only raw snapshot text past the stated age and leaves every diff and brief intact, tested by pruning and re-running history. A delivery failure leaves the brief file written and raises an alert, tested by breaking the mail configuration. Database growth per week is printed so the retention decision is informed. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, a paywall or a trial account, which is where the terms you agreed to start applying and where this stops being a public-page watcher. - Hiring feeds, funding data and traffic estimates, which are the analyst layer the paid product sells and each one a separate data licence. - Screenshots as the stored artefact. They cannot be diffed usefully, they cost far more disk, and text is what the brief is built from. - A dashboard. Until the false positive rate is low, an interface just makes it easier to ignore more alerts faster. ### README must contain - That only public pages are fetched, robots.txt is honoured, and a robots.txt server error stops that host entirely - That the extractor is versioned, and a diff across a version boundary is suppressed rather than reported as news - That suppressed diffs are kept with their reason, because the false positive rate is a number you compute from them - That every brief claim cites a stored snapshot and uncited bullets are dropped and counted - The retention policy, the weekly disk growth, and the per-brief token cost, plus a warning that month one is mostly tuning the noise rules
ARCHITECTURE.md
# Architecture · Debriefing ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a command line only, run from a scheduler | there is no interface worth building before the noise is under control | | Fetching | A headless browser for pages that render client-side, one request per URL per run | half of a modern pricing page does not exist in the HTML | | Storage | SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots | text diffs, compresses and searches; a screenshot does none of those | | Extraction | A versioned recipe that turns a page into plain text | a change to your own extractor must not look like five competitors relaunching at once | | Brief | One model call over the surviving diffs, every bullet required to cite a snapshot | an uncited sentence is a guess with good grammar | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Fetch | robots.txt, rendering and one request per URL per run. | Any client that honours the same rules and identifies itself. | | Extract | The versioned page-to-text recipe. | Any recipe, provided the version travels with the snapshot. | | Noise | Thresholds, patterns and suppressed evidence. | Any filter, provided suppressed diffs are kept and counted. | | Brief | One call with citations enforced. | Any model, provided uncited bullets are still dropped. | | History | Trends built from diffs alone. | Not swappable for anything that needs raw snapshots to exist. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Snapshots, diffs, briefs and claims. The history is the product. - `USER_AGENT` · required · Identifies the fetcher on every request, as the standard expects. - `MIN_CHANGED_CHARS` · required · Diffs smaller than this are suppressed with a reason, not deleted. - `SNAPSHOT_RETENTION_DAYS` · required · Raw text only. Diffs and briefs are kept for ever. - `LLM_API_KEY` · required, secret · One call per brief. Give it its own spend limit. - `BRIEF_TOKEN_CEILING` · required · Checked before the call. Refusing beats a surprise invoice. - `BRIEF_DIR` · required · Dated Markdown files. This is the record; email is delivery. - `SMTP_URL` · required, secret · Delivery only. A failure alerts and the file stays. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Debriefing product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with TypeScript, a command line only, run from a scheduler, A headless browser for pages that render client-side, one request per URL per run, SQLite for snapshots, diffs and briefs, with the extracted text stored, not screenshots, A versioned recipe that turns a page into plain text, One model call over the surviving diffs, every bullet required to cite a snapshot. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. - Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate.
MILESTONES.md
# Delivery milestones · Debriefing Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Fetching politely, storing only what changed Every fetch is permitted, identified, and produces a snapshot only when the content actually differs. ### Steps 1. Cache robots.txt per host with an expiry Daily at most, and a server error means stop. Files: `src/fetch/robots.ts`, `tests/robots.test.ts` 2. Render client-side pages properly A pricing page that is empty in the HTML is the normal case. Files: `src/fetch/browser.ts` 3. Hash first, store second An unchanged page should cost one request and nothing else. Files: `src/snapshot.ts` ### Done when - [ ] A disallowed path is never fetched - [ ] A robots.txt server error stops that host - [ ] An unchanged page stores no snapshot - [ ] One request per URL per run, with a named User-Agent ### Watch out - Fetching robots.txt once at setup and caching it forever means you keep crawling a path that was disallowed six months ago, and treating a 503 on robots.txt as 'no rules found' does the same thing on a day the site is merely struggling. Re-read at least daily, and treat a server error as complete disallow, which is exactly what the standard asks for. ## M2 · Extraction stable enough to diff A change to your own extraction never looks like a change on the competitor's site. ### Steps 1. Write the extraction recipe once, and version it The version travels with every snapshot it produced. Files: `src/extract/recipe.ts`, `src/extract/version.ts` 2. Pin fixture pages with expected text So an extractor change is a failing test rather than a surprise brief. Files: `tests/fixtures/`, `tests/extract.test.ts` 3. Suppress and re-baseline across a version change Your refactor is not their announcement. Files: `src/diff.ts` ### Done when - [ ] A cross-version diff is suppressed with a reason - [ ] Markup churn produces no diff - [ ] Fixture extraction matches stored expectations - [ ] Every snapshot records its extractor version ### Watch out - Diffing raw HTML makes a changed class name, a rebuilt asset hash or a reordered attribute into a competitive event, and your first brief announces that all five rivals relaunched on the same Tuesday. Diff extracted text, and when you improve the extractor, suppress the diff across the version boundary and re-baseline rather than reporting your own refactor as news. ## M3 · Noise, measured rather than felt The false positive rate is a number you can look at, and it falls. ### Steps 1. Keep suppressed diffs with reasons The evidence is the only way to tune anything. Files: `src/noise.ts`, `src/diff.ts` 2. Count hits per noise rule A rule that never fires is a rule that might be hiding something. Files: `src/noise.ts` 3. Label a fortnight and compute the rate Month one is tuning, so make tuning measurable. Files: `src/label.ts`, `src/report/falsepositive.ts` ### Done when - [ ] Suppressed diffs are kept with reasons - [ ] Each noise rule shows a hit count - [ ] A labelled sample yields a false positive rate - [ ] Changing the threshold moves the rate visibly ### Watch out - Deleting suppressed diffs to keep the database small destroys the only way to tell whether your noise rules are working or are quietly hiding real changes, and the second failure is the one that costs you. Keep them with the reason, count hits per rule, and treat a rule with no hits in a month as a rule to delete. ## M4 · A brief where every claim has a receipt No sentence reaches the file unless it names a URL and a snapshot you stored. ### Steps 1. Label every diff with a snapshot id in the prompt The model cannot cite what you did not give it. Files: `src/brief/prompt.ts` 2. Drop uncited bullets and count them The count is the health metric for this phase. Files: `src/brief/validate.ts`, `tests/citations.test.ts` 3. Write the file first, deliver second The Markdown file is the record; email can fail. Files: `src/brief/write.ts` ### Done when - [ ] Uncited bullets never reach the file - [ ] Every claim resolves to a stored snapshot - [ ] The drop count is printed and stored - [ ] A per-brief cost ceiling refuses rather than overspends ### Watch out - Asking for a nice summary and pasting it into the email gives you confident sentences about a price change nobody made, and because the format looks the same as the true ones you cannot tell which is which. Require a snapshot id per bullet, drop the rest, and print the drop count so a model having a bad day is visible rather than persuasive. ## M5 · History is the only thing you own (production only) A single change reads as a trend, and the archive survives a disk decision. ### Steps 1. Build history from diffs, not snapshots So pruning snapshots never costs you the trend. Files: `src/history.ts`, `tests/history-after-prune.test.ts` 2. Prune raw text on a stated age only And say so in the README, because it is a real trade-off. Files: `src/retention.ts` 3. Alert on delivery failure, keep the file The file is the brief. Email is a convenience. Files: `src/deliver.ts` ### Done when - [ ] History prints from diffs alone - [ ] Pruning leaves diffs and briefs intact - [ ] A delivery failure alerts and keeps the file - [ ] Weekly growth is printed ### Watch out - Pruning old snapshots to reclaim disk deletes the one asset this build accumulates that a five line script does not have: a year of history that turns 'they changed a price' into 'they have raised the entry tier three times since spring'. Prune raw snapshot text if you must, but keep every diff and every brief, and state the retention in the README so the decision is deliberate.
OPERATIONS.md
# Operations · Debriefing ## Backup The database and the brief folder. The database holds the diffs, which are the accumulated asset; the folder holds the briefs, which are what you actually read. Raw snapshot text is the largest part and the least valuable, which is why retention applies to it alone. ## Restore Restore both, run one fetch cycle, and confirm it produces no diffs for pages that have not changed. A restore that produces a flood of diffs means snapshots came back but hashes did not. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Suppressed diff counts per rule, the false positive rate from your labelled sample, dropped brief bullets, weekly database growth, and hosts currently disallowed by robots.txt. A host that silently became disallowed is the one that looks like a competitor going quiet. ## Incident checklist A brief full of nothing usually means the extractor changed or a site rebuilt, and the extractor version on the snapshots will say which. A brief with a claim you cannot verify should not exist; if one does, the citation enforcement has a hole and that is the bug to fix first. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A disallowed path is confirmed never fetched - [ ] A robots.txt server error is confirmed to stop that host - [ ] A cross-version diff is confirmed suppressed - [ ] Uncited bullets are confirmed dropped and counted - [ ] History is confirmed to survive a snapshot prune ## Launch constraint Do not market omitted Debriefing capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Snapshots, diffs, briefs and claims. The history is the product. DATABASE_PATH=./data/watch.db # Required. Identifies the fetcher on every request, as the standard expects. USER_AGENT=competitor-watch/0.1 (contact: you@example.com) # Required. Diffs smaller than this are suppressed with a reason, not deleted. MIN_CHANGED_CHARS=40 # Required. Raw text only. Diffs and briefs are kept for ever. SNAPSHOT_RETENTION_DAYS=365 # Required · secret. One call per brief. Give it its own spend limit. LLM_API_KEY=sk-...redacted # Required. Checked before the call. Refusing beats a surprise invoice. BRIEF_TOKEN_CEILING=60000 # Required. Dated Markdown files. This is the record; email is delivery. BRIEF_DIR=~/CompetitorBriefs # Required · secret. Delivery only. A failure alerts and the file stays. SMTP_URL=smtps://user:redacted@smtp.example:465
$ choose a build depth, inspect the files, then open the complete pack in your agent
Because a diff is not intelligence. The hard part is not fetching a pricing page every week, it is knowing that this particular change matters and the other eleven do not, then saying so in three sentences a founder can act on before a Monday call. That judgement lives in accumulated history and a lot of tuning against false positives. It also survives being ignored: a brief that arrives whether or not you remembered to look is worth more than a script you stop reading in week three.
xnoise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves
xhorizon watch, meaning the substitutes and new entrants you did not think to add to the list
xthe non-page signals stitched into the same brief: hiring, funding, traffic and LinkedIn movement
xthe analytical step from what changed to why it matters to what to do next
xan archive going back far enough that a change reads as a trend rather than an event
share on X ↗"I just replaced Debriefing ($80/mo) with one prompt"
Don't feel like building it? These folks already made it free.
changedetection.ioTells you a competitor's pricing page moved. Deciding whether that mattered is back to being your job.open source↗no votes, no pay-to-list · just what's real
Debriefing pricing
starter$80/mo · monthly · $960/yr
free tierNo standing free plan, but the first debrief is free and arrives in 5 to 10 minutes.
verified 2026-08-10 · source ↗
Is Debriefing free?
No standing free plan, but the first debrief is free and arrives in 5 to 10 minutes. Paid is Starter at $80/mo (checked 2026-08-10).
Vibecode Debriefing
Kinda. The core of Debriefing is buildable in a weekend with the prompt on this page, but there are real gaps: noise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves, horizon watch, meaning the substitutes and new entrants you did not think to add to the list. Read the honest list above before committing.
How much does Debriefing cost?
Debriefing costs about $80/month (Starter, checked 2026-08-10), which is $960 per year.
What do I lose by replacing Debriefing?
Honestly: noise filtering, which is the actual product: most diffs are rotated testimonials and changed asset hashes, not competitive moves; horizon watch, meaning the substitutes and new entrants you did not think to add to the list; the non-page signals stitched into the same brief: hiring, funding, traffic and LinkedIn movement; the analytical step from what changed to why it matters to what to do next; an archive going back far enough that a change reads as a trend rather than an event. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Debriefing?
Yes: changedetection.io (Tells you a competitor's pricing page moved. Deciding whether that mattered is back to being your job.) The prompt is for when you want it exactly your way.