Vibecode Ideabrowser

You are building a lean indie version of Ideabrowser.

Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt.

===== README.md =====
# Ideabrowser · indie build

A local scanner whose hard problems are collecting from sources at the rate they permit and keeping the raw evidence, recognising the same complaint reposted as one signal rather than ten, and producing briefs where every claim quotes a source and an unknown is allowed to stay unknown.

Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts.

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | Next.js with TypeScript and SQLite, single user on the loopback interface | a scanner is a tool you run, not a service anybody logs into |
| Sources | A registry where each source records its terms and its published rate limit | a source whose rules you cannot state is one you should not be using |
| Cache | Raw payloads stored with their fetch time and reused rather than re-requested | re-fetching the same page to read it twice is how access gets withdrawn |
| Briefs | One model call per candidate, every claim carrying a quote from the source | an unsourced brief is a sentence you could have written yourself |
| Cost | An estimate before the run and a hard ceiling during it | the cost is per candidate and the candidate count is whatever the feed did today |

## Before you start

Have every one of these ready. The plan assumes them from step one.

- [ ] **Node.js 22 or newer** · free
  - Why: Runs the application and the collection script.
  - Get it: nodejs.org, or nvm install 22.
  - Verify: node --version prints v22 or higher
- [ ] **A source list with a terms note for each** · free
  - Why: The registry refuses to enable a source without one, and writing the note is the step that makes you check.
  - Get it: For each source, read its published terms and rate limits and write one sentence on what you may do and how often.
  - Verify: Every enabled source has both notes recorded and the interval configured
- [ ] **A model API key** · pennies per brief, multiplied by however many candidates a day produces
  - Why: Brief writing is the one part that needs a model, and the cost is per candidate.
  - Get it: Create a key in your provider's console and put it in .env.
  - Verify: One candidate produces a brief with verified quotes
- [ ] **A cost ceiling per run** · free
  - Why: The candidate count is decided by the internet rather than by you, so the limit has to be in code.
  - Get it: Pick a figure per run and put it in the environment.
  - Verify: A run configured to exceed it stops and reports what it completed
- [ ] **A subject you know well** · free
  - Why: The only way to judge whether the briefs are any good is to read the first batch about something whose reality you can check.
  - Get it: Point the first run at a community you belong to and read every brief it produces.
  - Verify: You can say which briefs are wrong and why

## Quick start

Follow Phase 1 in `BUILD_PLAN.md`; it creates the project.

Then copy `.env.example` to `.env` and fill in the values it documents.

## Honest limits

This build deliberately does not replace:

- Editorial curation.
- A historical trend database.
- Paid trend data feeds.
- Anything behind a login.
- 1,000+ already-curated and human-vetted ideas
- historical trend database and idea archive
- AI Research Agent for arbitrary custom ideas
- community, coaching and Q&A strategist sessions
- data exports and idea-suggest personalization

If one of those is essential to you, that is the reason to keep paying for Ideabrowser, and the README should say so rather than pretend.

===== BRIEF.md =====
# Build brief · Ideabrowser

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a local idea scanner that keeps the evidence and refuses to invent a market size to replace Ideabrowser. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
Two rules, and the first is about a number you will be very tempted to print.
No invented market sizing. A trend index is not a volume: Google Trends divides each data point
by the total searches for its geography and time range and then scales the result onto 0 to 100,
so the figure describes relative popularity and says nothing about how many people searched. A
model asked for a market size will produce a confident figure out of nothing at all. This tool
reports the signals it actually collected and writes 'unknown' where it has no source.
Second, every claim in a brief quotes the post it came from and links to it. An idea with no
evidence behind it is just a sentence, and you can write those faster than this can.
Collect only from sources whose published terms allow it, at the rate they publish, identifying
yourself honestly. No logged in scraping and no pretending to be somebody's browser.

### Stack (fixed, do not substitute)
- Runtime: Next.js with TypeScript and SQLite, single user on the loopback interface · a scanner is a tool you run, not a service anybody logs into
- Sources: A registry where each source records its terms and its published rate limit · a source whose rules you cannot state is one you should not be using
- Cache: Raw payloads stored with their fetch time and reused rather than re-requested · re-fetching the same page to read it twice is how access gets withdrawn
- Briefs: One model call per candidate, every claim carrying a quote from the source · an unsourced brief is a sentence you could have written yourself
- Cost: An estimate before the run and a hard ceiling during it · the cost is per candidate and the candidate count is whatever the feed did today

### Data model (create this before Phase 1)
- sources(id, name, endpoint, terms_note, rate_limit_note, user_agent, enabled) · a source with no terms note cannot be enabled
- fetches(id, source_id, url, requested_at, status, bytes, raw_path, from_cache) · raw payloads kept so a brief can be rechecked later
- candidates(id, source_id, external_id, url, author_handle, posted_at, title, text, normalised_hash, first_seen_at, seen_count)
- briefs(id, candidate_id, model_id, created_at, problem, target_user, mvp_scope, pricing_idea, market_size, market_size_source) · market size is a cited figure or the word unknown
- claims(brief_id, ordinal, text, quote, candidate_id) · every claim quotes the post it came from
- verdicts(brief_id, verdict, reason, decided_at) · your own judgement, recorded, because an unjudged backlog only grows
- runs(id, started_at, finished_at, fetch_count, candidates_new, briefs_written, estimated_cost, actual_cost)

### Phase 1 · Sources you are allowed to read, cached once
Build: A source registry where each entry records the endpoint, a note on its published terms, its
  published rate limit and the identifying user agent to send. Enforce a minimum interval between
  requests per source. Store every raw payload with its fetch time and serve repeats from cache.
Trap: Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
Done when: A source without a terms note and a rate limit note cannot be enabled, enforced in code. Two runs an hour apart make no duplicate requests for unchanged pages, verified by counting fetch rows. The identifying user agent is configurable, is sent on every request, and appears in the logs. Every response is written to the raw cache with its status, size and fetch time.
Do not build yet: No candidate extraction, briefs, verdicts or exports.

### Phase 2 · One complaint is one signal
Build: Normalise candidate text, hash it, and match near duplicates across sources and runs. Keep a
  first seen date and a seen count rather than a new row each time. Show recurrence as a count
  with dates, so a complaint made weekly for a year reads as exactly that.
Trap: Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
Done when: The same post collected from two sources produces one candidate with two source links, tested with a fixture. Reposted text with small edits matches the existing candidate under the normalisation rules, which are documented. The seen count and the first seen date are shown wherever a candidate appears. Deleting a duplicate never removes the original, asserted.
Do not build yet: No briefs, verdicts or exports yet.

### Phase 3 · Briefs that cite, or say unknown
Build: For each candidate, produce a brief with problem, target user, MVP scope and a pricing idea.
  Every claim stores a quote checked to appear in the candidate text. The market size field takes
  a cited figure with its source or the word unknown, and there is no third option.
Trap: A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
Done when: Every claim row has a quote confirmed present in the candidate text, asserted in the suite. A market size without a source is refused at write time and the field stores unknown, tested. A brief renders with its quotes and a link to the original post beside each claim. Claims that fail verification are counted and the count is visible rather than logged.
Do not build yet: No verdicts, search or exports yet.

### Phase 4 · A backlog you can actually get through
Build: A feed with search across briefs and candidates. Record a verdict per brief with a written
  reason: pursue, park or discard. Filter by verdict, by recurrence and by the age of the evidence.
  Show how many briefs are still unjudged and how old the oldest is.
Trap: An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.
Done when: A verdict cannot be recorded without a reason, enforced in the interface and the schema. The unjudged count and the oldest unjudged date are shown on the first screen. Search returns both briefs and the candidate text behind them. Filtering by evidence age works and surfaces briefs built on posts older than a stated threshold.
Do not build yet: No cost ceiling, retention or export yet.

### Phase 5 · What it cost, and what it is honestly worth
Build: Print a cost estimate before a run and stop hard at a ceiling. Export briefs, claims and
  verdicts as JSON and Markdown, with the source links intact. Apply a retention to raw payloads
  and report what it removed. Put the limitations on a page in the application, not only in the
  README.
Trap: A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.
Done when: A run stops at the cost ceiling and reports how many briefs it completed. The estimate is shown with its arithmetic before any model call. Exported briefs keep their quotes and source links and open as readable Markdown. The limitations page states plainly that the tool reports collected signals, that a trend index is relative rather than a volume, and that unknown means unknown.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Editorial curation, which is the actual product being replaced here: a vetted backlog built over years by people making judgement calls is not something a script produces on its first run.
- A historical trend database, because backfilling signals you did not collect is impossible and pretending otherwise is the fastest way to a misleading chart.
- Paid trend data feeds, which change the economics entirely and deserve a separate decision rather than an API key in the same .env file.
- Anything behind a login, which means accepting terms on somebody's behalf and breaks the moment a session expires.

### README must contain
- That a trend index is relative popularity rather than search volume, with the source and the date checked
- That market size is either a cited figure or the word unknown, and there is no third state
- Which sources are enabled, what their terms notes say and what rate they are polled at
- That raw payloads are cached and retained, and for how long
- The per run cost arithmetic and where the ceiling is enforced

===== AGENTS.md =====
# Agent instructions · Ideabrowser indie build

- Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Next.js with TypeScript and SQLite, single user on the loopback interface, A registry where each source records its terms and its published rate limit, Raw payloads stored with their fetch time and reused rather than re-requested, One model call per candidate, every claim carrying a quote from the source, An estimate before the run and a hard ceiling during it. Do not substitute.
- Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes.
- Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name.
- Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced.
- Do not invent cryptography, security guarantees, APIs or compliance claims.
- Add a focused test for every destructive, security-sensitive or data-loss path the plan names.
- Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs".

## Known traps

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.

===== BUILD_PLAN.md =====
# Build plan · Ideabrowser

A local scanner whose hard problems are collecting from sources at the rate they permit and keeping the raw evidence, recognising the same complaint reposted as one signal rather than ten, and producing briefs where every claim quotes a source and an unknown is allowed to stay unknown.

Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note.

## Phase 1 · Sources you are allowed to read, cached once

Every fetch is permitted, rate limited, identified and kept.

### Steps

1. Make the terms note a required field
   If you cannot write the sentence, you have not read the terms.
   Files: `src/sources.ts`
2. Enforce the interval in the fetch client
   One place, so no code path can accidentally skip it.
   Files: `src/fetch.ts`, `tests/rate-limit.test.ts`
3. Keep every raw payload
   It is the evidence, and re-fetching it later may not be possible.
   Files: `src/cache.ts`

### Done when

- [ ] Sources without terms and rate notes cannot be enabled
- [ ] Repeat runs do not re-request unchanged pages
- [ ] An honest user agent is sent on every request
- [ ] Raw payloads are stored with status and time

### Watch out

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.

## Phase 2 · One complaint is one signal

The same thing said repeatedly is recognised as recurrence, not as volume.

### Steps

1. Normalise before hashing, and write down how
   Case, whitespace, punctuation and links, stated in the documentation.
   Files: `src/dedupe.ts`, `docs/normalisation.md`
2. Increment rather than insert
   Recurrence is the signal, and a row count is not.
   Files: `src/candidates.ts`
3. Show first seen and seen count everywhere
   Two dates and a number make a feed honest.
   Files: `src/components/Candidate.tsx`

### Done when

- [ ] One post from two sources is one candidate
- [ ] Near duplicates match under documented rules
- [ ] First seen and seen count are always visible
- [ ] Removing a duplicate leaves the original

### Watch out

- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.

## Phase 3 · Briefs that cite, or say unknown

No sentence in a brief exists without a quote behind it.

### Steps

1. Verify each quote against the candidate
   A substring check, run before the row is written.
   Files: `src/briefs/verify.ts`, `tests/quote-check.test.ts`
2. Make market size a citation or unknown
   Enforced by the column and by validation, not by the prompt.
   Files: `src/briefs/schema.ts`
3. Render the quote beside every claim
   So reading the brief is reading the evidence.
   Files: `src/components/Brief.tsx`

### Done when

- [ ] Every claim carries a verified quote
- [ ] Market size is a citation or the word unknown
- [ ] Quotes and source links render with each claim
- [ ] Failed verifications are counted and shown

### Watch out

- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.

## Phase 4 · A backlog you can actually get through

Every brief ends up judged rather than scrolled past.

### Steps

1. Require a reason with every verdict
   One sentence, and the backlog starts shrinking.
   Files: `src/verdicts.ts`
2. Put the unjudged count on the first screen
   A queue you can see the length of is a queue you finish.
   Files: `src/app/page.tsx`
3. Filter by evidence age
   A brief built on a two year old post is a different thing.
   Files: `src/search.ts`

### Done when

- [ ] Verdicts require a written reason
- [ ] The unjudged count and oldest date are visible
- [ ] Search covers briefs and candidate text
- [ ] Evidence age filtering works

### Watch out

- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.

## Not in this build

- Editorial curation.
- A historical trend database.
- Paid trend data feeds.
- Anything behind a login.

## After v1, if you want it

- A weekly digest of new candidates with high recurrence, which is a query and a Markdown file
- A second source in a different community, once the deduplication rules have proven themselves
- Tracking which pursued ideas went anywhere, which is the only feedback loop that would improve the briefs

===== .env.example =====
# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sources, candidates, briefs and verdicts.
DATABASE_PATH=./data/ideas.db

# Required. Raw payloads with their fetch times. The evidence behind every brief.
RAW_CACHE_DIR=./data/raw

# Required. Endpoints with their terms and rate limit notes.
SOURCES_FILE=./sources.json

# Required. Sent on every request. Identify yourself honestly.
USER_AGENT=idea-scanner/0.1 (contact: you@example.com)

# Required. Minimum gap between requests to one source, enforced in the client.
FETCH_MIN_INTERVAL_MS=2000

# Required. Per run, in your billing currency. Enforced, not remembered.
COST_CEILING=5.00

# Required. How long payloads are kept. The cleanup reports what it removed.
RAW_RETENTION_DAYS=180

# Required · secret. Brief writing. Without it the tool still collects and deduplicates.
ANTHROPIC_API_KEY=sk-ant-...redacted

You are building a lean indie version of Ideabrowser.

Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt.

===== README.md =====
# Ideabrowser · indie build

A local scanner whose hard problems are collecting from sources at the rate they permit and keeping the raw evidence, recognising the same complaint reposted as one signal rather than ten, and producing briefs where every claim quotes a source and an unknown is allowed to stay unknown.

Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts.

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | Next.js with TypeScript and SQLite, single user on the loopback interface | a scanner is a tool you run, not a service anybody logs into |
| Sources | A registry where each source records its terms and its published rate limit | a source whose rules you cannot state is one you should not be using |
| Cache | Raw payloads stored with their fetch time and reused rather than re-requested | re-fetching the same page to read it twice is how access gets withdrawn |
| Briefs | One model call per candidate, every claim carrying a quote from the source | an unsourced brief is a sentence you could have written yourself |
| Cost | An estimate before the run and a hard ceiling during it | the cost is per candidate and the candidate count is whatever the feed did today |

## Before you start

Have every one of these ready. The plan assumes them from step one.

- [ ] **Node.js 22 or newer** · free
  - Why: Runs the application and the collection script.
  - Get it: nodejs.org, or nvm install 22.
  - Verify: node --version prints v22 or higher
- [ ] **A source list with a terms note for each** · free
  - Why: The registry refuses to enable a source without one, and writing the note is the step that makes you check.
  - Get it: For each source, read its published terms and rate limits and write one sentence on what you may do and how often.
  - Verify: Every enabled source has both notes recorded and the interval configured
- [ ] **A model API key** · pennies per brief, multiplied by however many candidates a day produces
  - Why: Brief writing is the one part that needs a model, and the cost is per candidate.
  - Get it: Create a key in your provider's console and put it in .env.
  - Verify: One candidate produces a brief with verified quotes
- [ ] **A cost ceiling per run** · free
  - Why: The candidate count is decided by the internet rather than by you, so the limit has to be in code.
  - Get it: Pick a figure per run and put it in the environment.
  - Verify: A run configured to exceed it stops and reports what it completed
- [ ] **A subject you know well** · free
  - Why: The only way to judge whether the briefs are any good is to read the first batch about something whose reality you can check.
  - Get it: Point the first run at a community you belong to and read every brief it produces.
  - Verify: You can say which briefs are wrong and why

## Quick start

Follow Phase 1 in `BUILD_PLAN.md`; it creates the project.

Then copy `.env.example` to `.env` and fill in the values it documents.

## Honest limits

This build deliberately does not replace:

- Editorial curation.
- A historical trend database.
- Paid trend data feeds.
- Anything behind a login.
- 1,000+ already-curated and human-vetted ideas
- historical trend database and idea archive
- AI Research Agent for arbitrary custom ideas
- community, coaching and Q&A strategist sessions
- data exports and idea-suggest personalization

If one of those is essential to you, that is the reason to keep paying for Ideabrowser, and the README should say so rather than pretend.

===== BRIEF.md =====
# Build brief · Ideabrowser

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a local idea scanner that keeps the evidence and refuses to invent a market size to replace Ideabrowser. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
Two rules, and the first is about a number you will be very tempted to print.
No invented market sizing. A trend index is not a volume: Google Trends divides each data point
by the total searches for its geography and time range and then scales the result onto 0 to 100,
so the figure describes relative popularity and says nothing about how many people searched. A
model asked for a market size will produce a confident figure out of nothing at all. This tool
reports the signals it actually collected and writes 'unknown' where it has no source.
Second, every claim in a brief quotes the post it came from and links to it. An idea with no
evidence behind it is just a sentence, and you can write those faster than this can.
Collect only from sources whose published terms allow it, at the rate they publish, identifying
yourself honestly. No logged in scraping and no pretending to be somebody's browser.

### Stack (fixed, do not substitute)
- Runtime: Next.js with TypeScript and SQLite, single user on the loopback interface · a scanner is a tool you run, not a service anybody logs into
- Sources: A registry where each source records its terms and its published rate limit · a source whose rules you cannot state is one you should not be using
- Cache: Raw payloads stored with their fetch time and reused rather than re-requested · re-fetching the same page to read it twice is how access gets withdrawn
- Briefs: One model call per candidate, every claim carrying a quote from the source · an unsourced brief is a sentence you could have written yourself
- Cost: An estimate before the run and a hard ceiling during it · the cost is per candidate and the candidate count is whatever the feed did today

### Data model (create this before Phase 1)
- sources(id, name, endpoint, terms_note, rate_limit_note, user_agent, enabled) · a source with no terms note cannot be enabled
- fetches(id, source_id, url, requested_at, status, bytes, raw_path, from_cache) · raw payloads kept so a brief can be rechecked later
- candidates(id, source_id, external_id, url, author_handle, posted_at, title, text, normalised_hash, first_seen_at, seen_count)
- briefs(id, candidate_id, model_id, created_at, problem, target_user, mvp_scope, pricing_idea, market_size, market_size_source) · market size is a cited figure or the word unknown
- claims(brief_id, ordinal, text, quote, candidate_id) · every claim quotes the post it came from
- verdicts(brief_id, verdict, reason, decided_at) · your own judgement, recorded, because an unjudged backlog only grows
- runs(id, started_at, finished_at, fetch_count, candidates_new, briefs_written, estimated_cost, actual_cost)

### Phase 1 · Sources you are allowed to read, cached once
Build: A source registry where each entry records the endpoint, a note on its published terms, its
  published rate limit and the identifying user agent to send. Enforce a minimum interval between
  requests per source. Store every raw payload with its fetch time and serve repeats from cache.
Trap: Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
Done when: A source without a terms note and a rate limit note cannot be enabled, enforced in code. Two runs an hour apart make no duplicate requests for unchanged pages, verified by counting fetch rows. The identifying user agent is configurable, is sent on every request, and appears in the logs. Every response is written to the raw cache with its status, size and fetch time.
Do not build yet: No candidate extraction, briefs, verdicts or exports.

### Phase 2 · One complaint is one signal
Build: Normalise candidate text, hash it, and match near duplicates across sources and runs. Keep a
  first seen date and a seen count rather than a new row each time. Show recurrence as a count
  with dates, so a complaint made weekly for a year reads as exactly that.
Trap: Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
Done when: The same post collected from two sources produces one candidate with two source links, tested with a fixture. Reposted text with small edits matches the existing candidate under the normalisation rules, which are documented. The seen count and the first seen date are shown wherever a candidate appears. Deleting a duplicate never removes the original, asserted.
Do not build yet: No briefs, verdicts or exports yet.

### Phase 3 · Briefs that cite, or say unknown
Build: For each candidate, produce a brief with problem, target user, MVP scope and a pricing idea.
  Every claim stores a quote checked to appear in the candidate text. The market size field takes
  a cited figure with its source or the word unknown, and there is no third option.
Trap: A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
Done when: Every claim row has a quote confirmed present in the candidate text, asserted in the suite. A market size without a source is refused at write time and the field stores unknown, tested. A brief renders with its quotes and a link to the original post beside each claim. Claims that fail verification are counted and the count is visible rather than logged.
Do not build yet: No verdicts, search or exports yet.

### Phase 4 · A backlog you can actually get through
Build: A feed with search across briefs and candidates. Record a verdict per brief with a written
  reason: pursue, park or discard. Filter by verdict, by recurrence and by the age of the evidence.
  Show how many briefs are still unjudged and how old the oldest is.
Trap: An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.
Done when: A verdict cannot be recorded without a reason, enforced in the interface and the schema. The unjudged count and the oldest unjudged date are shown on the first screen. Search returns both briefs and the candidate text behind them. Filtering by evidence age works and surfaces briefs built on posts older than a stated threshold.
Do not build yet: No cost ceiling, retention or export yet.

### Phase 5 · What it cost, and what it is honestly worth
Build: Print a cost estimate before a run and stop hard at a ceiling. Export briefs, claims and
  verdicts as JSON and Markdown, with the source links intact. Apply a retention to raw payloads
  and report what it removed. Put the limitations on a page in the application, not only in the
  README.
Trap: A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.
Done when: A run stops at the cost ceiling and reports how many briefs it completed. The estimate is shown with its arithmetic before any model call. Exported briefs keep their quotes and source links and open as readable Markdown. The limitations page states plainly that the tool reports collected signals, that a trend index is relative rather than a volume, and that unknown means unknown.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Editorial curation, which is the actual product being replaced here: a vetted backlog built over years by people making judgement calls is not something a script produces on its first run.
- A historical trend database, because backfilling signals you did not collect is impossible and pretending otherwise is the fastest way to a misleading chart.
- Paid trend data feeds, which change the economics entirely and deserve a separate decision rather than an API key in the same .env file.
- Anything behind a login, which means accepting terms on somebody's behalf and breaks the moment a session expires.

### README must contain
- That a trend index is relative popularity rather than search volume, with the source and the date checked
- That market size is either a cited figure or the word unknown, and there is no third state
- Which sources are enabled, what their terms notes say and what rate they are polled at
- That raw payloads are cached and retained, and for how long
- The per run cost arithmetic and where the ceiling is enforced

===== AGENTS.md =====
# Agent instructions · Ideabrowser indie build

- Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Next.js with TypeScript and SQLite, single user on the loopback interface, A registry where each source records its terms and its published rate limit, Raw payloads stored with their fetch time and reused rather than re-requested, One model call per candidate, every claim carrying a quote from the source, An estimate before the run and a hard ceiling during it. Do not substitute.
- Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes.
- Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name.
- Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced.
- Do not invent cryptography, security guarantees, APIs or compliance claims.
- Add a focused test for every destructive, security-sensitive or data-loss path the plan names.
- Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs".

## Known traps

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.

===== BUILD_PLAN.md =====
# Build plan · Ideabrowser

A local scanner whose hard problems are collecting from sources at the rate they permit and keeping the raw evidence, recognising the same complaint reposted as one signal rather than ten, and producing briefs where every claim quotes a source and an unknown is allowed to stay unknown.

Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note.

## Phase 1 · Sources you are allowed to read, cached once

Every fetch is permitted, rate limited, identified and kept.

### Steps

1. Make the terms note a required field
   If you cannot write the sentence, you have not read the terms.
   Files: `src/sources.ts`
2. Enforce the interval in the fetch client
   One place, so no code path can accidentally skip it.
   Files: `src/fetch.ts`, `tests/rate-limit.test.ts`
3. Keep every raw payload
   It is the evidence, and re-fetching it later may not be possible.
   Files: `src/cache.ts`

### Done when

- [ ] Sources without terms and rate notes cannot be enabled
- [ ] Repeat runs do not re-request unchanged pages
- [ ] An honest user agent is sent on every request
- [ ] Raw payloads are stored with status and time

### Watch out

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.

## Phase 2 · One complaint is one signal

The same thing said repeatedly is recognised as recurrence, not as volume.

### Steps

1. Normalise before hashing, and write down how
   Case, whitespace, punctuation and links, stated in the documentation.
   Files: `src/dedupe.ts`, `docs/normalisation.md`
2. Increment rather than insert
   Recurrence is the signal, and a row count is not.
   Files: `src/candidates.ts`
3. Show first seen and seen count everywhere
   Two dates and a number make a feed honest.
   Files: `src/components/Candidate.tsx`

### Done when

- [ ] One post from two sources is one candidate
- [ ] Near duplicates match under documented rules
- [ ] First seen and seen count are always visible
- [ ] Removing a duplicate leaves the original

### Watch out

- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.

## Phase 3 · Briefs that cite, or say unknown

No sentence in a brief exists without a quote behind it.

### Steps

1. Verify each quote against the candidate
   A substring check, run before the row is written.
   Files: `src/briefs/verify.ts`, `tests/quote-check.test.ts`
2. Make market size a citation or unknown
   Enforced by the column and by validation, not by the prompt.
   Files: `src/briefs/schema.ts`
3. Render the quote beside every claim
   So reading the brief is reading the evidence.
   Files: `src/components/Brief.tsx`

### Done when

- [ ] Every claim carries a verified quote
- [ ] Market size is a citation or the word unknown
- [ ] Quotes and source links render with each claim
- [ ] Failed verifications are counted and shown

### Watch out

- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.

## Phase 4 · A backlog you can actually get through

Every brief ends up judged rather than scrolled past.

### Steps

1. Require a reason with every verdict
   One sentence, and the backlog starts shrinking.
   Files: `src/verdicts.ts`
2. Put the unjudged count on the first screen
   A queue you can see the length of is a queue you finish.
   Files: `src/app/page.tsx`
3. Filter by evidence age
   A brief built on a two year old post is a different thing.
   Files: `src/search.ts`

### Done when

- [ ] Verdicts require a written reason
- [ ] The unjudged count and oldest date are visible
- [ ] Search covers briefs and candidate text
- [ ] Evidence age filtering works

### Watch out

- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.

## Not in this build

- Editorial curation.
- A historical trend database.
- Paid trend data feeds.
- Anything behind a login.

## After v1, if you want it

- A weekly digest of new candidates with high recurrence, which is a query and a Markdown file
- A second source in a different community, once the deduplication rules have proven themselves
- Tracking which pursued ideas went anywhere, which is the only feedback loop that would improve the briefs

===== .env.example =====
# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sources, candidates, briefs and verdicts.
DATABASE_PATH=./data/ideas.db

# Required. Raw payloads with their fetch times. The evidence behind every brief.
RAW_CACHE_DIR=./data/raw

# Required. Endpoints with their terms and rate limit notes.
SOURCES_FILE=./sources.json

# Required. Sent on every request. Identify yourself honestly.
USER_AGENT=idea-scanner/0.1 (contact: you@example.com)

# Required. Minimum gap between requests to one source, enforced in the client.
FETCH_MIN_INTERVAL_MS=2000

# Required. Per run, in your billing currency. Enforced, not remembered.
COST_CEILING=5.00

# Required. How long payloads are kept. The cleanup reports what it removed.
RAW_RETENTION_DAYS=180

# Required · secret. Brief writing. Without it the tool still collects and deduplicates.
ANTHROPIC_API_KEY=sk-ant-...redacted

You are building a production product version of Ideabrowser.

Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt.

===== PRODUCT.md =====
# Ideabrowser · product brief

## Problem

Scraping Reddit/search trends and having an LLM draft a business-opportunity writeup per idea is a weekend build. What's genuinely hard to replicate is the editorial curation, the accumulated database of 1,000+ pre-researched ideas, and the trend-signal pipeline built up over time. A DIY version can generate plausible-sounding ideas on demand but starts from zero data and zero curation.

## Product outcome

A scanner that reads only what it is allowed to, keeps the raw evidence, counts a repeated complaint once, writes briefs whose every claim quotes a real post, and prints unknown where it does not know.

## Target user

A builder who needs a maintainable product foundation, not a one-off demo.

## Required capabilities

- OpenAI/Anthropic API key
- Reddit or Google Trends API access

## Explicit non-goals for v1

- Editorial curation.
- A historical trend database.
- Paid trend data feeds.
- Anything behind a login.
- 1,000+ already-curated and human-vetted ideas
- historical trend database and idea archive
- AI Research Agent for arbitrary custom ideas
- community, coaching and Q&A strategist sessions
- data exports and idea-suggest personalization

## Success criteria

- Sources are confirmed unenablable without terms and rate notes
- Repeat runs are confirmed not to re-request unchanged pages
- Every claim is confirmed to carry a verified quote
- Market size is confirmed to be a citation or unknown
- The cost ceiling is confirmed to stop a run

===== BRIEF.md =====
# Build brief · Ideabrowser

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a local idea scanner that keeps the evidence and refuses to invent a market size to replace Ideabrowser. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
Two rules, and the first is about a number you will be very tempted to print.
No invented market sizing. A trend index is not a volume: Google Trends divides each data point
by the total searches for its geography and time range and then scales the result onto 0 to 100,
so the figure describes relative popularity and says nothing about how many people searched. A
model asked for a market size will produce a confident figure out of nothing at all. This tool
reports the signals it actually collected and writes 'unknown' where it has no source.
Second, every claim in a brief quotes the post it came from and links to it. An idea with no
evidence behind it is just a sentence, and you can write those faster than this can.
Collect only from sources whose published terms allow it, at the rate they publish, identifying
yourself honestly. No logged in scraping and no pretending to be somebody's browser.

### Stack (fixed, do not substitute)
- Runtime: Next.js with TypeScript and SQLite, single user on the loopback interface · a scanner is a tool you run, not a service anybody logs into
- Sources: A registry where each source records its terms and its published rate limit · a source whose rules you cannot state is one you should not be using
- Cache: Raw payloads stored with their fetch time and reused rather than re-requested · re-fetching the same page to read it twice is how access gets withdrawn
- Briefs: One model call per candidate, every claim carrying a quote from the source · an unsourced brief is a sentence you could have written yourself
- Cost: An estimate before the run and a hard ceiling during it · the cost is per candidate and the candidate count is whatever the feed did today

### Data model (create this before Phase 1)
- sources(id, name, endpoint, terms_note, rate_limit_note, user_agent, enabled) · a source with no terms note cannot be enabled
- fetches(id, source_id, url, requested_at, status, bytes, raw_path, from_cache) · raw payloads kept so a brief can be rechecked later
- candidates(id, source_id, external_id, url, author_handle, posted_at, title, text, normalised_hash, first_seen_at, seen_count)
- briefs(id, candidate_id, model_id, created_at, problem, target_user, mvp_scope, pricing_idea, market_size, market_size_source) · market size is a cited figure or the word unknown
- claims(brief_id, ordinal, text, quote, candidate_id) · every claim quotes the post it came from
- verdicts(brief_id, verdict, reason, decided_at) · your own judgement, recorded, because an unjudged backlog only grows
- runs(id, started_at, finished_at, fetch_count, candidates_new, briefs_written, estimated_cost, actual_cost)

### Phase 1 · Sources you are allowed to read, cached once
Build: A source registry where each entry records the endpoint, a note on its published terms, its
  published rate limit and the identifying user agent to send. Enforce a minimum interval between
  requests per source. Store every raw payload with its fetch time and serve repeats from cache.
Trap: Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
Done when: A source without a terms note and a rate limit note cannot be enabled, enforced in code. Two runs an hour apart make no duplicate requests for unchanged pages, verified by counting fetch rows. The identifying user agent is configurable, is sent on every request, and appears in the logs. Every response is written to the raw cache with its status, size and fetch time.
Do not build yet: No candidate extraction, briefs, verdicts or exports.

### Phase 2 · One complaint is one signal
Build: Normalise candidate text, hash it, and match near duplicates across sources and runs. Keep a
  first seen date and a seen count rather than a new row each time. Show recurrence as a count
  with dates, so a complaint made weekly for a year reads as exactly that.
Trap: Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
Done when: The same post collected from two sources produces one candidate with two source links, tested with a fixture. Reposted text with small edits matches the existing candidate under the normalisation rules, which are documented. The seen count and the first seen date are shown wherever a candidate appears. Deleting a duplicate never removes the original, asserted.
Do not build yet: No briefs, verdicts or exports yet.

### Phase 3 · Briefs that cite, or say unknown
Build: For each candidate, produce a brief with problem, target user, MVP scope and a pricing idea.
  Every claim stores a quote checked to appear in the candidate text. The market size field takes
  a cited figure with its source or the word unknown, and there is no third option.
Trap: A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
Done when: Every claim row has a quote confirmed present in the candidate text, asserted in the suite. A market size without a source is refused at write time and the field stores unknown, tested. A brief renders with its quotes and a link to the original post beside each claim. Claims that fail verification are counted and the count is visible rather than logged.
Do not build yet: No verdicts, search or exports yet.

### Phase 4 · A backlog you can actually get through
Build: A feed with search across briefs and candidates. Record a verdict per brief with a written
  reason: pursue, park or discard. Filter by verdict, by recurrence and by the age of the evidence.
  Show how many briefs are still unjudged and how old the oldest is.
Trap: An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.
Done when: A verdict cannot be recorded without a reason, enforced in the interface and the schema. The unjudged count and the oldest unjudged date are shown on the first screen. Search returns both briefs and the candidate text behind them. Filtering by evidence age works and surfaces briefs built on posts older than a stated threshold.
Do not build yet: No cost ceiling, retention or export yet.

### Phase 5 · What it cost, and what it is honestly worth
Build: Print a cost estimate before a run and stop hard at a ceiling. Export briefs, claims and
  verdicts as JSON and Markdown, with the source links intact. Apply a retention to raw payloads
  and report what it removed. Put the limitations on a page in the application, not only in the
  README.
Trap: A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.
Done when: A run stops at the cost ceiling and reports how many briefs it completed. The estimate is shown with its arithmetic before any model call. Exported briefs keep their quotes and source links and open as readable Markdown. The limitations page states plainly that the tool reports collected signals, that a trend index is relative rather than a volume, and that unknown means unknown.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Editorial curation, which is the actual product being replaced here: a vetted backlog built over years by people making judgement calls is not something a script produces on its first run.
- A historical trend database, because backfilling signals you did not collect is impossible and pretending otherwise is the fastest way to a misleading chart.
- Paid trend data feeds, which change the economics entirely and deserve a separate decision rather than an API key in the same .env file.
- Anything behind a login, which means accepting terms on somebody's behalf and breaks the moment a session expires.

### README must contain
- That a trend index is relative popularity rather than search volume, with the source and the date checked
- That market size is either a cited figure or the word unknown, and there is no third state
- Which sources are enabled, what their terms notes say and what rate they are polled at
- That raw payloads are cached and retained, and for how long
- The per run cost arithmetic and where the ceiling is enforced

===== ARCHITECTURE.md =====
# Architecture · Ideabrowser

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | Next.js with TypeScript and SQLite, single user on the loopback interface | a scanner is a tool you run, not a service anybody logs into |
| Sources | A registry where each source records its terms and its published rate limit | a source whose rules you cannot state is one you should not be using |
| Cache | Raw payloads stored with their fetch time and reused rather than re-requested | re-fetching the same page to read it twice is how access gets withdrawn |
| Briefs | One model call per candidate, every claim carrying a quote from the source | an unsourced brief is a sentence you could have written yourself |
| Cost | An estimate before the run and a hard ceiling during it | the cost is per candidate and the candidate count is whatever the feed did today |

## Modules

Each module has one owner concern and a documented way to replace it.

| Module | Owns | How to replace it |
| --- | --- | --- |
| Sources | The registry, terms notes and intervals. | Any source with published terms, added as a registry row. |
| Fetch | Rate limiting, caching and raw storage. | Any client, provided the interval stays inside it. |
| Dedupe | Normalisation, hashing and recurrence counts. | Any matching rule, provided it is documented. |
| Briefs | Claims with verified quotes and the unknown rule. | Any model, provided verification stays in front of the write. |
| Backlog | Verdicts with reasons and the unjudged count. | Any interface that still requires a reason. |

## Configuration

Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists.

- `DATABASE_PATH` · required · Sources, candidates, briefs and verdicts.
- `RAW_CACHE_DIR` · required · Raw payloads with their fetch times. The evidence behind every brief.
- `SOURCES_FILE` · required · Endpoints with their terms and rate limit notes.
- `USER_AGENT` · required · Sent on every request. Identify yourself honestly.
- `FETCH_MIN_INTERVAL_MS` · required · Minimum gap between requests to one source, enforced in the client.
- `COST_CEILING` · required · Per run, in your billing currency. Enforced, not remembered.
- `RAW_RETENTION_DAYS` · required · How long payloads are kept. The cleanup reports what it removed.
- `ANTHROPIC_API_KEY` · required, secret · Brief writing. Without it the tool still collects and deduplicates.

## Production baseline

- Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives.
- Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised.
- Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts.
- Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists.
- Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path.

## Decision records

For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.

===== AGENTS.md =====
# Agent instructions · Ideabrowser product build

- Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Next.js with TypeScript and SQLite, single user on the loopback interface, A registry where each source records its terms and its published rate limit, Raw payloads stored with their fetch time and reused rather than re-requested, One model call per candidate, every claim carrying a quote from the source, An estimate before the run and a hard ceiling during it.
- Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit.
- Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present.
- Never invent cryptography or silently weaken a requirement to make a check pass.
- Put every external service behind an interface with a deterministic fake for tests.
- Add migrations and rollback or recovery notes for every persistent data change.
- Log useful operational context without credentials, tokens, passwords or personal data.
- Update documentation and run every check before completing a milestone.

## Known traps

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.
- A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.

===== MILESTONES.md =====
# Delivery milestones · Ideabrowser

Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer.

## M1 · Sources you are allowed to read, cached once

Every fetch is permitted, rate limited, identified and kept.

### Steps

1. Make the terms note a required field
   If you cannot write the sentence, you have not read the terms.
   Files: `src/sources.ts`
2. Enforce the interval in the fetch client
   One place, so no code path can accidentally skip it.
   Files: `src/fetch.ts`, `tests/rate-limit.test.ts`
3. Keep every raw payload
   It is the evidence, and re-fetching it later may not be possible.
   Files: `src/cache.ts`

### Done when

- [ ] Sources without terms and rate notes cannot be enabled
- [ ] Repeat runs do not re-request unchanged pages
- [ ] An honest user agent is sent on every request
- [ ] Raw payloads are stored with status and time

### Watch out

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.

## M2 · One complaint is one signal

The same thing said repeatedly is recognised as recurrence, not as volume.

### Steps

1. Normalise before hashing, and write down how
   Case, whitespace, punctuation and links, stated in the documentation.
   Files: `src/dedupe.ts`, `docs/normalisation.md`
2. Increment rather than insert
   Recurrence is the signal, and a row count is not.
   Files: `src/candidates.ts`
3. Show first seen and seen count everywhere
   Two dates and a number make a feed honest.
   Files: `src/components/Candidate.tsx`

### Done when

- [ ] One post from two sources is one candidate
- [ ] Near duplicates match under documented rules
- [ ] First seen and seen count are always visible
- [ ] Removing a duplicate leaves the original

### Watch out

- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.

## M3 · Briefs that cite, or say unknown

No sentence in a brief exists without a quote behind it.

### Steps

1. Verify each quote against the candidate
   A substring check, run before the row is written.
   Files: `src/briefs/verify.ts`, `tests/quote-check.test.ts`
2. Make market size a citation or unknown
   Enforced by the column and by validation, not by the prompt.
   Files: `src/briefs/schema.ts`
3. Render the quote beside every claim
   So reading the brief is reading the evidence.
   Files: `src/components/Brief.tsx`

### Done when

- [ ] Every claim carries a verified quote
- [ ] Market size is a citation or the word unknown
- [ ] Quotes and source links render with each claim
- [ ] Failed verifications are counted and shown

### Watch out

- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.

## M4 · A backlog you can actually get through

Every brief ends up judged rather than scrolled past.

### Steps

1. Require a reason with every verdict
   One sentence, and the backlog starts shrinking.
   Files: `src/verdicts.ts`
2. Put the unjudged count on the first screen
   A queue you can see the length of is a queue you finish.
   Files: `src/app/page.tsx`
3. Filter by evidence age
   A brief built on a two year old post is a different thing.
   Files: `src/search.ts`

### Done when

- [ ] Verdicts require a written reason
- [ ] The unjudged count and oldest date are visible
- [ ] Search covers briefs and candidate text
- [ ] Evidence age filtering works

### Watch out

- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.

## M5 · What it cost, and what it is honestly worth (production only)

Spending is bounded, evidence has a retention, and the tool states its own limits.

### Steps

1. Estimate from the candidate count, then confirm
   The multiplication before the money, every time.
   Files: `src/cost.ts`
2. Export with quotes and links intact
   A brief without its evidence is worth nothing outside the tool.
   Files: `src/export.ts`
3. Put the limits in the product
   In the interface, where somebody reading a brief will see them.
   Files: `src/app/limits/page.tsx`

### Done when

- [ ] The ceiling stops the run and reports progress
- [ ] The estimate is shown before any model call
- [ ] Exports keep quotes and source links
- [ ] The limitations page exists in the application

### Watch out

- A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.

===== OPERATIONS.md =====
# Operations · Ideabrowser

## Backup

The database and the raw cache together, since a brief whose payload has been deleted can no longer be checked against its source. The verdicts are the part you cannot regenerate at any price.

## Restore

Restore both, then open three briefs and confirm their quotes still resolve against the cached payloads. Quotes that no longer verify mean the cache was left out of the backup.

Do a restore drill before the first real user, and write the date here when it passes.

## Monitoring

Fetch counts per source against the interval, the quote verification failure rate, spend against the estimate, and the age of the oldest unjudged brief. A rising verification failure rate usually means a model change.

## Incident checklist

The incident worth naming is a number leaving the tool without a source, because it will be repeated. The unknown rule and the claim verification both exist to make that hard rather than discouraged.

1. Contain the issue without destroying evidence or user data.
2. Record the timeline and affected scope.
3. Rotate exposed secrets and revoke compromised sessions or credentials.
4. Restore from a verified backup when needed.
5. Document the root cause, the remediation and the regression test.

## Release gate

- [ ] Sources are confirmed unenablable without terms and rate notes
- [ ] Repeat runs are confirmed not to re-request unchanged pages
- [ ] Every claim is confirmed to carry a verified quote
- [ ] Market size is confirmed to be a citation or unknown
- [ ] The cost ceiling is confirmed to stop a run

## Launch constraint

Do not market omitted Ideabrowser capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.

===== .env.example =====
# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sources, candidates, briefs and verdicts.
DATABASE_PATH=./data/ideas.db

# Required. Raw payloads with their fetch times. The evidence behind every brief.
RAW_CACHE_DIR=./data/raw

# Required. Endpoints with their terms and rate limit notes.
SOURCES_FILE=./sources.json

# Required. Sent on every request. Identify yourself honestly.
USER_AGENT=idea-scanner/0.1 (contact: you@example.com)

# Required. Minimum gap between requests to one source, enforced in the client.
FETCH_MIN_INTERVAL_MS=2000

# Required. Per run, in your billing currency. Enforced, not remembered.
COST_CEILING=5.00

# Required. How long payloads are kept. The cleanup reports what it removed.
RAW_RETENTION_DAYS=180

# Required · secret. Brief writing. Without it the tool still collects and deduplicates.
ANTHROPIC_API_KEY=sk-ant-...redacted

# Ideabrowser · indie build

A local scanner whose hard problems are collecting from sources at the rate they permit and keeping the raw evidence, recognising the same complaint reposted as one signal rather than ten, and producing briefs where every claim quotes a source and an unknown is allowed to stay unknown.

Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts.

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | Next.js with TypeScript and SQLite, single user on the loopback interface | a scanner is a tool you run, not a service anybody logs into |
| Sources | A registry where each source records its terms and its published rate limit | a source whose rules you cannot state is one you should not be using |
| Cache | Raw payloads stored with their fetch time and reused rather than re-requested | re-fetching the same page to read it twice is how access gets withdrawn |
| Briefs | One model call per candidate, every claim carrying a quote from the source | an unsourced brief is a sentence you could have written yourself |
| Cost | An estimate before the run and a hard ceiling during it | the cost is per candidate and the candidate count is whatever the feed did today |

## Before you start

Have every one of these ready. The plan assumes them from step one.

- [ ] **Node.js 22 or newer** · free
  - Why: Runs the application and the collection script.
  - Get it: nodejs.org, or nvm install 22.
  - Verify: node --version prints v22 or higher
- [ ] **A source list with a terms note for each** · free
  - Why: The registry refuses to enable a source without one, and writing the note is the step that makes you check.
  - Get it: For each source, read its published terms and rate limits and write one sentence on what you may do and how often.
  - Verify: Every enabled source has both notes recorded and the interval configured
- [ ] **A model API key** · pennies per brief, multiplied by however many candidates a day produces
  - Why: Brief writing is the one part that needs a model, and the cost is per candidate.
  - Get it: Create a key in your provider's console and put it in .env.
  - Verify: One candidate produces a brief with verified quotes
- [ ] **A cost ceiling per run** · free
  - Why: The candidate count is decided by the internet rather than by you, so the limit has to be in code.
  - Get it: Pick a figure per run and put it in the environment.
  - Verify: A run configured to exceed it stops and reports what it completed
- [ ] **A subject you know well** · free
  - Why: The only way to judge whether the briefs are any good is to read the first batch about something whose reality you can check.
  - Get it: Point the first run at a community you belong to and read every brief it produces.
  - Verify: You can say which briefs are wrong and why

## Quick start

Follow Phase 1 in `BUILD_PLAN.md`; it creates the project.

Then copy `.env.example` to `.env` and fill in the values it documents.

## Honest limits

This build deliberately does not replace:

- Editorial curation.
- A historical trend database.
- Paid trend data feeds.
- Anything behind a login.
- 1,000+ already-curated and human-vetted ideas
- historical trend database and idea archive
- AI Research Agent for arbitrary custom ideas
- community, coaching and Q&A strategist sessions
- data exports and idea-suggest personalization

If one of those is essential to you, that is the reason to keep paying for Ideabrowser, and the README should say so rather than pretend.

# Build brief · Ideabrowser

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a local idea scanner that keeps the evidence and refuses to invent a market size to replace Ideabrowser. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
Two rules, and the first is about a number you will be very tempted to print.
No invented market sizing. A trend index is not a volume: Google Trends divides each data point
by the total searches for its geography and time range and then scales the result onto 0 to 100,
so the figure describes relative popularity and says nothing about how many people searched. A
model asked for a market size will produce a confident figure out of nothing at all. This tool
reports the signals it actually collected and writes 'unknown' where it has no source.
Second, every claim in a brief quotes the post it came from and links to it. An idea with no
evidence behind it is just a sentence, and you can write those faster than this can.
Collect only from sources whose published terms allow it, at the rate they publish, identifying
yourself honestly. No logged in scraping and no pretending to be somebody's browser.

### Stack (fixed, do not substitute)
- Runtime: Next.js with TypeScript and SQLite, single user on the loopback interface · a scanner is a tool you run, not a service anybody logs into
- Sources: A registry where each source records its terms and its published rate limit · a source whose rules you cannot state is one you should not be using
- Cache: Raw payloads stored with their fetch time and reused rather than re-requested · re-fetching the same page to read it twice is how access gets withdrawn
- Briefs: One model call per candidate, every claim carrying a quote from the source · an unsourced brief is a sentence you could have written yourself
- Cost: An estimate before the run and a hard ceiling during it · the cost is per candidate and the candidate count is whatever the feed did today

### Data model (create this before Phase 1)
- sources(id, name, endpoint, terms_note, rate_limit_note, user_agent, enabled) · a source with no terms note cannot be enabled
- fetches(id, source_id, url, requested_at, status, bytes, raw_path, from_cache) · raw payloads kept so a brief can be rechecked later
- candidates(id, source_id, external_id, url, author_handle, posted_at, title, text, normalised_hash, first_seen_at, seen_count)
- briefs(id, candidate_id, model_id, created_at, problem, target_user, mvp_scope, pricing_idea, market_size, market_size_source) · market size is a cited figure or the word unknown
- claims(brief_id, ordinal, text, quote, candidate_id) · every claim quotes the post it came from
- verdicts(brief_id, verdict, reason, decided_at) · your own judgement, recorded, because an unjudged backlog only grows
- runs(id, started_at, finished_at, fetch_count, candidates_new, briefs_written, estimated_cost, actual_cost)

### Phase 1 · Sources you are allowed to read, cached once
Build: A source registry where each entry records the endpoint, a note on its published terms, its
  published rate limit and the identifying user agent to send. Enforce a minimum interval between
  requests per source. Store every raw payload with its fetch time and serve repeats from cache.
Trap: Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
Done when: A source without a terms note and a rate limit note cannot be enabled, enforced in code. Two runs an hour apart make no duplicate requests for unchanged pages, verified by counting fetch rows. The identifying user agent is configurable, is sent on every request, and appears in the logs. Every response is written to the raw cache with its status, size and fetch time.
Do not build yet: No candidate extraction, briefs, verdicts or exports.

### Phase 2 · One complaint is one signal
Build: Normalise candidate text, hash it, and match near duplicates across sources and runs. Keep a
  first seen date and a seen count rather than a new row each time. Show recurrence as a count
  with dates, so a complaint made weekly for a year reads as exactly that.
Trap: Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
Done when: The same post collected from two sources produces one candidate with two source links, tested with a fixture. Reposted text with small edits matches the existing candidate under the normalisation rules, which are documented. The seen count and the first seen date are shown wherever a candidate appears. Deleting a duplicate never removes the original, asserted.
Do not build yet: No briefs, verdicts or exports yet.

### Phase 3 · Briefs that cite, or say unknown
Build: For each candidate, produce a brief with problem, target user, MVP scope and a pricing idea.
  Every claim stores a quote checked to appear in the candidate text. The market size field takes
  a cited figure with its source or the word unknown, and there is no third option.
Trap: A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
Done when: Every claim row has a quote confirmed present in the candidate text, asserted in the suite. A market size without a source is refused at write time and the field stores unknown, tested. A brief renders with its quotes and a link to the original post beside each claim. Claims that fail verification are counted and the count is visible rather than logged.
Do not build yet: No verdicts, search or exports yet.

### Phase 4 · A backlog you can actually get through
Build: A feed with search across briefs and candidates. Record a verdict per brief with a written
  reason: pursue, park or discard. Filter by verdict, by recurrence and by the age of the evidence.
  Show how many briefs are still unjudged and how old the oldest is.
Trap: An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.
Done when: A verdict cannot be recorded without a reason, enforced in the interface and the schema. The unjudged count and the oldest unjudged date are shown on the first screen. Search returns both briefs and the candidate text behind them. Filtering by evidence age works and surfaces briefs built on posts older than a stated threshold.
Do not build yet: No cost ceiling, retention or export yet.

### Phase 5 · What it cost, and what it is honestly worth
Build: Print a cost estimate before a run and stop hard at a ceiling. Export briefs, claims and
  verdicts as JSON and Markdown, with the source links intact. Apply a retention to raw payloads
  and report what it removed. Put the limitations on a page in the application, not only in the
  README.
Trap: A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.
Done when: A run stops at the cost ceiling and reports how many briefs it completed. The estimate is shown with its arithmetic before any model call. Exported briefs keep their quotes and source links and open as readable Markdown. The limitations page states plainly that the tool reports collected signals, that a trend index is relative rather than a volume, and that unknown means unknown.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Editorial curation, which is the actual product being replaced here: a vetted backlog built over years by people making judgement calls is not something a script produces on its first run.
- A historical trend database, because backfilling signals you did not collect is impossible and pretending otherwise is the fastest way to a misleading chart.
- Paid trend data feeds, which change the economics entirely and deserve a separate decision rather than an API key in the same .env file.
- Anything behind a login, which means accepting terms on somebody's behalf and breaks the moment a session expires.

### README must contain
- That a trend index is relative popularity rather than search volume, with the source and the date checked
- That market size is either a cited figure or the word unknown, and there is no third state
- Which sources are enabled, what their terms notes say and what rate they are polled at
- That raw payloads are cached and retained, and for how long
- The per run cost arithmetic and where the ceiling is enforced

# Agent instructions · Ideabrowser indie build

- Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Next.js with TypeScript and SQLite, single user on the loopback interface, A registry where each source records its terms and its published rate limit, Raw payloads stored with their fetch time and reused rather than re-requested, One model call per candidate, every claim carrying a quote from the source, An estimate before the run and a hard ceiling during it. Do not substitute.
- Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes.
- Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name.
- Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced.
- Do not invent cryptography, security guarantees, APIs or compliance claims.
- Add a focused test for every destructive, security-sensitive or data-loss path the plan names.
- Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs".

## Known traps

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.

# Build plan · Ideabrowser

A local scanner whose hard problems are collecting from sources at the rate they permit and keeping the raw evidence, recognising the same complaint reposted as one signal rather than ten, and producing briefs where every claim quotes a source and an unknown is allowed to stay unknown.

Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note.

## Phase 1 · Sources you are allowed to read, cached once

Every fetch is permitted, rate limited, identified and kept.

### Steps

1. Make the terms note a required field
   If you cannot write the sentence, you have not read the terms.
   Files: `src/sources.ts`
2. Enforce the interval in the fetch client
   One place, so no code path can accidentally skip it.
   Files: `src/fetch.ts`, `tests/rate-limit.test.ts`
3. Keep every raw payload
   It is the evidence, and re-fetching it later may not be possible.
   Files: `src/cache.ts`

### Done when

- [ ] Sources without terms and rate notes cannot be enabled
- [ ] Repeat runs do not re-request unchanged pages
- [ ] An honest user agent is sent on every request
- [ ] Raw payloads are stored with status and time

### Watch out

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.

## Phase 2 · One complaint is one signal

The same thing said repeatedly is recognised as recurrence, not as volume.

### Steps

1. Normalise before hashing, and write down how
   Case, whitespace, punctuation and links, stated in the documentation.
   Files: `src/dedupe.ts`, `docs/normalisation.md`
2. Increment rather than insert
   Recurrence is the signal, and a row count is not.
   Files: `src/candidates.ts`
3. Show first seen and seen count everywhere
   Two dates and a number make a feed honest.
   Files: `src/components/Candidate.tsx`

### Done when

- [ ] One post from two sources is one candidate
- [ ] Near duplicates match under documented rules
- [ ] First seen and seen count are always visible
- [ ] Removing a duplicate leaves the original

### Watch out

- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.

## Phase 3 · Briefs that cite, or say unknown

No sentence in a brief exists without a quote behind it.

### Steps

1. Verify each quote against the candidate
   A substring check, run before the row is written.
   Files: `src/briefs/verify.ts`, `tests/quote-check.test.ts`
2. Make market size a citation or unknown
   Enforced by the column and by validation, not by the prompt.
   Files: `src/briefs/schema.ts`
3. Render the quote beside every claim
   So reading the brief is reading the evidence.
   Files: `src/components/Brief.tsx`

### Done when

- [ ] Every claim carries a verified quote
- [ ] Market size is a citation or the word unknown
- [ ] Quotes and source links render with each claim
- [ ] Failed verifications are counted and shown

### Watch out

- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.

## Phase 4 · A backlog you can actually get through

Every brief ends up judged rather than scrolled past.

### Steps

1. Require a reason with every verdict
   One sentence, and the backlog starts shrinking.
   Files: `src/verdicts.ts`
2. Put the unjudged count on the first screen
   A queue you can see the length of is a queue you finish.
   Files: `src/app/page.tsx`
3. Filter by evidence age
   A brief built on a two year old post is a different thing.
   Files: `src/search.ts`

### Done when

- [ ] Verdicts require a written reason
- [ ] The unjudged count and oldest date are visible
- [ ] Search covers briefs and candidate text
- [ ] Evidence age filtering works

### Watch out

- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.

## Not in this build

- Editorial curation.
- A historical trend database.
- Paid trend data feeds.
- Anything behind a login.

## After v1, if you want it

- A weekly digest of new candidates with high recurrence, which is a query and a Markdown file
- A second source in a different community, once the deduplication rules have proven themselves
- Tracking which pursued ideas went anywhere, which is the only feedback loop that would improve the briefs

# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sources, candidates, briefs and verdicts.
DATABASE_PATH=./data/ideas.db

# Required. Raw payloads with their fetch times. The evidence behind every brief.
RAW_CACHE_DIR=./data/raw

# Required. Endpoints with their terms and rate limit notes.
SOURCES_FILE=./sources.json

# Required. Sent on every request. Identify yourself honestly.
USER_AGENT=idea-scanner/0.1 (contact: you@example.com)

# Required. Minimum gap between requests to one source, enforced in the client.
FETCH_MIN_INTERVAL_MS=2000

# Required. Per run, in your billing currency. Enforced, not remembered.
COST_CEILING=5.00

# Required. How long payloads are kept. The cleanup reports what it removed.
RAW_RETENTION_DAYS=180

# Required · secret. Brief writing. Without it the tool still collects and deduplicates.
ANTHROPIC_API_KEY=sk-ant-...redacted

# Ideabrowser · product brief

## Problem

Scraping Reddit/search trends and having an LLM draft a business-opportunity writeup per idea is a weekend build. What's genuinely hard to replicate is the editorial curation, the accumulated database of 1,000+ pre-researched ideas, and the trend-signal pipeline built up over time. A DIY version can generate plausible-sounding ideas on demand but starts from zero data and zero curation.

## Product outcome

A scanner that reads only what it is allowed to, keeps the raw evidence, counts a repeated complaint once, writes briefs whose every claim quotes a real post, and prints unknown where it does not know.

## Target user

A builder who needs a maintainable product foundation, not a one-off demo.

## Required capabilities

- OpenAI/Anthropic API key
- Reddit or Google Trends API access

## Explicit non-goals for v1

- Editorial curation.
- A historical trend database.
- Paid trend data feeds.
- Anything behind a login.
- 1,000+ already-curated and human-vetted ideas
- historical trend database and idea archive
- AI Research Agent for arbitrary custom ideas
- community, coaching and Q&A strategist sessions
- data exports and idea-suggest personalization

## Success criteria

- Sources are confirmed unenablable without terms and rate notes
- Repeat runs are confirmed not to re-request unchanged pages
- Every claim is confirmed to carry a verified quote
- Market size is confirmed to be a citation or unknown
- The cost ceiling is confirmed to stop a run

# Build brief · Ideabrowser

The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins.

Build me a local idea scanner that keeps the evidence and refuses to invent a market size to replace Ideabrowser. Build it in phases, in the order below.
Do not write the whole thing in one pass. Finish a phase, run its "Done when"
check, fix what fails, and only then start the next phase.

### Rule zero
Two rules, and the first is about a number you will be very tempted to print.
No invented market sizing. A trend index is not a volume: Google Trends divides each data point
by the total searches for its geography and time range and then scales the result onto 0 to 100,
so the figure describes relative popularity and says nothing about how many people searched. A
model asked for a market size will produce a confident figure out of nothing at all. This tool
reports the signals it actually collected and writes 'unknown' where it has no source.
Second, every claim in a brief quotes the post it came from and links to it. An idea with no
evidence behind it is just a sentence, and you can write those faster than this can.
Collect only from sources whose published terms allow it, at the rate they publish, identifying
yourself honestly. No logged in scraping and no pretending to be somebody's browser.

### Stack (fixed, do not substitute)
- Runtime: Next.js with TypeScript and SQLite, single user on the loopback interface · a scanner is a tool you run, not a service anybody logs into
- Sources: A registry where each source records its terms and its published rate limit · a source whose rules you cannot state is one you should not be using
- Cache: Raw payloads stored with their fetch time and reused rather than re-requested · re-fetching the same page to read it twice is how access gets withdrawn
- Briefs: One model call per candidate, every claim carrying a quote from the source · an unsourced brief is a sentence you could have written yourself
- Cost: An estimate before the run and a hard ceiling during it · the cost is per candidate and the candidate count is whatever the feed did today

### Data model (create this before Phase 1)
- sources(id, name, endpoint, terms_note, rate_limit_note, user_agent, enabled) · a source with no terms note cannot be enabled
- fetches(id, source_id, url, requested_at, status, bytes, raw_path, from_cache) · raw payloads kept so a brief can be rechecked later
- candidates(id, source_id, external_id, url, author_handle, posted_at, title, text, normalised_hash, first_seen_at, seen_count)
- briefs(id, candidate_id, model_id, created_at, problem, target_user, mvp_scope, pricing_idea, market_size, market_size_source) · market size is a cited figure or the word unknown
- claims(brief_id, ordinal, text, quote, candidate_id) · every claim quotes the post it came from
- verdicts(brief_id, verdict, reason, decided_at) · your own judgement, recorded, because an unjudged backlog only grows
- runs(id, started_at, finished_at, fetch_count, candidates_new, briefs_written, estimated_cost, actual_cost)

### Phase 1 · Sources you are allowed to read, cached once
Build: A source registry where each entry records the endpoint, a note on its published terms, its
  published rate limit and the identifying user agent to send. Enforce a minimum interval between
  requests per source. Store every raw payload with its fetch time and serve repeats from cache.
Trap: Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
Done when: A source without a terms note and a rate limit note cannot be enabled, enforced in code. Two runs an hour apart make no duplicate requests for unchanged pages, verified by counting fetch rows. The identifying user agent is configurable, is sent on every request, and appears in the logs. Every response is written to the raw cache with its status, size and fetch time.
Do not build yet: No candidate extraction, briefs, verdicts or exports.

### Phase 2 · One complaint is one signal
Build: Normalise candidate text, hash it, and match near duplicates across sources and runs. Keep a
  first seen date and a seen count rather than a new row each time. Show recurrence as a count
  with dates, so a complaint made weekly for a year reads as exactly that.
Trap: Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
Done when: The same post collected from two sources produces one candidate with two source links, tested with a fixture. Reposted text with small edits matches the existing candidate under the normalisation rules, which are documented. The seen count and the first seen date are shown wherever a candidate appears. Deleting a duplicate never removes the original, asserted.
Do not build yet: No briefs, verdicts or exports yet.

### Phase 3 · Briefs that cite, or say unknown
Build: For each candidate, produce a brief with problem, target user, MVP scope and a pricing idea.
  Every claim stores a quote checked to appear in the candidate text. The market size field takes
  a cited figure with its source or the word unknown, and there is no third option.
Trap: A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
Done when: Every claim row has a quote confirmed present in the candidate text, asserted in the suite. A market size without a source is refused at write time and the field stores unknown, tested. A brief renders with its quotes and a link to the original post beside each claim. Claims that fail verification are counted and the count is visible rather than logged.
Do not build yet: No verdicts, search or exports yet.

### Phase 4 · A backlog you can actually get through
Build: A feed with search across briefs and candidates. Record a verdict per brief with a written
  reason: pursue, park or discard. Filter by verdict, by recurrence and by the age of the evidence.
  Show how many briefs are still unjudged and how old the oldest is.
Trap: An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.
Done when: A verdict cannot be recorded without a reason, enforced in the interface and the schema. The unjudged count and the oldest unjudged date are shown on the first screen. Search returns both briefs and the candidate text behind them. Filtering by evidence age works and surfaces briefs built on posts older than a stated threshold.
Do not build yet: No cost ceiling, retention or export yet.

### Phase 5 · What it cost, and what it is honestly worth
Build: Print a cost estimate before a run and stop hard at a ceiling. Export briefs, claims and
  verdicts as JSON and Markdown, with the source links intact. Apply a retention to raw payloads
  and report what it removed. Put the limitations on a page in the application, not only in the
  README.
Trap: A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.
Done when: A run stops at the cost ceiling and reports how many briefs it completed. The estimate is shown with its arithmetic before any model call. Exported briefs keep their quotes and source links and open as readable Markdown. The limitations page states plainly that the tool reports collected signals, that a trend index is relative rather than a volume, and that unknown means unknown.
Do not build yet: Nothing after this.

### Out of scope (and why)
- Editorial curation, which is the actual product being replaced here: a vetted backlog built over years by people making judgement calls is not something a script produces on its first run.
- A historical trend database, because backfilling signals you did not collect is impossible and pretending otherwise is the fastest way to a misleading chart.
- Paid trend data feeds, which change the economics entirely and deserve a separate decision rather than an API key in the same .env file.
- Anything behind a login, which means accepting terms on somebody's behalf and breaks the moment a session expires.

### README must contain
- That a trend index is relative popularity rather than search volume, with the source and the date checked
- That market size is either a cited figure or the word unknown, and there is no third state
- Which sources are enabled, what their terms notes say and what rate they are polled at
- That raw payloads are cached and retained, and for how long
- The per run cost arithmetic and where the ceiling is enforced

# Architecture · Ideabrowser

## Stack

| Part | Choice | Why |
| --- | --- | --- |
| Runtime | Next.js with TypeScript and SQLite, single user on the loopback interface | a scanner is a tool you run, not a service anybody logs into |
| Sources | A registry where each source records its terms and its published rate limit | a source whose rules you cannot state is one you should not be using |
| Cache | Raw payloads stored with their fetch time and reused rather than re-requested | re-fetching the same page to read it twice is how access gets withdrawn |
| Briefs | One model call per candidate, every claim carrying a quote from the source | an unsourced brief is a sentence you could have written yourself |
| Cost | An estimate before the run and a hard ceiling during it | the cost is per candidate and the candidate count is whatever the feed did today |

## Modules

Each module has one owner concern and a documented way to replace it.

| Module | Owns | How to replace it |
| --- | --- | --- |
| Sources | The registry, terms notes and intervals. | Any source with published terms, added as a registry row. |
| Fetch | Rate limiting, caching and raw storage. | Any client, provided the interval stays inside it. |
| Dedupe | Normalisation, hashing and recurrence counts. | Any matching rule, provided it is documented. |
| Briefs | Claims with verified quotes and the unknown rule. | Any model, provided verification stays in front of the write. |
| Backlog | Verdicts with reasons and the unjudged count. | Any interface that still requires a reason. |

## Configuration

Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists.

- `DATABASE_PATH` · required · Sources, candidates, briefs and verdicts.
- `RAW_CACHE_DIR` · required · Raw payloads with their fetch times. The evidence behind every brief.
- `SOURCES_FILE` · required · Endpoints with their terms and rate limit notes.
- `USER_AGENT` · required · Sent on every request. Identify yourself honestly.
- `FETCH_MIN_INTERVAL_MS` · required · Minimum gap between requests to one source, enforced in the client.
- `COST_CEILING` · required · Per run, in your billing currency. Enforced, not remembered.
- `RAW_RETENTION_DAYS` · required · How long payloads are kept. The cleanup reports what it removed.
- `ANTHROPIC_API_KEY` · required, secret · Brief writing. Without it the tool still collects and deduplicates.

## Production baseline

- Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives.
- Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised.
- Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts.
- Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists.
- Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path.

## Decision records

For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.

# Agent instructions · Ideabrowser product build

- Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Next.js with TypeScript and SQLite, single user on the loopback interface, A registry where each source records its terms and its published rate limit, Raw payloads stored with their fetch time and reused rather than re-requested, One model call per candidate, every claim carrying a quote from the source, An estimate before the run and a hard ceiling during it.
- Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit.
- Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present.
- Never invent cryptography or silently weaken a requirement to make a check pass.
- Put every external service behind an interface with a deterministic fake for tests.
- Add migrations and rollback or recovery notes for every persistent data change.
- Log useful operational context without credentials, tokens, passwords or personal data.
- Update documentation and run every check before completing a milestone.

## Known traps

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.
- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.
- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.
- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.
- A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.

# Delivery milestones · Ideabrowser

Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer.

## M1 · Sources you are allowed to read, cached once

Every fetch is permitted, rate limited, identified and kept.

### Steps

1. Make the terms note a required field
   If you cannot write the sentence, you have not read the terms.
   Files: `src/sources.ts`
2. Enforce the interval in the fetch client
   One place, so no code path can accidentally skip it.
   Files: `src/fetch.ts`, `tests/rate-limit.test.ts`
3. Keep every raw payload
   It is the evidence, and re-fetching it later may not be possible.
   Files: `src/cache.ts`

### Done when

- [ ] Sources without terms and rate notes cannot be enabled
- [ ] Repeat runs do not re-request unchanged pages
- [ ] An honest user agent is sent on every request
- [ ] Raw payloads are stored with status and time

### Watch out

- Fetching the same listing on every page load while developing turns a few hundred requests into a few thousand in an afternoon, and the first sign of trouble is a blanket block that also ends your reading of the site as a human. Cache raw payloads, enforce a minimum interval in the client rather than in your habits, and identify the tool honestly.

## M2 · One complaint is one signal

The same thing said repeatedly is recognised as recurrence, not as volume.

### Steps

1. Normalise before hashing, and write down how
   Case, whitespace, punctuation and links, stated in the documentation.
   Files: `src/dedupe.ts`, `docs/normalisation.md`
2. Increment rather than insert
   Recurrence is the signal, and a row count is not.
   Files: `src/candidates.ts`
3. Show first seen and seen count everywhere
   Two dates and a number make a feed honest.
   Files: `src/components/Candidate.tsx`

### Done when

- [ ] One post from two sources is one candidate
- [ ] Near duplicates match under documented rules
- [ ] First seen and seen count are always visible
- [ ] Removing a duplicate leaves the original

### Watch out

- Treating every post as a fresh candidate turns one grumble reposted in five places into five ideas, and the resulting feed looks like a trend when it is one person having a bad week. Normalise and hash the text, increment a seen count, and let recurrence be a number with dates behind it.

## M3 · Briefs that cite, or say unknown

No sentence in a brief exists without a quote behind it.

### Steps

1. Verify each quote against the candidate
   A substring check, run before the row is written.
   Files: `src/briefs/verify.ts`, `tests/quote-check.test.ts`
2. Make market size a citation or unknown
   Enforced by the column and by validation, not by the prompt.
   Files: `src/briefs/schema.ts`
3. Render the quote beside every claim
   So reading the brief is reading the evidence.
   Files: `src/components/Brief.tsx`

### Done when

- [ ] Every claim carries a verified quote
- [ ] Market size is a citation or the word unknown
- [ ] Quotes and source links render with each claim
- [ ] Failed verifications are counted and shown

### Watch out

- A model asked to size a market will produce a plausible number with a currency symbol on it, and that number will be repeated in a pitch deck three weeks later by somebody who has forgotten where it came from. Make the field accept only a citation or the word unknown, and reject anything else before it is saved.

## M4 · A backlog you can actually get through

Every brief ends up judged rather than scrolled past.

### Steps

1. Require a reason with every verdict
   One sentence, and the backlog starts shrinking.
   Files: `src/verdicts.ts`
2. Put the unjudged count on the first screen
   A queue you can see the length of is a queue you finish.
   Files: `src/app/page.tsx`
3. Filter by evidence age
   A brief built on a two year old post is a different thing.
   Files: `src/search.ts`

### Done when

- [ ] Verdicts require a written reason
- [ ] The unjudged count and oldest date are visible
- [ ] Search covers briefs and candidate text
- [ ] Evidence age filtering works

### Watch out

- An endless feed with a star button produces four hundred starred items and no decisions, which is the same state you were in before you built anything. Require a written reason with each verdict and put the unjudged count on the first screen, so the backlog is visibly a queue rather than a library.

## M5 · What it cost, and what it is honestly worth (production only)

Spending is bounded, evidence has a retention, and the tool states its own limits.

### Steps

1. Estimate from the candidate count, then confirm
   The multiplication before the money, every time.
   Files: `src/cost.ts`
2. Export with quotes and links intact
   A brief without its evidence is worth nothing outside the tool.
   Files: `src/export.ts`
3. Put the limits in the product
   In the interface, where somebody reading a brief will see them.
   Files: `src/app/limits/page.tsx`

### Done when

- [ ] The ceiling stops the run and reports progress
- [ ] The estimate is shown before any model call
- [ ] Exports keep quotes and source links
- [ ] The limitations page exists in the application

### Watch out

- A run that briefs every candidate it finds costs whatever the feed happened to produce that day, which is fine on a quiet Tuesday and unpleasant after something went viral. Estimate from the candidate count before spending, confirm, and stop at the ceiling with a report of what was completed.

# Operations · Ideabrowser

## Backup

The database and the raw cache together, since a brief whose payload has been deleted can no longer be checked against its source. The verdicts are the part you cannot regenerate at any price.

## Restore

Restore both, then open three briefs and confirm their quotes still resolve against the cached payloads. Quotes that no longer verify mean the cache was left out of the backup.

Do a restore drill before the first real user, and write the date here when it passes.

## Monitoring

Fetch counts per source against the interval, the quote verification failure rate, spend against the estimate, and the age of the oldest unjudged brief. A rising verification failure rate usually means a model change.

## Incident checklist

The incident worth naming is a number leaving the tool without a source, because it will be repeated. The unknown rule and the claim verification both exist to make that hard rather than discouraged.

1. Contain the issue without destroying evidence or user data.
2. Record the timeline and affected scope.
3. Rotate exposed secrets and revoke compromised sessions or credentials.
4. Restore from a verified backup when needed.
5. Document the root cause, the remediation and the regression test.

## Release gate

- [ ] Sources are confirmed unenablable without terms and rate notes
- [ ] Repeat runs are confirmed not to re-request unchanged pages
- [ ] Every claim is confirmed to carry a verified quote
- [ ] Market size is confirmed to be a citation or unknown
- [ ] The cost ceiling is confirmed to stop a run

## Launch constraint

Do not market omitted Ideabrowser capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.

# Copy to .env and fill in. Never commit .env; this file documents it.

# Required. Sources, candidates, briefs and verdicts.
DATABASE_PATH=./data/ideas.db

# Required. Raw payloads with their fetch times. The evidence behind every brief.
RAW_CACHE_DIR=./data/raw

# Required. Endpoints with their terms and rate limit notes.
SOURCES_FILE=./sources.json

# Required. Sent on every request. Identify yourself honestly.
USER_AGENT=idea-scanner/0.1 (contact: you@example.com)

# Required. Minimum gap between requests to one source, enforced in the client.
FETCH_MIN_INTERVAL_MS=2000

# Required. Per run, in your billing currency. Enforced, not remembered.
COST_CEILING=5.00

# Required. How long payloads are kept. The cleanup reports what it removed.
RAW_RETENTION_DAYS=180

# Required · secret. Brief writing. Without it the tool still collects and deduplicates.
ANTHROPIC_API_KEY=sk-ant-...redacted

$ choose a build depth, inspect the files, then open the complete pack in your agent