Vibecode IdeaFast
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The pipeline is honest work an agent can do: pull public Reddit JSON, prefilter complaint-shaped text, classify with an LLM, embed and cluster, rank by frequency times severity times recency. A weekend gets you ranked pain themes with real permalinks for two or three subreddits you already know. What does not fall out of one session is everything after the demo: staying inside Reddit's rate limits at scale, picking which communities are worth scanning when you do not already know, deduping the same pain across runs so week two is not week one again, and keeping the LLM bill under the price of the subscription. Verdict is kinda, not yes, because the first run is easy and the tenth is where the product actually lives.
the project pack5 files · written for this build, step by step
You are building a lean indie version of IdeaFast. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # IdeaFast · indie build A forum mining pipeline whose hard problems are ingesting politely and resumably within a platform's published limits, keeping the classification bill under the price of the subscription it replaces, and clustering in a way you would defend to someone who disagrees with the result. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a CLI for scanning and a small local dashboard | one repository, one scan command, no accounts | | Storage | SQLite via better-sqlite3, every raw response cached by request | a re-run must cost nothing, or you will not re-run | | Ingestion | Authenticated, rate-limited, resumable, with the limit read from the platform | the fetcher is the part that gets you blocked | | Classification | A cheap model over prefiltered text, cached by content hash | the prefilter is the entire cost story | | Clustering | Embeddings with a threshold you tuned and can explain | a threshold nobody looked at produces one blob or four hundred singletons | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the scanner and the dashboard. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A registered application on the platform you are reading** · free for non-commercial use on most platforms, but read the current terms - Why: Authenticated access is what the published rate limit applies to, and what makes your client identifiable rather than anonymous traffic to be throttled. - Get it: Register an application in the platform's developer settings and note the credentials and the terms you accepted. - Verify: An authenticated request succeeds and the response carries rate-limit information - [ ] **A model API key for classification and embeddings** · pay per use, and the prefilter is what keeps it small - Why: Classification and embedding are the only paid parts, and the key is what the cost ceiling is protecting. - Get it: Create an account with a provider and generate a key, then record the current prices with today's date. - Verify: A ten item classification run completes and the ledger shows tokens - [ ] **The current pricing page for your chosen model** · free - Why: The cost estimate is only useful if the rates behind it are today's, and every provider in this space has changed them. - Get it: Open the provider's pricing page, record input and output rates and any batch discount, with the date. - Verify: The configuration holds the rates and the date, and the estimate uses them - [ ] **Fifty items you have labelled by hand** · free, an hour of reading - Why: The severity scale means whatever the model decides unless you have your own labels to compare against, and fifty is enough to catch a classifier that is systematically wrong. - Get it: Read fifty prefiltered items and mark is_pain and severity yourself before looking at any model output. - Verify: The labels exist in the repository and the agreement rate is computed ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Sources beyond the first platform. - Automatic community discovery. - Idea generation and validation. - Logged-in scraping or limit evasion. - community discovery, you can only scan subreddits you already thought of - cross-scan dedupe, so repeat runs resurface the same pains as if they were new - a warmed corpus, every fresh scan pays the full ingestion wait - cost control, naive LLM classification of a busy subreddit gets expensive fast - the idea generation and validation layer on top of the raw clusters If one of those is essential to you, that is the reason to keep paying for IdeaFast, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · IdeaFast The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local pipeline that reads public forum complaints and turns them into ranked pain themes backed by quotes you can click to replace IdeaFast. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and the first one decides whether this project survives its second week. Read the platform's own developer terms and rate limits before writing the fetcher, then obey them. Register an application, authenticate the documented way, send a descriptive user agent, stay under the published request rate, and back off when the response tells you to. Reddit in particular publishes both a rate limit and an access policy that has changed more than once, so check the current pages rather than copying a number from a blog post, and record what you found and when. An aggressive client gets an application key revoked, and then the corpus stops growing for everyone using your code. Second, evidence is verbatim or it is not evidence. Every quote shown to a user is the exact text from the source with a working permanent link. A paraphrased complaint is an opinion with a citation stapled to it, and the whole value of this tool is that you can click through and read the actual person being actually annoyed. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a CLI for scanning and a small local dashboard · one repository, one scan command, no accounts - Storage: SQLite via better-sqlite3, every raw response cached by request · a re-run must cost nothing, or you will not re-run - Ingestion: Authenticated, rate-limited, resumable, with the limit read from the platform · the fetcher is the part that gets you blocked - Classification: A cheap model over prefiltered text, cached by content hash · the prefilter is the entire cost story - Clustering: Embeddings with a threshold you tuned and can explain · a threshold nobody looked at produces one blob or four hundred singletons ### Data model (create this before Phase 1) - sources(id, platform, community, added_at, last_scanned_at, notes) - raw_responses(id, source_id, request_url, fetched_at, status, body_sha256, body_path) · cached so a re-run is free - items(id, source_id, kind, platform_id, permalink, author_hash, created_utc, text, parent_id) · kind is post or comment - prefilter(item_id, matched_patterns_json, passed) · which cheap rule let this one through to a model - classifications(item_id, content_sha256, is_pain, severity, one_line, model_id, tokens_in, tokens_out, classified_at) · keyed by content hash so identical text is never paid for twice - clusters(run_id, id, name, size, mean_severity, recency_weight, score) · named by a model, scored by a formula you can read - cluster_quotes(cluster_id, item_id, ordinal, verbatim, permalink) · verbatim is asserted to appear in the item's text - runs(id, started_at, finished_at, sources_json, items_seen, items_classified, cost_estimate, cost_actual, threshold) - The classifications table is keyed by content hash rather than by item, so the same complaint quoted in three threads costs one call. ### Phase 1 · An ingester that is welcome back tomorrow Build: Register an application and authenticate the documented way. Put the platform's published rate limit in configuration with the date you read it, and enforce it in the client rather than hoping. Honour whatever rate-limit headers the responses carry and back off on 429 and 5xx. Store every raw response keyed by its request so a second scan reads from the cache. Trap: Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. Done when: A scan interrupted at any point resumes without refetching anything already cached, tested by killing it mid-scan and comparing request counts. The client refuses to exceed the configured rate, asserted by a test that counts requests per interval against a fake clock. A 429 response causes a backoff of at least the interval the response asks for, tested. The configuration records the rate limit, the source page and the date it was checked, and the scan prints them before starting. Do not build yet: No prefiltering, classification, clustering or dashboard yet. ### Phase 2 · Spend nothing on text that is not a complaint Build: Match complaint-shaped phrases with plain patterns: expressions of frustration, of giving up, of workarounds, of missing features. Record which patterns matched each item. Report the pass rate per source, and sample a hundred rejected items by hand to see what you are throwing away. Trap: Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. Done when: The pass rate per source is reported and a hand-reviewed sample of a hundred rejected items is recorded in the repository with a note on what was missed. Prefiltering a cached corpus runs without any network or model call, tested. Which patterns matched is stored per item, so the filter can be tuned from data rather than from intuition. A source whose pass rate is above a configured ceiling warns, because a filter that passes everything is not a filter. Do not build yet: No classification, clustering or dashboard yet. ### Phase 3 · Classification with a bill you predicted Build: Classify each item into is_pain, a severity and a one-line summary, keyed by content hash so identical text is paid for once. Estimate the cost before the run and refuse to start above the ceiling. Use the cheapest model that passes your own spot check, and batch the work when it is not urgent: Anthropic's published pricing puts its smallest current model at 1 US dollar per million input tokens and 5 per million output, and its batch processing at half those rates for asynchronous work (checked on their pricing page). Put whichever provider's figures you use in configuration with the date. Trap: Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. Done when: Re-running a scan with no new items makes zero model calls, tested by counting calls. Identical text appearing under two different items is classified once, tested. The run prints an estimated cost before the first call and refuses to start above the configured ceiling. A hand-labelled sample of fifty items is compared against the model's labels and the agreement rate is recorded, so the choice of model is a measurement rather than a preference. Do not build yet: No clustering, scoring or dashboard yet. ### Phase 4 · Clusters you would defend Build: Embed the one-line summaries and group them. Make the similarity threshold a parameter, sweep it over the corpus, and print the resulting cluster size distribution so you choose it by looking. Name each cluster with a model and select its strongest quotes, asserting that each quote appears verbatim in its item's text before storing it. Trap: Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. Done when: A threshold sweep over the corpus prints cluster counts and size distributions for at least five values, and the chosen value is recorded with a one-line reason. Every stored quote is asserted to appear verbatim in its item's text, and a deliberately altered quote fails the assertion in a test. Each cluster's score is computed from a formula written in the README, and changing one input changes the score in the direction the formula says. Re-running clustering on an unchanged corpus at the same threshold produces the same clusters. Do not build yet: No cross-run comparison, dashboard or export yet. ### Phase 5 · The tenth run, not the first Build: Match this run's clusters to previous runs by embedding similarity and shared items, and mark each as new, growing, shrinking or unchanged. The dashboard defaults to what changed. Export clusters, quotes and scores as CSV and JSON. State plainly in the README what this corpus does not contain, starting with communities you did not think to add. Trap: A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent. Done when: A second scan over the same sources shows clusters marked new, growing, shrinking or unchanged, and the default view shows only the first three. Cluster matching across runs survives a cluster being renamed by the model, tested by forcing a rename. Exports contain clusters, their scores, and their verbatim quotes with permanent links, and open correctly elsewhere. The README states that community selection is manual and that this tool cannot find a community you did not name. Do not build yet: Nothing after this. ### Out of scope (and why) - Sources beyond the one platform you start with, because a second platform means a second set of terms, limits and quirks before the first one is producing clusters you trust. - Automatic community discovery, which is the expensive half of the paid products and needs a corpus you do not have. - Idea generation and validation on top of the clusters, which is a separate product and a much weaker claim. - Anything requiring a logged-in session or evading a rate limit, which breaks the terms and ends the project. ### README must contain - The platform's published rate limit, the page it came from and the date it was checked, and that an aggressive client gets blocked - That the prefilter runs before any model call and what its measured pass rate is per source - How the classification cache is keyed on content hash, and the estimated cost per thousand items with the pricing source and date - The clustering threshold, how it was chosen, and the scoring formula in full - That community selection is manual, so this cannot surface a community you did not name ===== AGENTS.md ===== # Agent instructions · IdeaFast indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript, a CLI for scanning and a small local dashboard, SQLite via better-sqlite3, every raw response cached by request, Authenticated, rate-limited, resumable, with the limit read from the platform, A cheap model over prefiltered text, cached by content hash, Embeddings with a threshold you tuned and can explain. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. ===== BUILD_PLAN.md ===== # Build plan · IdeaFast A forum mining pipeline whose hard problems are ingesting politely and resumably within a platform's published limits, keeping the classification bill under the price of the subscription it replaces, and clustering in a way you would defend to someone who disagrees with the result. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · An ingester that is welcome back tomorrow Fetching is authenticated, under the published rate, resumable, and cached so a re-run costs nothing. ### Steps 1. Put the limit and its source in configuration With a date, because these policies change. Files: `src/config/limits.ts`, `docs/platform-policy.md` 2. Enforce the rate in the client A sleep in a loop is not a rate limiter when anything runs concurrently. Files: `src/fetch/limiter.ts`, `tests/rate-limit.test.ts` 3. Cache every raw response So the second scan is free and the parser can be fixed offline. Files: `src/fetch/cache.ts` ### Done when - [ ] A killed scan resumes without refetching - [ ] The client cannot exceed the configured rate, asserted - [ ] A 429 triggers at least the requested backoff - [ ] The limit, its source and check date print before a scan ### Watch out - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. ## Phase 2 · Spend nothing on text that is not a complaint Cheap rules reduce the corpus to complaint-shaped text before a single token is spent. ### Steps 1. Write patterns, then measure the pass rate The number that predicts your bill. Files: `src/prefilter.ts` 2. Read a hundred rejected items yourself The only way to know what the filter costs you in recall. Files: `docs/prefilter-review.md` 3. Store which patterns matched Tuning from data beats tuning from memory. Files: `src/prefilter.ts`, `tests/prefilter.test.ts` ### Done when - [ ] Pass rate is reported per source - [ ] A hundred rejected items are reviewed and recorded - [ ] Prefiltering runs with no network or model calls - [ ] An excessive pass rate raises a warning ### Watch out - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. ## Phase 3 · Classification with a bill you predicted Each surviving item is classified once, cached by content, and the run's cost is estimated before it starts. ### Steps 1. Key the cache on content hash The same complaint in three threads is one call. Files: `src/classify.ts`, `tests/classify-cache.test.ts` 2. Estimate before the run, refuse above the ceiling A ceiling checked after the fact is a receipt. Files: `src/cost.ts` 3. Hand-label fifty items and compare Model choice is a measurement you can afford to make. Files: `docs/classifier-agreement.md` ### Done when - [ ] A re-run with no new items makes zero calls - [ ] Duplicate text is classified once - [ ] An estimate prints and blocks above the ceiling - [ ] Agreement against fifty hand labels is recorded ### Watch out - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. ## Phase 4 · Clusters you would defend Themes are grouped at a threshold you chose deliberately, named clearly, and evidenced verbatim. ### Steps 1. Sweep the threshold and look at it Five values and a distribution, before choosing. Files: `scripts/threshold-sweep.ts`, `docs/clustering.md` 2. Assert every quote is verbatim String comparison against the source text, on write. Files: `src/cluster/quotes.ts`, `tests/verbatim.test.ts` 3. Write the scoring formula in the README A score nobody can recompute is a number with a halo. Files: `README.md`, `src/cluster/score.ts` ### Done when - [ ] A threshold sweep is printed and the choice recorded - [ ] An altered quote fails the verbatim assertion - [ ] Scores follow the documented formula - [ ] Clustering is reproducible on an unchanged corpus ### Watch out - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. ## Not in this build - Sources beyond the first platform. - Automatic community discovery. - Idea generation and validation. - Logged-in scraping or limit evasion. ## After v1, if you want it - A second platform added as an ingest module, once the first has run for a month without a 429 - Per-cluster alerting when a theme grows past a threshold, using data the compare step already produces - Storing embeddings for the raw items rather than the summaries, which changes clustering quality and needs the sweep repeated ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Raw responses, items, classifications and clusters. DATABASE_PATH=./data/pain.db # Required. The communities to scan. Manual, and that is the stated limitation. SOURCES_FILE=./sources.txt # Required. From your registered application. Identifies the client the rate limit applies to. PLATFORM_CLIENT_ID=your-app-client-id # Required · secret. The other half of the application credentials. PLATFORM_CLIENT_SECRET=...redacted # Required. Descriptive and contactable, as the platform's rules ask. USER_AGENT=pain-finder/0.1 (contact: you@example.com) # Required. Kept under the platform's published limit, which is recorded with its check date. REQUESTS_PER_MINUTE=60 # Optional · secret. Classification and naming. Unset means the pipeline still ingests and prefilters. ANTHROPIC_API_KEY=sk-ant-...redacted # Required. A run estimated above this refuses to start. RUN_COST_CEILING_USD=5
You are building a lean indie version of IdeaFast. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # IdeaFast · indie build A forum mining pipeline whose hard problems are ingesting politely and resumably within a platform's published limits, keeping the classification bill under the price of the subscription it replaces, and clustering in a way you would defend to someone who disagrees with the result. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a CLI for scanning and a small local dashboard | one repository, one scan command, no accounts | | Storage | SQLite via better-sqlite3, every raw response cached by request | a re-run must cost nothing, or you will not re-run | | Ingestion | Authenticated, rate-limited, resumable, with the limit read from the platform | the fetcher is the part that gets you blocked | | Classification | A cheap model over prefiltered text, cached by content hash | the prefilter is the entire cost story | | Clustering | Embeddings with a threshold you tuned and can explain | a threshold nobody looked at produces one blob or four hundred singletons | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the scanner and the dashboard. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A registered application on the platform you are reading** · free for non-commercial use on most platforms, but read the current terms - Why: Authenticated access is what the published rate limit applies to, and what makes your client identifiable rather than anonymous traffic to be throttled. - Get it: Register an application in the platform's developer settings and note the credentials and the terms you accepted. - Verify: An authenticated request succeeds and the response carries rate-limit information - [ ] **A model API key for classification and embeddings** · pay per use, and the prefilter is what keeps it small - Why: Classification and embedding are the only paid parts, and the key is what the cost ceiling is protecting. - Get it: Create an account with a provider and generate a key, then record the current prices with today's date. - Verify: A ten item classification run completes and the ledger shows tokens - [ ] **The current pricing page for your chosen model** · free - Why: The cost estimate is only useful if the rates behind it are today's, and every provider in this space has changed them. - Get it: Open the provider's pricing page, record input and output rates and any batch discount, with the date. - Verify: The configuration holds the rates and the date, and the estimate uses them - [ ] **Fifty items you have labelled by hand** · free, an hour of reading - Why: The severity scale means whatever the model decides unless you have your own labels to compare against, and fifty is enough to catch a classifier that is systematically wrong. - Get it: Read fifty prefiltered items and mark is_pain and severity yourself before looking at any model output. - Verify: The labels exist in the repository and the agreement rate is computed ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Sources beyond the first platform. - Automatic community discovery. - Idea generation and validation. - Logged-in scraping or limit evasion. - community discovery, you can only scan subreddits you already thought of - cross-scan dedupe, so repeat runs resurface the same pains as if they were new - a warmed corpus, every fresh scan pays the full ingestion wait - cost control, naive LLM classification of a busy subreddit gets expensive fast - the idea generation and validation layer on top of the raw clusters If one of those is essential to you, that is the reason to keep paying for IdeaFast, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · IdeaFast The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local pipeline that reads public forum complaints and turns them into ranked pain themes backed by quotes you can click to replace IdeaFast. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and the first one decides whether this project survives its second week. Read the platform's own developer terms and rate limits before writing the fetcher, then obey them. Register an application, authenticate the documented way, send a descriptive user agent, stay under the published request rate, and back off when the response tells you to. Reddit in particular publishes both a rate limit and an access policy that has changed more than once, so check the current pages rather than copying a number from a blog post, and record what you found and when. An aggressive client gets an application key revoked, and then the corpus stops growing for everyone using your code. Second, evidence is verbatim or it is not evidence. Every quote shown to a user is the exact text from the source with a working permanent link. A paraphrased complaint is an opinion with a citation stapled to it, and the whole value of this tool is that you can click through and read the actual person being actually annoyed. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a CLI for scanning and a small local dashboard · one repository, one scan command, no accounts - Storage: SQLite via better-sqlite3, every raw response cached by request · a re-run must cost nothing, or you will not re-run - Ingestion: Authenticated, rate-limited, resumable, with the limit read from the platform · the fetcher is the part that gets you blocked - Classification: A cheap model over prefiltered text, cached by content hash · the prefilter is the entire cost story - Clustering: Embeddings with a threshold you tuned and can explain · a threshold nobody looked at produces one blob or four hundred singletons ### Data model (create this before Phase 1) - sources(id, platform, community, added_at, last_scanned_at, notes) - raw_responses(id, source_id, request_url, fetched_at, status, body_sha256, body_path) · cached so a re-run is free - items(id, source_id, kind, platform_id, permalink, author_hash, created_utc, text, parent_id) · kind is post or comment - prefilter(item_id, matched_patterns_json, passed) · which cheap rule let this one through to a model - classifications(item_id, content_sha256, is_pain, severity, one_line, model_id, tokens_in, tokens_out, classified_at) · keyed by content hash so identical text is never paid for twice - clusters(run_id, id, name, size, mean_severity, recency_weight, score) · named by a model, scored by a formula you can read - cluster_quotes(cluster_id, item_id, ordinal, verbatim, permalink) · verbatim is asserted to appear in the item's text - runs(id, started_at, finished_at, sources_json, items_seen, items_classified, cost_estimate, cost_actual, threshold) - The classifications table is keyed by content hash rather than by item, so the same complaint quoted in three threads costs one call. ### Phase 1 · An ingester that is welcome back tomorrow Build: Register an application and authenticate the documented way. Put the platform's published rate limit in configuration with the date you read it, and enforce it in the client rather than hoping. Honour whatever rate-limit headers the responses carry and back off on 429 and 5xx. Store every raw response keyed by its request so a second scan reads from the cache. Trap: Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. Done when: A scan interrupted at any point resumes without refetching anything already cached, tested by killing it mid-scan and comparing request counts. The client refuses to exceed the configured rate, asserted by a test that counts requests per interval against a fake clock. A 429 response causes a backoff of at least the interval the response asks for, tested. The configuration records the rate limit, the source page and the date it was checked, and the scan prints them before starting. Do not build yet: No prefiltering, classification, clustering or dashboard yet. ### Phase 2 · Spend nothing on text that is not a complaint Build: Match complaint-shaped phrases with plain patterns: expressions of frustration, of giving up, of workarounds, of missing features. Record which patterns matched each item. Report the pass rate per source, and sample a hundred rejected items by hand to see what you are throwing away. Trap: Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. Done when: The pass rate per source is reported and a hand-reviewed sample of a hundred rejected items is recorded in the repository with a note on what was missed. Prefiltering a cached corpus runs without any network or model call, tested. Which patterns matched is stored per item, so the filter can be tuned from data rather than from intuition. A source whose pass rate is above a configured ceiling warns, because a filter that passes everything is not a filter. Do not build yet: No classification, clustering or dashboard yet. ### Phase 3 · Classification with a bill you predicted Build: Classify each item into is_pain, a severity and a one-line summary, keyed by content hash so identical text is paid for once. Estimate the cost before the run and refuse to start above the ceiling. Use the cheapest model that passes your own spot check, and batch the work when it is not urgent: Anthropic's published pricing puts its smallest current model at 1 US dollar per million input tokens and 5 per million output, and its batch processing at half those rates for asynchronous work (checked on their pricing page). Put whichever provider's figures you use in configuration with the date. Trap: Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. Done when: Re-running a scan with no new items makes zero model calls, tested by counting calls. Identical text appearing under two different items is classified once, tested. The run prints an estimated cost before the first call and refuses to start above the configured ceiling. A hand-labelled sample of fifty items is compared against the model's labels and the agreement rate is recorded, so the choice of model is a measurement rather than a preference. Do not build yet: No clustering, scoring or dashboard yet. ### Phase 4 · Clusters you would defend Build: Embed the one-line summaries and group them. Make the similarity threshold a parameter, sweep it over the corpus, and print the resulting cluster size distribution so you choose it by looking. Name each cluster with a model and select its strongest quotes, asserting that each quote appears verbatim in its item's text before storing it. Trap: Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. Done when: A threshold sweep over the corpus prints cluster counts and size distributions for at least five values, and the chosen value is recorded with a one-line reason. Every stored quote is asserted to appear verbatim in its item's text, and a deliberately altered quote fails the assertion in a test. Each cluster's score is computed from a formula written in the README, and changing one input changes the score in the direction the formula says. Re-running clustering on an unchanged corpus at the same threshold produces the same clusters. Do not build yet: No cross-run comparison, dashboard or export yet. ### Phase 5 · The tenth run, not the first Build: Match this run's clusters to previous runs by embedding similarity and shared items, and mark each as new, growing, shrinking or unchanged. The dashboard defaults to what changed. Export clusters, quotes and scores as CSV and JSON. State plainly in the README what this corpus does not contain, starting with communities you did not think to add. Trap: A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent. Done when: A second scan over the same sources shows clusters marked new, growing, shrinking or unchanged, and the default view shows only the first three. Cluster matching across runs survives a cluster being renamed by the model, tested by forcing a rename. Exports contain clusters, their scores, and their verbatim quotes with permanent links, and open correctly elsewhere. The README states that community selection is manual and that this tool cannot find a community you did not name. Do not build yet: Nothing after this. ### Out of scope (and why) - Sources beyond the one platform you start with, because a second platform means a second set of terms, limits and quirks before the first one is producing clusters you trust. - Automatic community discovery, which is the expensive half of the paid products and needs a corpus you do not have. - Idea generation and validation on top of the clusters, which is a separate product and a much weaker claim. - Anything requiring a logged-in session or evading a rate limit, which breaks the terms and ends the project. ### README must contain - The platform's published rate limit, the page it came from and the date it was checked, and that an aggressive client gets blocked - That the prefilter runs before any model call and what its measured pass rate is per source - How the classification cache is keyed on content hash, and the estimated cost per thousand items with the pricing source and date - The clustering threshold, how it was chosen, and the scoring formula in full - That community selection is manual, so this cannot surface a community you did not name ===== AGENTS.md ===== # Agent instructions · IdeaFast indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript, a CLI for scanning and a small local dashboard, SQLite via better-sqlite3, every raw response cached by request, Authenticated, rate-limited, resumable, with the limit read from the platform, A cheap model over prefiltered text, cached by content hash, Embeddings with a threshold you tuned and can explain. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. ===== BUILD_PLAN.md ===== # Build plan · IdeaFast A forum mining pipeline whose hard problems are ingesting politely and resumably within a platform's published limits, keeping the classification bill under the price of the subscription it replaces, and clustering in a way you would defend to someone who disagrees with the result. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · An ingester that is welcome back tomorrow Fetching is authenticated, under the published rate, resumable, and cached so a re-run costs nothing. ### Steps 1. Put the limit and its source in configuration With a date, because these policies change. Files: `src/config/limits.ts`, `docs/platform-policy.md` 2. Enforce the rate in the client A sleep in a loop is not a rate limiter when anything runs concurrently. Files: `src/fetch/limiter.ts`, `tests/rate-limit.test.ts` 3. Cache every raw response So the second scan is free and the parser can be fixed offline. Files: `src/fetch/cache.ts` ### Done when - [ ] A killed scan resumes without refetching - [ ] The client cannot exceed the configured rate, asserted - [ ] A 429 triggers at least the requested backoff - [ ] The limit, its source and check date print before a scan ### Watch out - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. ## Phase 2 · Spend nothing on text that is not a complaint Cheap rules reduce the corpus to complaint-shaped text before a single token is spent. ### Steps 1. Write patterns, then measure the pass rate The number that predicts your bill. Files: `src/prefilter.ts` 2. Read a hundred rejected items yourself The only way to know what the filter costs you in recall. Files: `docs/prefilter-review.md` 3. Store which patterns matched Tuning from data beats tuning from memory. Files: `src/prefilter.ts`, `tests/prefilter.test.ts` ### Done when - [ ] Pass rate is reported per source - [ ] A hundred rejected items are reviewed and recorded - [ ] Prefiltering runs with no network or model calls - [ ] An excessive pass rate raises a warning ### Watch out - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. ## Phase 3 · Classification with a bill you predicted Each surviving item is classified once, cached by content, and the run's cost is estimated before it starts. ### Steps 1. Key the cache on content hash The same complaint in three threads is one call. Files: `src/classify.ts`, `tests/classify-cache.test.ts` 2. Estimate before the run, refuse above the ceiling A ceiling checked after the fact is a receipt. Files: `src/cost.ts` 3. Hand-label fifty items and compare Model choice is a measurement you can afford to make. Files: `docs/classifier-agreement.md` ### Done when - [ ] A re-run with no new items makes zero calls - [ ] Duplicate text is classified once - [ ] An estimate prints and blocks above the ceiling - [ ] Agreement against fifty hand labels is recorded ### Watch out - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. ## Phase 4 · Clusters you would defend Themes are grouped at a threshold you chose deliberately, named clearly, and evidenced verbatim. ### Steps 1. Sweep the threshold and look at it Five values and a distribution, before choosing. Files: `scripts/threshold-sweep.ts`, `docs/clustering.md` 2. Assert every quote is verbatim String comparison against the source text, on write. Files: `src/cluster/quotes.ts`, `tests/verbatim.test.ts` 3. Write the scoring formula in the README A score nobody can recompute is a number with a halo. Files: `README.md`, `src/cluster/score.ts` ### Done when - [ ] A threshold sweep is printed and the choice recorded - [ ] An altered quote fails the verbatim assertion - [ ] Scores follow the documented formula - [ ] Clustering is reproducible on an unchanged corpus ### Watch out - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. ## Not in this build - Sources beyond the first platform. - Automatic community discovery. - Idea generation and validation. - Logged-in scraping or limit evasion. ## After v1, if you want it - A second platform added as an ingest module, once the first has run for a month without a 429 - Per-cluster alerting when a theme grows past a threshold, using data the compare step already produces - Storing embeddings for the raw items rather than the summaries, which changes clustering quality and needs the sweep repeated ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Raw responses, items, classifications and clusters. DATABASE_PATH=./data/pain.db # Required. The communities to scan. Manual, and that is the stated limitation. SOURCES_FILE=./sources.txt # Required. From your registered application. Identifies the client the rate limit applies to. PLATFORM_CLIENT_ID=your-app-client-id # Required · secret. The other half of the application credentials. PLATFORM_CLIENT_SECRET=...redacted # Required. Descriptive and contactable, as the platform's rules ask. USER_AGENT=pain-finder/0.1 (contact: you@example.com) # Required. Kept under the platform's published limit, which is recorded with its check date. REQUESTS_PER_MINUTE=60 # Optional · secret. Classification and naming. Unset means the pipeline still ingests and prefilters. ANTHROPIC_API_KEY=sk-ant-...redacted # Required. A run estimated above this refuses to start. RUN_COST_CEILING_USD=5
You are building a production product version of IdeaFast. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # IdeaFast · product brief ## Problem The pipeline is honest work an agent can do: pull public Reddit JSON, prefilter complaint-shaped text, classify with an LLM, embed and cluster, rank by frequency times severity times recency. A weekend gets you ranked pain themes with real permalinks for two or three subreddits you already know. What does not fall out of one session is everything after the demo: staying inside Reddit's rate limits at scale, picking which communities are worth scanning when you do not already know, deduping the same pain across runs so week two is not week one again, and keeping the LLM bill under the price of the subscription. Verdict is kinda, not yes, because the first run is easy and the tenth is where the product actually lives. ## Product outcome A miner that stays welcome on the platform it reads, costs what you predicted, groups complaints at a threshold you chose on purpose, and tells you on the tenth run what is actually new. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Anthropic or OpenAI API key - embedding model - SQLite - patience with Reddit rate limits ## Explicit non-goals for v1 - Sources beyond the first platform. - Automatic community discovery. - Idea generation and validation. - Logged-in scraping or limit evasion. - community discovery, you can only scan subreddits you already thought of - cross-scan dedupe, so repeat runs resurface the same pains as if they were new - a warmed corpus, every fresh scan pays the full ingestion wait - cost control, naive LLM classification of a busy subreddit gets expensive fast - the idea generation and validation layer on top of the raw clusters ## Success criteria - The client is confirmed unable to exceed the configured rate - A re-run with no new items is confirmed to make zero model calls - An altered quote is confirmed to fail the verbatim assertion - The threshold sweep and its chosen value are confirmed recorded - A second scan is confirmed to mark what changed ===== BRIEF.md ===== # Build brief · IdeaFast The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local pipeline that reads public forum complaints and turns them into ranked pain themes backed by quotes you can click to replace IdeaFast. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and the first one decides whether this project survives its second week. Read the platform's own developer terms and rate limits before writing the fetcher, then obey them. Register an application, authenticate the documented way, send a descriptive user agent, stay under the published request rate, and back off when the response tells you to. Reddit in particular publishes both a rate limit and an access policy that has changed more than once, so check the current pages rather than copying a number from a blog post, and record what you found and when. An aggressive client gets an application key revoked, and then the corpus stops growing for everyone using your code. Second, evidence is verbatim or it is not evidence. Every quote shown to a user is the exact text from the source with a working permanent link. A paraphrased complaint is an opinion with a citation stapled to it, and the whole value of this tool is that you can click through and read the actual person being actually annoyed. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a CLI for scanning and a small local dashboard · one repository, one scan command, no accounts - Storage: SQLite via better-sqlite3, every raw response cached by request · a re-run must cost nothing, or you will not re-run - Ingestion: Authenticated, rate-limited, resumable, with the limit read from the platform · the fetcher is the part that gets you blocked - Classification: A cheap model over prefiltered text, cached by content hash · the prefilter is the entire cost story - Clustering: Embeddings with a threshold you tuned and can explain · a threshold nobody looked at produces one blob or four hundred singletons ### Data model (create this before Phase 1) - sources(id, platform, community, added_at, last_scanned_at, notes) - raw_responses(id, source_id, request_url, fetched_at, status, body_sha256, body_path) · cached so a re-run is free - items(id, source_id, kind, platform_id, permalink, author_hash, created_utc, text, parent_id) · kind is post or comment - prefilter(item_id, matched_patterns_json, passed) · which cheap rule let this one through to a model - classifications(item_id, content_sha256, is_pain, severity, one_line, model_id, tokens_in, tokens_out, classified_at) · keyed by content hash so identical text is never paid for twice - clusters(run_id, id, name, size, mean_severity, recency_weight, score) · named by a model, scored by a formula you can read - cluster_quotes(cluster_id, item_id, ordinal, verbatim, permalink) · verbatim is asserted to appear in the item's text - runs(id, started_at, finished_at, sources_json, items_seen, items_classified, cost_estimate, cost_actual, threshold) - The classifications table is keyed by content hash rather than by item, so the same complaint quoted in three threads costs one call. ### Phase 1 · An ingester that is welcome back tomorrow Build: Register an application and authenticate the documented way. Put the platform's published rate limit in configuration with the date you read it, and enforce it in the client rather than hoping. Honour whatever rate-limit headers the responses carry and back off on 429 and 5xx. Store every raw response keyed by its request so a second scan reads from the cache. Trap: Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. Done when: A scan interrupted at any point resumes without refetching anything already cached, tested by killing it mid-scan and comparing request counts. The client refuses to exceed the configured rate, asserted by a test that counts requests per interval against a fake clock. A 429 response causes a backoff of at least the interval the response asks for, tested. The configuration records the rate limit, the source page and the date it was checked, and the scan prints them before starting. Do not build yet: No prefiltering, classification, clustering or dashboard yet. ### Phase 2 · Spend nothing on text that is not a complaint Build: Match complaint-shaped phrases with plain patterns: expressions of frustration, of giving up, of workarounds, of missing features. Record which patterns matched each item. Report the pass rate per source, and sample a hundred rejected items by hand to see what you are throwing away. Trap: Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. Done when: The pass rate per source is reported and a hand-reviewed sample of a hundred rejected items is recorded in the repository with a note on what was missed. Prefiltering a cached corpus runs without any network or model call, tested. Which patterns matched is stored per item, so the filter can be tuned from data rather than from intuition. A source whose pass rate is above a configured ceiling warns, because a filter that passes everything is not a filter. Do not build yet: No classification, clustering or dashboard yet. ### Phase 3 · Classification with a bill you predicted Build: Classify each item into is_pain, a severity and a one-line summary, keyed by content hash so identical text is paid for once. Estimate the cost before the run and refuse to start above the ceiling. Use the cheapest model that passes your own spot check, and batch the work when it is not urgent: Anthropic's published pricing puts its smallest current model at 1 US dollar per million input tokens and 5 per million output, and its batch processing at half those rates for asynchronous work (checked on their pricing page). Put whichever provider's figures you use in configuration with the date. Trap: Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. Done when: Re-running a scan with no new items makes zero model calls, tested by counting calls. Identical text appearing under two different items is classified once, tested. The run prints an estimated cost before the first call and refuses to start above the configured ceiling. A hand-labelled sample of fifty items is compared against the model's labels and the agreement rate is recorded, so the choice of model is a measurement rather than a preference. Do not build yet: No clustering, scoring or dashboard yet. ### Phase 4 · Clusters you would defend Build: Embed the one-line summaries and group them. Make the similarity threshold a parameter, sweep it over the corpus, and print the resulting cluster size distribution so you choose it by looking. Name each cluster with a model and select its strongest quotes, asserting that each quote appears verbatim in its item's text before storing it. Trap: Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. Done when: A threshold sweep over the corpus prints cluster counts and size distributions for at least five values, and the chosen value is recorded with a one-line reason. Every stored quote is asserted to appear verbatim in its item's text, and a deliberately altered quote fails the assertion in a test. Each cluster's score is computed from a formula written in the README, and changing one input changes the score in the direction the formula says. Re-running clustering on an unchanged corpus at the same threshold produces the same clusters. Do not build yet: No cross-run comparison, dashboard or export yet. ### Phase 5 · The tenth run, not the first Build: Match this run's clusters to previous runs by embedding similarity and shared items, and mark each as new, growing, shrinking or unchanged. The dashboard defaults to what changed. Export clusters, quotes and scores as CSV and JSON. State plainly in the README what this corpus does not contain, starting with communities you did not think to add. Trap: A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent. Done when: A second scan over the same sources shows clusters marked new, growing, shrinking or unchanged, and the default view shows only the first three. Cluster matching across runs survives a cluster being renamed by the model, tested by forcing a rename. Exports contain clusters, their scores, and their verbatim quotes with permanent links, and open correctly elsewhere. The README states that community selection is manual and that this tool cannot find a community you did not name. Do not build yet: Nothing after this. ### Out of scope (and why) - Sources beyond the one platform you start with, because a second platform means a second set of terms, limits and quirks before the first one is producing clusters you trust. - Automatic community discovery, which is the expensive half of the paid products and needs a corpus you do not have. - Idea generation and validation on top of the clusters, which is a separate product and a much weaker claim. - Anything requiring a logged-in session or evading a rate limit, which breaks the terms and ends the project. ### README must contain - The platform's published rate limit, the page it came from and the date it was checked, and that an aggressive client gets blocked - That the prefilter runs before any model call and what its measured pass rate is per source - How the classification cache is keyed on content hash, and the estimated cost per thousand items with the pricing source and date - The clustering threshold, how it was chosen, and the scoring formula in full - That community selection is manual, so this cannot surface a community you did not name ===== ARCHITECTURE.md ===== # Architecture · IdeaFast ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a CLI for scanning and a small local dashboard | one repository, one scan command, no accounts | | Storage | SQLite via better-sqlite3, every raw response cached by request | a re-run must cost nothing, or you will not re-run | | Ingestion | Authenticated, rate-limited, resumable, with the limit read from the platform | the fetcher is the part that gets you blocked | | Classification | A cheap model over prefiltered text, cached by content hash | the prefilter is the entire cost story | | Clustering | Embeddings with a threshold you tuned and can explain | a threshold nobody looked at produces one blob or four hundred singletons | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Ingest | Authentication, rate limiting, caching and resume. | Any platform, provided its terms and limits are read and recorded first. | | Prefilter | Cheap rules and the measured pass rate. | Any rules, provided the pass rate is measured and a rejected sample is reviewed. | | Classify | Content-hash cached labelling and the cost ceiling. | Any model, provided the agreement rate against hand labels is recomputed. | | Cluster | Threshold, naming, verbatim quotes and scoring. | Any embedding model, provided the threshold is swept again. | | Compare | Cross-run matching and the changed view. | Any matching method, provided a renamed cluster still matches. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Raw responses, items, classifications and clusters. - `SOURCES_FILE` · required · The communities to scan. Manual, and that is the stated limitation. - `PLATFORM_CLIENT_ID` · required · From your registered application. Identifies the client the rate limit applies to. - `PLATFORM_CLIENT_SECRET` · required, secret · The other half of the application credentials. - `USER_AGENT` · required · Descriptive and contactable, as the platform's rules ask. - `REQUESTS_PER_MINUTE` · required · Kept under the platform's published limit, which is recorded with its check date. - `ANTHROPIC_API_KEY` · optional, secret · Classification and naming. Unset means the pipeline still ingests and prefilters. - `RUN_COST_CEILING_USD` · required · A run estimated above this refuses to start. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · IdeaFast product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with TypeScript, a CLI for scanning and a small local dashboard, SQLite via better-sqlite3, every raw response cached by request, Authenticated, rate-limited, resumable, with the limit read from the platform, A cheap model over prefiltered text, cached by content hash, Embeddings with a threshold you tuned and can explain. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. - A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent. ===== MILESTONES.md ===== # Delivery milestones · IdeaFast Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · An ingester that is welcome back tomorrow Fetching is authenticated, under the published rate, resumable, and cached so a re-run costs nothing. ### Steps 1. Put the limit and its source in configuration With a date, because these policies change. Files: `src/config/limits.ts`, `docs/platform-policy.md` 2. Enforce the rate in the client A sleep in a loop is not a rate limiter when anything runs concurrently. Files: `src/fetch/limiter.ts`, `tests/rate-limit.test.ts` 3. Cache every raw response So the second scan is free and the parser can be fixed offline. Files: `src/fetch/cache.ts` ### Done when - [ ] A killed scan resumes without refetching - [ ] The client cannot exceed the configured rate, asserted - [ ] A 429 triggers at least the requested backoff - [ ] The limit, its source and check date print before a scan ### Watch out - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. ## M2 · Spend nothing on text that is not a complaint Cheap rules reduce the corpus to complaint-shaped text before a single token is spent. ### Steps 1. Write patterns, then measure the pass rate The number that predicts your bill. Files: `src/prefilter.ts` 2. Read a hundred rejected items yourself The only way to know what the filter costs you in recall. Files: `docs/prefilter-review.md` 3. Store which patterns matched Tuning from data beats tuning from memory. Files: `src/prefilter.ts`, `tests/prefilter.test.ts` ### Done when - [ ] Pass rate is reported per source - [ ] A hundred rejected items are reviewed and recorded - [ ] Prefiltering runs with no network or model calls - [ ] An excessive pass rate raises a warning ### Watch out - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. ## M3 · Classification with a bill you predicted Each surviving item is classified once, cached by content, and the run's cost is estimated before it starts. ### Steps 1. Key the cache on content hash The same complaint in three threads is one call. Files: `src/classify.ts`, `tests/classify-cache.test.ts` 2. Estimate before the run, refuse above the ceiling A ceiling checked after the fact is a receipt. Files: `src/cost.ts` 3. Hand-label fifty items and compare Model choice is a measurement you can afford to make. Files: `docs/classifier-agreement.md` ### Done when - [ ] A re-run with no new items makes zero calls - [ ] Duplicate text is classified once - [ ] An estimate prints and blocks above the ceiling - [ ] Agreement against fifty hand labels is recorded ### Watch out - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. ## M4 · Clusters you would defend Themes are grouped at a threshold you chose deliberately, named clearly, and evidenced verbatim. ### Steps 1. Sweep the threshold and look at it Five values and a distribution, before choosing. Files: `scripts/threshold-sweep.ts`, `docs/clustering.md` 2. Assert every quote is verbatim String comparison against the source text, on write. Files: `src/cluster/quotes.ts`, `tests/verbatim.test.ts` 3. Write the scoring formula in the README A score nobody can recompute is a number with a halo. Files: `README.md`, `src/cluster/score.ts` ### Done when - [ ] A threshold sweep is printed and the choice recorded - [ ] An altered quote fails the verbatim assertion - [ ] Scores follow the documented formula - [ ] Clustering is reproducible on an unchanged corpus ### Watch out - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. ## M5 · The tenth run, not the first (production only) A later scan shows what is new and what moved, rather than presenting last month's themes again. ### Steps 1. Match clusters across runs By shared items and embedding similarity, not by name. Files: `src/cluster/match.ts`, `tests/cross-run-match.test.ts` 2. Default the dashboard to what changed The reason to open it a second time. Files: `src/views/changes.tsx` 3. Say what the corpus is missing Manual community selection is the real limitation here. Files: `README.md` ### Done when - [ ] A second scan marks new, growing, shrinking and unchanged - [ ] Matching survives a model-renamed cluster - [ ] Exports carry scores and verbatim permalinks - [ ] The README names the community selection limitation ### Watch out - A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent. ===== OPERATIONS.md ===== # Operations · IdeaFast ## Backup The database, because the raw response cache is what makes every re-run free and refetching it would mean going back to the platform for data you already politely collected once. ## Restore Restore and run a scan: it should make almost no network calls and no model calls, which confirms both caches survived. If it starts refetching, the cache key changed and that is a bug. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Requests per minute against the configured limit, 429 counts, pass rate per source, model calls per run against the estimate, and the age of the recorded rate limit and prices. A rising 429 count means you are close to being unwelcome. ## Incident checklist Being blocked is the incident this design exists to avoid, and the response is to stop, read the current policy, and correct the configured limit before running again. A cost surprise is diagnosed from the classifications table, which records tokens per call. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The client is confirmed unable to exceed the configured rate - [ ] A re-run with no new items is confirmed to make zero model calls - [ ] An altered quote is confirmed to fail the verbatim assertion - [ ] The threshold sweep and its chosen value are confirmed recorded - [ ] A second scan is confirmed to mark what changed ## Launch constraint Do not market omitted IdeaFast capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Raw responses, items, classifications and clusters. DATABASE_PATH=./data/pain.db # Required. The communities to scan. Manual, and that is the stated limitation. SOURCES_FILE=./sources.txt # Required. From your registered application. Identifies the client the rate limit applies to. PLATFORM_CLIENT_ID=your-app-client-id # Required · secret. The other half of the application credentials. PLATFORM_CLIENT_SECRET=...redacted # Required. Descriptive and contactable, as the platform's rules ask. USER_AGENT=pain-finder/0.1 (contact: you@example.com) # Required. Kept under the platform's published limit, which is recorded with its check date. REQUESTS_PER_MINUTE=60 # Optional · secret. Classification and naming. Unset means the pipeline still ingests and prefilters. ANTHROPIC_API_KEY=sk-ant-...redacted # Required. A run estimated above this refuses to start. RUN_COST_CEILING_USD=5
README.md
# IdeaFast · indie build A forum mining pipeline whose hard problems are ingesting politely and resumably within a platform's published limits, keeping the classification bill under the price of the subscription it replaces, and clustering in a way you would defend to someone who disagrees with the result. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a CLI for scanning and a small local dashboard | one repository, one scan command, no accounts | | Storage | SQLite via better-sqlite3, every raw response cached by request | a re-run must cost nothing, or you will not re-run | | Ingestion | Authenticated, rate-limited, resumable, with the limit read from the platform | the fetcher is the part that gets you blocked | | Classification | A cheap model over prefiltered text, cached by content hash | the prefilter is the entire cost story | | Clustering | Embeddings with a threshold you tuned and can explain | a threshold nobody looked at produces one blob or four hundred singletons | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the scanner and the dashboard. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A registered application on the platform you are reading** · free for non-commercial use on most platforms, but read the current terms - Why: Authenticated access is what the published rate limit applies to, and what makes your client identifiable rather than anonymous traffic to be throttled. - Get it: Register an application in the platform's developer settings and note the credentials and the terms you accepted. - Verify: An authenticated request succeeds and the response carries rate-limit information - [ ] **A model API key for classification and embeddings** · pay per use, and the prefilter is what keeps it small - Why: Classification and embedding are the only paid parts, and the key is what the cost ceiling is protecting. - Get it: Create an account with a provider and generate a key, then record the current prices with today's date. - Verify: A ten item classification run completes and the ledger shows tokens - [ ] **The current pricing page for your chosen model** · free - Why: The cost estimate is only useful if the rates behind it are today's, and every provider in this space has changed them. - Get it: Open the provider's pricing page, record input and output rates and any batch discount, with the date. - Verify: The configuration holds the rates and the date, and the estimate uses them - [ ] **Fifty items you have labelled by hand** · free, an hour of reading - Why: The severity scale means whatever the model decides unless you have your own labels to compare against, and fifty is enough to catch a classifier that is systematically wrong. - Get it: Read fifty prefiltered items and mark is_pain and severity yourself before looking at any model output. - Verify: The labels exist in the repository and the agreement rate is computed ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Sources beyond the first platform. - Automatic community discovery. - Idea generation and validation. - Logged-in scraping or limit evasion. - community discovery, you can only scan subreddits you already thought of - cross-scan dedupe, so repeat runs resurface the same pains as if they were new - a warmed corpus, every fresh scan pays the full ingestion wait - cost control, naive LLM classification of a busy subreddit gets expensive fast - the idea generation and validation layer on top of the raw clusters If one of those is essential to you, that is the reason to keep paying for IdeaFast, and the README should say so rather than pretend.
BRIEF.md
# Build brief · IdeaFast The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local pipeline that reads public forum complaints and turns them into ranked pain themes backed by quotes you can click to replace IdeaFast. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and the first one decides whether this project survives its second week. Read the platform's own developer terms and rate limits before writing the fetcher, then obey them. Register an application, authenticate the documented way, send a descriptive user agent, stay under the published request rate, and back off when the response tells you to. Reddit in particular publishes both a rate limit and an access policy that has changed more than once, so check the current pages rather than copying a number from a blog post, and record what you found and when. An aggressive client gets an application key revoked, and then the corpus stops growing for everyone using your code. Second, evidence is verbatim or it is not evidence. Every quote shown to a user is the exact text from the source with a working permanent link. A paraphrased complaint is an opinion with a citation stapled to it, and the whole value of this tool is that you can click through and read the actual person being actually annoyed. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a CLI for scanning and a small local dashboard · one repository, one scan command, no accounts - Storage: SQLite via better-sqlite3, every raw response cached by request · a re-run must cost nothing, or you will not re-run - Ingestion: Authenticated, rate-limited, resumable, with the limit read from the platform · the fetcher is the part that gets you blocked - Classification: A cheap model over prefiltered text, cached by content hash · the prefilter is the entire cost story - Clustering: Embeddings with a threshold you tuned and can explain · a threshold nobody looked at produces one blob or four hundred singletons ### Data model (create this before Phase 1) - sources(id, platform, community, added_at, last_scanned_at, notes) - raw_responses(id, source_id, request_url, fetched_at, status, body_sha256, body_path) · cached so a re-run is free - items(id, source_id, kind, platform_id, permalink, author_hash, created_utc, text, parent_id) · kind is post or comment - prefilter(item_id, matched_patterns_json, passed) · which cheap rule let this one through to a model - classifications(item_id, content_sha256, is_pain, severity, one_line, model_id, tokens_in, tokens_out, classified_at) · keyed by content hash so identical text is never paid for twice - clusters(run_id, id, name, size, mean_severity, recency_weight, score) · named by a model, scored by a formula you can read - cluster_quotes(cluster_id, item_id, ordinal, verbatim, permalink) · verbatim is asserted to appear in the item's text - runs(id, started_at, finished_at, sources_json, items_seen, items_classified, cost_estimate, cost_actual, threshold) - The classifications table is keyed by content hash rather than by item, so the same complaint quoted in three threads costs one call. ### Phase 1 · An ingester that is welcome back tomorrow Build: Register an application and authenticate the documented way. Put the platform's published rate limit in configuration with the date you read it, and enforce it in the client rather than hoping. Honour whatever rate-limit headers the responses carry and back off on 429 and 5xx. Store every raw response keyed by its request so a second scan reads from the cache. Trap: Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. Done when: A scan interrupted at any point resumes without refetching anything already cached, tested by killing it mid-scan and comparing request counts. The client refuses to exceed the configured rate, asserted by a test that counts requests per interval against a fake clock. A 429 response causes a backoff of at least the interval the response asks for, tested. The configuration records the rate limit, the source page and the date it was checked, and the scan prints them before starting. Do not build yet: No prefiltering, classification, clustering or dashboard yet. ### Phase 2 · Spend nothing on text that is not a complaint Build: Match complaint-shaped phrases with plain patterns: expressions of frustration, of giving up, of workarounds, of missing features. Record which patterns matched each item. Report the pass rate per source, and sample a hundred rejected items by hand to see what you are throwing away. Trap: Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. Done when: The pass rate per source is reported and a hand-reviewed sample of a hundred rejected items is recorded in the repository with a note on what was missed. Prefiltering a cached corpus runs without any network or model call, tested. Which patterns matched is stored per item, so the filter can be tuned from data rather than from intuition. A source whose pass rate is above a configured ceiling warns, because a filter that passes everything is not a filter. Do not build yet: No classification, clustering or dashboard yet. ### Phase 3 · Classification with a bill you predicted Build: Classify each item into is_pain, a severity and a one-line summary, keyed by content hash so identical text is paid for once. Estimate the cost before the run and refuse to start above the ceiling. Use the cheapest model that passes your own spot check, and batch the work when it is not urgent: Anthropic's published pricing puts its smallest current model at 1 US dollar per million input tokens and 5 per million output, and its batch processing at half those rates for asynchronous work (checked on their pricing page). Put whichever provider's figures you use in configuration with the date. Trap: Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. Done when: Re-running a scan with no new items makes zero model calls, tested by counting calls. Identical text appearing under two different items is classified once, tested. The run prints an estimated cost before the first call and refuses to start above the configured ceiling. A hand-labelled sample of fifty items is compared against the model's labels and the agreement rate is recorded, so the choice of model is a measurement rather than a preference. Do not build yet: No clustering, scoring or dashboard yet. ### Phase 4 · Clusters you would defend Build: Embed the one-line summaries and group them. Make the similarity threshold a parameter, sweep it over the corpus, and print the resulting cluster size distribution so you choose it by looking. Name each cluster with a model and select its strongest quotes, asserting that each quote appears verbatim in its item's text before storing it. Trap: Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. Done when: A threshold sweep over the corpus prints cluster counts and size distributions for at least five values, and the chosen value is recorded with a one-line reason. Every stored quote is asserted to appear verbatim in its item's text, and a deliberately altered quote fails the assertion in a test. Each cluster's score is computed from a formula written in the README, and changing one input changes the score in the direction the formula says. Re-running clustering on an unchanged corpus at the same threshold produces the same clusters. Do not build yet: No cross-run comparison, dashboard or export yet. ### Phase 5 · The tenth run, not the first Build: Match this run's clusters to previous runs by embedding similarity and shared items, and mark each as new, growing, shrinking or unchanged. The dashboard defaults to what changed. Export clusters, quotes and scores as CSV and JSON. State plainly in the README what this corpus does not contain, starting with communities you did not think to add. Trap: A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent. Done when: A second scan over the same sources shows clusters marked new, growing, shrinking or unchanged, and the default view shows only the first three. Cluster matching across runs survives a cluster being renamed by the model, tested by forcing a rename. Exports contain clusters, their scores, and their verbatim quotes with permanent links, and open correctly elsewhere. The README states that community selection is manual and that this tool cannot find a community you did not name. Do not build yet: Nothing after this. ### Out of scope (and why) - Sources beyond the one platform you start with, because a second platform means a second set of terms, limits and quirks before the first one is producing clusters you trust. - Automatic community discovery, which is the expensive half of the paid products and needs a corpus you do not have. - Idea generation and validation on top of the clusters, which is a separate product and a much weaker claim. - Anything requiring a logged-in session or evading a rate limit, which breaks the terms and ends the project. ### README must contain - The platform's published rate limit, the page it came from and the date it was checked, and that an aggressive client gets blocked - That the prefilter runs before any model call and what its measured pass rate is per source - How the classification cache is keyed on content hash, and the estimated cost per thousand items with the pricing source and date - The clustering threshold, how it was chosen, and the scoring formula in full - That community selection is manual, so this cannot surface a community you did not name
AGENTS.md
# Agent instructions · IdeaFast indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript, a CLI for scanning and a small local dashboard, SQLite via better-sqlite3, every raw response cached by request, Authenticated, rate-limited, resumable, with the limit read from the platform, A cheap model over prefiltered text, cached by content hash, Embeddings with a threshold you tuned and can explain. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you.
BUILD_PLAN.md
# Build plan · IdeaFast A forum mining pipeline whose hard problems are ingesting politely and resumably within a platform's published limits, keeping the classification bill under the price of the subscription it replaces, and clustering in a way you would defend to someone who disagrees with the result. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · An ingester that is welcome back tomorrow Fetching is authenticated, under the published rate, resumable, and cached so a re-run costs nothing. ### Steps 1. Put the limit and its source in configuration With a date, because these policies change. Files: `src/config/limits.ts`, `docs/platform-policy.md` 2. Enforce the rate in the client A sleep in a loop is not a rate limiter when anything runs concurrently. Files: `src/fetch/limiter.ts`, `tests/rate-limit.test.ts` 3. Cache every raw response So the second scan is free and the parser can be fixed offline. Files: `src/fetch/cache.ts` ### Done when - [ ] A killed scan resumes without refetching - [ ] The client cannot exceed the configured rate, asserted - [ ] A 429 triggers at least the requested backoff - [ ] The limit, its source and check date print before a scan ### Watch out - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. ## Phase 2 · Spend nothing on text that is not a complaint Cheap rules reduce the corpus to complaint-shaped text before a single token is spent. ### Steps 1. Write patterns, then measure the pass rate The number that predicts your bill. Files: `src/prefilter.ts` 2. Read a hundred rejected items yourself The only way to know what the filter costs you in recall. Files: `docs/prefilter-review.md` 3. Store which patterns matched Tuning from data beats tuning from memory. Files: `src/prefilter.ts`, `tests/prefilter.test.ts` ### Done when - [ ] Pass rate is reported per source - [ ] A hundred rejected items are reviewed and recorded - [ ] Prefiltering runs with no network or model calls - [ ] An excessive pass rate raises a warning ### Watch out - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. ## Phase 3 · Classification with a bill you predicted Each surviving item is classified once, cached by content, and the run's cost is estimated before it starts. ### Steps 1. Key the cache on content hash The same complaint in three threads is one call. Files: `src/classify.ts`, `tests/classify-cache.test.ts` 2. Estimate before the run, refuse above the ceiling A ceiling checked after the fact is a receipt. Files: `src/cost.ts` 3. Hand-label fifty items and compare Model choice is a measurement you can afford to make. Files: `docs/classifier-agreement.md` ### Done when - [ ] A re-run with no new items makes zero calls - [ ] Duplicate text is classified once - [ ] An estimate prints and blocks above the ceiling - [ ] Agreement against fifty hand labels is recorded ### Watch out - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. ## Phase 4 · Clusters you would defend Themes are grouped at a threshold you chose deliberately, named clearly, and evidenced verbatim. ### Steps 1. Sweep the threshold and look at it Five values and a distribution, before choosing. Files: `scripts/threshold-sweep.ts`, `docs/clustering.md` 2. Assert every quote is verbatim String comparison against the source text, on write. Files: `src/cluster/quotes.ts`, `tests/verbatim.test.ts` 3. Write the scoring formula in the README A score nobody can recompute is a number with a halo. Files: `README.md`, `src/cluster/score.ts` ### Done when - [ ] A threshold sweep is printed and the choice recorded - [ ] An altered quote fails the verbatim assertion - [ ] Scores follow the documented formula - [ ] Clustering is reproducible on an unchanged corpus ### Watch out - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. ## Not in this build - Sources beyond the first platform. - Automatic community discovery. - Idea generation and validation. - Logged-in scraping or limit evasion. ## After v1, if you want it - A second platform added as an ingest module, once the first has run for a month without a 429 - Per-cluster alerting when a theme grows past a threshold, using data the compare step already produces - Storing embeddings for the raw items rather than the summaries, which changes clustering quality and needs the sweep repeated
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Raw responses, items, classifications and clusters. DATABASE_PATH=./data/pain.db # Required. The communities to scan. Manual, and that is the stated limitation. SOURCES_FILE=./sources.txt # Required. From your registered application. Identifies the client the rate limit applies to. PLATFORM_CLIENT_ID=your-app-client-id # Required · secret. The other half of the application credentials. PLATFORM_CLIENT_SECRET=...redacted # Required. Descriptive and contactable, as the platform's rules ask. USER_AGENT=pain-finder/0.1 (contact: you@example.com) # Required. Kept under the platform's published limit, which is recorded with its check date. REQUESTS_PER_MINUTE=60 # Optional · secret. Classification and naming. Unset means the pipeline still ingests and prefilters. ANTHROPIC_API_KEY=sk-ant-...redacted # Required. A run estimated above this refuses to start. RUN_COST_CEILING_USD=5
PRODUCT.md
# IdeaFast · product brief ## Problem The pipeline is honest work an agent can do: pull public Reddit JSON, prefilter complaint-shaped text, classify with an LLM, embed and cluster, rank by frequency times severity times recency. A weekend gets you ranked pain themes with real permalinks for two or three subreddits you already know. What does not fall out of one session is everything after the demo: staying inside Reddit's rate limits at scale, picking which communities are worth scanning when you do not already know, deduping the same pain across runs so week two is not week one again, and keeping the LLM bill under the price of the subscription. Verdict is kinda, not yes, because the first run is easy and the tenth is where the product actually lives. ## Product outcome A miner that stays welcome on the platform it reads, costs what you predicted, groups complaints at a threshold you chose on purpose, and tells you on the tenth run what is actually new. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Anthropic or OpenAI API key - embedding model - SQLite - patience with Reddit rate limits ## Explicit non-goals for v1 - Sources beyond the first platform. - Automatic community discovery. - Idea generation and validation. - Logged-in scraping or limit evasion. - community discovery, you can only scan subreddits you already thought of - cross-scan dedupe, so repeat runs resurface the same pains as if they were new - a warmed corpus, every fresh scan pays the full ingestion wait - cost control, naive LLM classification of a busy subreddit gets expensive fast - the idea generation and validation layer on top of the raw clusters ## Success criteria - The client is confirmed unable to exceed the configured rate - A re-run with no new items is confirmed to make zero model calls - An altered quote is confirmed to fail the verbatim assertion - The threshold sweep and its chosen value are confirmed recorded - A second scan is confirmed to mark what changed
BRIEF.md
# Build brief · IdeaFast The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local pipeline that reads public forum complaints and turns them into ranked pain themes backed by quotes you can click to replace IdeaFast. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and the first one decides whether this project survives its second week. Read the platform's own developer terms and rate limits before writing the fetcher, then obey them. Register an application, authenticate the documented way, send a descriptive user agent, stay under the published request rate, and back off when the response tells you to. Reddit in particular publishes both a rate limit and an access policy that has changed more than once, so check the current pages rather than copying a number from a blog post, and record what you found and when. An aggressive client gets an application key revoked, and then the corpus stops growing for everyone using your code. Second, evidence is verbatim or it is not evidence. Every quote shown to a user is the exact text from the source with a working permanent link. A paraphrased complaint is an opinion with a citation stapled to it, and the whole value of this tool is that you can click through and read the actual person being actually annoyed. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript, a CLI for scanning and a small local dashboard · one repository, one scan command, no accounts - Storage: SQLite via better-sqlite3, every raw response cached by request · a re-run must cost nothing, or you will not re-run - Ingestion: Authenticated, rate-limited, resumable, with the limit read from the platform · the fetcher is the part that gets you blocked - Classification: A cheap model over prefiltered text, cached by content hash · the prefilter is the entire cost story - Clustering: Embeddings with a threshold you tuned and can explain · a threshold nobody looked at produces one blob or four hundred singletons ### Data model (create this before Phase 1) - sources(id, platform, community, added_at, last_scanned_at, notes) - raw_responses(id, source_id, request_url, fetched_at, status, body_sha256, body_path) · cached so a re-run is free - items(id, source_id, kind, platform_id, permalink, author_hash, created_utc, text, parent_id) · kind is post or comment - prefilter(item_id, matched_patterns_json, passed) · which cheap rule let this one through to a model - classifications(item_id, content_sha256, is_pain, severity, one_line, model_id, tokens_in, tokens_out, classified_at) · keyed by content hash so identical text is never paid for twice - clusters(run_id, id, name, size, mean_severity, recency_weight, score) · named by a model, scored by a formula you can read - cluster_quotes(cluster_id, item_id, ordinal, verbatim, permalink) · verbatim is asserted to appear in the item's text - runs(id, started_at, finished_at, sources_json, items_seen, items_classified, cost_estimate, cost_actual, threshold) - The classifications table is keyed by content hash rather than by item, so the same complaint quoted in three threads costs one call. ### Phase 1 · An ingester that is welcome back tomorrow Build: Register an application and authenticate the documented way. Put the platform's published rate limit in configuration with the date you read it, and enforce it in the client rather than hoping. Honour whatever rate-limit headers the responses carry and back off on 429 and 5xx. Store every raw response keyed by its request so a second scan reads from the cache. Trap: Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. Done when: A scan interrupted at any point resumes without refetching anything already cached, tested by killing it mid-scan and comparing request counts. The client refuses to exceed the configured rate, asserted by a test that counts requests per interval against a fake clock. A 429 response causes a backoff of at least the interval the response asks for, tested. The configuration records the rate limit, the source page and the date it was checked, and the scan prints them before starting. Do not build yet: No prefiltering, classification, clustering or dashboard yet. ### Phase 2 · Spend nothing on text that is not a complaint Build: Match complaint-shaped phrases with plain patterns: expressions of frustration, of giving up, of workarounds, of missing features. Record which patterns matched each item. Report the pass rate per source, and sample a hundred rejected items by hand to see what you are throwing away. Trap: Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. Done when: The pass rate per source is reported and a hand-reviewed sample of a hundred rejected items is recorded in the repository with a note on what was missed. Prefiltering a cached corpus runs without any network or model call, tested. Which patterns matched is stored per item, so the filter can be tuned from data rather than from intuition. A source whose pass rate is above a configured ceiling warns, because a filter that passes everything is not a filter. Do not build yet: No classification, clustering or dashboard yet. ### Phase 3 · Classification with a bill you predicted Build: Classify each item into is_pain, a severity and a one-line summary, keyed by content hash so identical text is paid for once. Estimate the cost before the run and refuse to start above the ceiling. Use the cheapest model that passes your own spot check, and batch the work when it is not urgent: Anthropic's published pricing puts its smallest current model at 1 US dollar per million input tokens and 5 per million output, and its batch processing at half those rates for asynchronous work (checked on their pricing page). Put whichever provider's figures you use in configuration with the date. Trap: Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. Done when: Re-running a scan with no new items makes zero model calls, tested by counting calls. Identical text appearing under two different items is classified once, tested. The run prints an estimated cost before the first call and refuses to start above the configured ceiling. A hand-labelled sample of fifty items is compared against the model's labels and the agreement rate is recorded, so the choice of model is a measurement rather than a preference. Do not build yet: No clustering, scoring or dashboard yet. ### Phase 4 · Clusters you would defend Build: Embed the one-line summaries and group them. Make the similarity threshold a parameter, sweep it over the corpus, and print the resulting cluster size distribution so you choose it by looking. Name each cluster with a model and select its strongest quotes, asserting that each quote appears verbatim in its item's text before storing it. Trap: Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. Done when: A threshold sweep over the corpus prints cluster counts and size distributions for at least five values, and the chosen value is recorded with a one-line reason. Every stored quote is asserted to appear verbatim in its item's text, and a deliberately altered quote fails the assertion in a test. Each cluster's score is computed from a formula written in the README, and changing one input changes the score in the direction the formula says. Re-running clustering on an unchanged corpus at the same threshold produces the same clusters. Do not build yet: No cross-run comparison, dashboard or export yet. ### Phase 5 · The tenth run, not the first Build: Match this run's clusters to previous runs by embedding similarity and shared items, and mark each as new, growing, shrinking or unchanged. The dashboard defaults to what changed. Export clusters, quotes and scores as CSV and JSON. State plainly in the README what this corpus does not contain, starting with communities you did not think to add. Trap: A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent. Done when: A second scan over the same sources shows clusters marked new, growing, shrinking or unchanged, and the default view shows only the first three. Cluster matching across runs survives a cluster being renamed by the model, tested by forcing a rename. Exports contain clusters, their scores, and their verbatim quotes with permanent links, and open correctly elsewhere. The README states that community selection is manual and that this tool cannot find a community you did not name. Do not build yet: Nothing after this. ### Out of scope (and why) - Sources beyond the one platform you start with, because a second platform means a second set of terms, limits and quirks before the first one is producing clusters you trust. - Automatic community discovery, which is the expensive half of the paid products and needs a corpus you do not have. - Idea generation and validation on top of the clusters, which is a separate product and a much weaker claim. - Anything requiring a logged-in session or evading a rate limit, which breaks the terms and ends the project. ### README must contain - The platform's published rate limit, the page it came from and the date it was checked, and that an aggressive client gets blocked - That the prefilter runs before any model call and what its measured pass rate is per source - How the classification cache is keyed on content hash, and the estimated cost per thousand items with the pricing source and date - The clustering threshold, how it was chosen, and the scoring formula in full - That community selection is manual, so this cannot surface a community you did not name
ARCHITECTURE.md
# Architecture · IdeaFast ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript, a CLI for scanning and a small local dashboard | one repository, one scan command, no accounts | | Storage | SQLite via better-sqlite3, every raw response cached by request | a re-run must cost nothing, or you will not re-run | | Ingestion | Authenticated, rate-limited, resumable, with the limit read from the platform | the fetcher is the part that gets you blocked | | Classification | A cheap model over prefiltered text, cached by content hash | the prefilter is the entire cost story | | Clustering | Embeddings with a threshold you tuned and can explain | a threshold nobody looked at produces one blob or four hundred singletons | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Ingest | Authentication, rate limiting, caching and resume. | Any platform, provided its terms and limits are read and recorded first. | | Prefilter | Cheap rules and the measured pass rate. | Any rules, provided the pass rate is measured and a rejected sample is reviewed. | | Classify | Content-hash cached labelling and the cost ceiling. | Any model, provided the agreement rate against hand labels is recomputed. | | Cluster | Threshold, naming, verbatim quotes and scoring. | Any embedding model, provided the threshold is swept again. | | Compare | Cross-run matching and the changed view. | Any matching method, provided a renamed cluster still matches. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Raw responses, items, classifications and clusters. - `SOURCES_FILE` · required · The communities to scan. Manual, and that is the stated limitation. - `PLATFORM_CLIENT_ID` · required · From your registered application. Identifies the client the rate limit applies to. - `PLATFORM_CLIENT_SECRET` · required, secret · The other half of the application credentials. - `USER_AGENT` · required · Descriptive and contactable, as the platform's rules ask. - `REQUESTS_PER_MINUTE` · required · Kept under the platform's published limit, which is recorded with its check date. - `ANTHROPIC_API_KEY` · optional, secret · Classification and naming. Unset means the pipeline still ingests and prefilters. - `RUN_COST_CEILING_USD` · required · A run estimated above this refuses to start. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · IdeaFast product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with TypeScript, a CLI for scanning and a small local dashboard, SQLite via better-sqlite3, every raw response cached by request, Authenticated, rate-limited, resumable, with the limit read from the platform, A cheap model over prefiltered text, cached by content hash, Embeddings with a threshold you tuned and can explain. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. - A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent.
MILESTONES.md
# Delivery milestones · IdeaFast Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · An ingester that is welcome back tomorrow Fetching is authenticated, under the published rate, resumable, and cached so a re-run costs nothing. ### Steps 1. Put the limit and its source in configuration With a date, because these policies change. Files: `src/config/limits.ts`, `docs/platform-policy.md` 2. Enforce the rate in the client A sleep in a loop is not a rate limiter when anything runs concurrently. Files: `src/fetch/limiter.ts`, `tests/rate-limit.test.ts` 3. Cache every raw response So the second scan is free and the parser can be fixed offline. Files: `src/fetch/cache.ts` ### Done when - [ ] A killed scan resumes without refetching - [ ] The client cannot exceed the configured rate, asserted - [ ] A 429 triggers at least the requested backoff - [ ] The limit, its source and check date print before a scan ### Watch out - Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup. ## M2 · Spend nothing on text that is not a complaint Cheap rules reduce the corpus to complaint-shaped text before a single token is spent. ### Steps 1. Write patterns, then measure the pass rate The number that predicts your bill. Files: `src/prefilter.ts` 2. Read a hundred rejected items yourself The only way to know what the filter costs you in recall. Files: `docs/prefilter-review.md` 3. Store which patterns matched Tuning from data beats tuning from memory. Files: `src/prefilter.ts`, `tests/prefilter.test.ts` ### Done when - [ ] Pass rate is reported per source - [ ] A hundred rejected items are reviewed and recorded - [ ] Prefiltering runs with no network or model calls - [ ] An excessive pass rate raises a warning ### Watch out - Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model. ## M3 · Classification with a bill you predicted Each surviving item is classified once, cached by content, and the run's cost is estimated before it starts. ### Steps 1. Key the cache on content hash The same complaint in three threads is one call. Files: `src/classify.ts`, `tests/classify-cache.test.ts` 2. Estimate before the run, refuse above the ceiling A ceiling checked after the fact is a receipt. Files: `src/cost.ts` 3. Hand-label fifty items and compare Model choice is a measurement you can afford to make. Files: `docs/classifier-agreement.md` ### Done when - [ ] A re-run with no new items makes zero calls - [ ] Duplicate text is classified once - [ ] An estimate prints and blocks above the ceiling - [ ] Agreement against fifty hand labels is recorded ### Watch out - Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing. ## M4 · Clusters you would defend Themes are grouped at a threshold you chose deliberately, named clearly, and evidenced verbatim. ### Steps 1. Sweep the threshold and look at it Five values and a distribution, before choosing. Files: `scripts/threshold-sweep.ts`, `docs/clustering.md` 2. Assert every quote is verbatim String comparison against the source text, on write. Files: `src/cluster/quotes.ts`, `tests/verbatim.test.ts` 3. Write the scoring formula in the README A score nobody can recompute is a number with a halo. Files: `README.md`, `src/cluster/score.ts` ### Done when - [ ] A threshold sweep is printed and the choice recorded - [ ] An altered quote fails the verbatim assertion - [ ] Scores follow the documented formula - [ ] Clustering is reproducible on an unchanged corpus ### Watch out - Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you. ## M5 · The tenth run, not the first (production only) A later scan shows what is new and what moved, rather than presenting last month's themes again. ### Steps 1. Match clusters across runs By shared items and embedding similarity, not by name. Files: `src/cluster/match.ts`, `tests/cross-run-match.test.ts` 2. Default the dashboard to what changed The reason to open it a second time. Files: `src/views/changes.tsx` 3. Say what the corpus is missing Manual community selection is the real limitation here. Files: `README.md` ### Done when - [ ] A second scan marks new, growing, shrinking and unchanged - [ ] Matching survives a model-renamed cluster - [ ] Exports carry scores and verbatim permalinks - [ ] The README names the community selection limitation ### Watch out - A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent.
OPERATIONS.md
# Operations · IdeaFast ## Backup The database, because the raw response cache is what makes every re-run free and refetching it would mean going back to the platform for data you already politely collected once. ## Restore Restore and run a scan: it should make almost no network calls and no model calls, which confirms both caches survived. If it starts refetching, the cache key changed and that is a bug. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Requests per minute against the configured limit, 429 counts, pass rate per source, model calls per run against the estimate, and the age of the recorded rate limit and prices. A rising 429 count means you are close to being unwelcome. ## Incident checklist Being blocked is the incident this design exists to avoid, and the response is to stop, read the current policy, and correct the configured limit before running again. A cost surprise is diagnosed from the classifications table, which records tokens per call. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The client is confirmed unable to exceed the configured rate - [ ] A re-run with no new items is confirmed to make zero model calls - [ ] An altered quote is confirmed to fail the verbatim assertion - [ ] The threshold sweep and its chosen value are confirmed recorded - [ ] A second scan is confirmed to mark what changed ## Launch constraint Do not market omitted IdeaFast capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Raw responses, items, classifications and clusters. DATABASE_PATH=./data/pain.db # Required. The communities to scan. Manual, and that is the stated limitation. SOURCES_FILE=./sources.txt # Required. From your registered application. Identifies the client the rate limit applies to. PLATFORM_CLIENT_ID=your-app-client-id # Required · secret. The other half of the application credentials. PLATFORM_CLIENT_SECRET=...redacted # Required. Descriptive and contactable, as the platform's rules ask. USER_AGENT=pain-finder/0.1 (contact: you@example.com) # Required. Kept under the platform's published limit, which is recorded with its check date. REQUESTS_PER_MINUTE=60 # Optional · secret. Classification and naming. Unset means the pipeline still ingests and prefilters. ANTHROPIC_API_KEY=sk-ant-...redacted # Required. A run estimated above this refuses to start. RUN_COST_CEILING_USD=5
$ choose a build depth, inspect the files, then open the complete pack in your agent
The clustering is not the hard part, the boring infrastructure around it is. Reddit throttles aggressive clients, so a real corpus takes patient background ingestion rather than a scan you kick off and watch. People pay to skip the warm-up and the API bill, not because the data is secret. It is all public.
xcommunity discovery, you can only scan subreddits you already thought of
xcross-scan dedupe, so repeat runs resurface the same pains as if they were new
xa warmed corpus, every fresh scan pays the full ingestion wait
xcost control, naive LLM classification of a busy subreddit gets expensive fast
xthe idea generation and validation layer on top of the raw clusters
share on X ↗"I just replaced IdeaFast ($19/mo) with one prompt"
IdeaFast pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| explorer | $9/workspace | — | 1 scan/day; 1 subreddit/scan; 3 evidence quotes/pain point; 2 ideas/pain point. |
| founder | $19/workspace | — | 5 scans/day; 2 subreddits/scan; 10 evidence quotes/pain point; 5 ideas/pain point. |
| builder | $49/workspace | — | 10 scans/day; 5 subreddits/scan; 30 evidence quotes/pain point; 10 ideas/pain point. |
free tierno free tier
billingMonthly only; no annual plan was publicly offered. Each paid plan has a 7-day trial.
hidden costsExtra scan packs cost $5 for 2 scans and do not expire.
verified 2026-08-12 · source ↗
Vibecode IdeaFast
Kinda. The core of IdeaFast is buildable in a weekend with the prompt on this page, but there are real gaps: community discovery, you can only scan subreddits you already thought of, cross-scan dedupe, so repeat runs resurface the same pains as if they were new. Read the honest list above before committing.
How much does IdeaFast cost?
IdeaFast costs about $19/month (Founder, checked 2026-08-01), which is $228 per year.
What do I lose by replacing IdeaFast?
Honestly: community discovery, you can only scan subreddits you already thought of; cross-scan dedupe, so repeat runs resurface the same pains as if they were new; a warmed corpus, every fresh scan pays the full ingestion wait; cost control, naive LLM classification of a busy subreddit gets expensive fast; the idea generation and validation layer on top of the raw clusters. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to IdeaFast?
Yes: PRAW (Python Reddit API wrapper, the usual starting point for the ingestion half), BERTopic (Topic clustering over embeddings, covers the grouping step without an LLM). Using prior art is also vibecoding; the prompt is for when you want it exactly your way.