Build IdeaFast

KINDAreplaces $19/mosaves $228/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A forum mining pipeline whose hard problems are ingesting politely and resumably within a platform's published limits, keeping the classification bill under the price of the subscription it replaces, and clustering in a way you would defend to someone who disagrees with the result.

estimated effort weekendthe files for this build are in the project pack

RuntimeNode 22 with TypeScript, a CLI for scanning and a small local dashboardStorageSQLite via better-sqlite3, every raw response cached by requestIngestionAuthenticated, rate-limited, resumable, with the limit read from the platformClassificationA cheap model over prefiltered text, cached by content hashClusteringEmbeddings with a threshold you tuned and can explain

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the scanner and the dashboard.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
accountfree for non-commercial use on most platforms, but read the current terms
Why Authenticated access is what the published rate limit applies to, and what makes your client identifiable rather than anonymous traffic to be throttled.
Get it Register an application in the platform's developer settings and note the credentials and the terms you accepted.
Verify An authenticated request succeeds and the response carries rate-limit information
API keypay per use, and the prefilter is what keeps it small
Why Classification and embedding are the only paid parts, and the key is what the cost ceiling is protecting.
Get it Create an account with a provider and generate a key, then record the current prices with today's date.
Verify A ten item classification run completes and the ledger shows tokens
have readyfree
Why The cost estimate is only useful if the rates behind it are today's, and every provider in this space has changed them.
Get it Open the provider's pricing page, record input and output rates and any batch discount, with the date.
Verify The configuration holds the rates and the date, and the estimate uses them
decidefree, an hour of reading
Why The severity scale means whatever the model decides unless you have your own labels to compare against, and fifty is enough to catch a classifier that is systematically wrong.
Get it Read fifty prefiltered items and mark is_pain and severity yourself before looking at any model output.
Verify The labels exist in the repository and the agreement rate is computed

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- sources(id, platform, community, added_at, last_scanned_at, notes)
- raw_responses(id, source_id, request_url, fetched_at, status, body_sha256, body_path) · cached so a re-run is free
- items(id, source_id, kind, platform_id, permalink, author_hash, created_utc, text, parent_id) · kind is post or comment
- prefilter(item_id, matched_patterns_json, passed) · which cheap rule let this one through to a model
- classifications(item_id, content_sha256, is_pain, severity, one_line, model_id, tokens_in, tokens_out, classified_at) · keyed by content hash so identical text is never paid for twice
- clusters(run_id, id, name, size, mean_severity, recency_weight, score) · named by a model, scored by a formula you can read
- cluster_quotes(cluster_id, item_id, ordinal, verbatim, permalink) · verbatim is asserted to appear in the item's text
- runs(id, started_at, finished_at, sources_json, items_seen, items_classified, cost_estimate, cost_actual, threshold)
- The classifications table is keyed by content hash rather than by item, so the same complaint quoted in three threads costs one call.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_PATH`	required	`./data/pain.db`	Raw responses, items, classifications and clusters.
`SOURCES_FILE`	required	`./sources.txt`	The communities to scan. Manual, and that is the stated limitation.
`PLATFORM_CLIENT_ID`	required	`your-app-client-id`	From your registered application. Identifies the client the rate limit applies to.
`PLATFORM_CLIENT_SECRET`secret	required	`...redacted`	The other half of the application credentials.
`USER_AGENT`	required	`pain-finder/0.1 (contact: you@example.com)`	Descriptive and contactable, as the platform's rules ask.
`REQUESTS_PER_MINUTE`	required	`60`	Kept under the platform's published limit, which is recorded with its check date.
`ANTHROPIC_API_KEY`secret	optional	`sk-ant-...redacted`	Classification and naming. Unset means the pipeline still ingests and prefilters.
`RUN_COST_CEILING_USD`	required	`5`	A run estimated above this refuses to start.

The build, in order

1
An ingester that is welcome back tomorrow
Fetching is authenticated, under the published rate, resumable, and cached so a re-run costs nothing.
1. With a date, because these policies change.
  Files src/config/limits.tsdocs/platform-policy.md
2. A sleep in a loop is not a rate limiter when anything runs concurrently.
  Files src/fetch/limiter.tstests/rate-limit.test.ts
3. So the second scan is free and the parser can be fixed offline.
  Files src/fetch/cache.ts
done when · tick each as it passes
watch out
- Hammering a platform's endpoints from a bare script with a default user agent gets the address throttled and then blocked, and because the failure arrives as empty pages rather than an error, you will spend an evening debugging your parser instead of your manners. Authenticate, identify yourself, rate limit in the client, and treat a 429 as an instruction rather than a hiccup.
next phase
2
Spend nothing on text that is not a complaint
Cheap rules reduce the corpus to complaint-shaped text before a single token is spent.
1. The number that predicts your bill.
  Files src/prefilter.ts
2. The only way to know what the filter costs you in recall.
  Files docs/prefilter-review.md
3. Tuning from data beats tuning from memory.
  Files src/prefilter.tstests/prefilter.test.ts
done when · tick each as it passes
watch out
- Skipping the prefilter and classifying everything is the difference between a few pounds and a few hundred on a busy community, and because the bill arrives after the run you will find out once. Prefilter first, measure the pass rate, and only then let anything reach a model.
next phase
3
Classification with a bill you predicted
Each surviving item is classified once, cached by content, and the run's cost is estimated before it starts.
1. The same complaint in three threads is one call.
  Files src/classify.tstests/classify-cache.test.ts
2. A ceiling checked after the fact is a receipt.
  Files src/cost.ts
3. Model choice is a measurement you can afford to make.
  Files docs/classifier-agreement.md
done when · tick each as it passes
watch out
- Calling a large model per item without caching means re-running a scan after a parser fix pays the entire classification bill again, and the identical complaint cross-posted to three communities is paid for three times. Key the cache on the content hash, not the item id, and make a re-run with no new items cost nothing.
next phase
4
Clusters you would defend
Themes are grouped at a threshold you chose deliberately, named clearly, and evidenced verbatim.
1. Five values and a distribution, before choosing.
  Files scripts/threshold-sweep.tsdocs/clustering.md
2. String comparison against the source text, on write.
  Files src/cluster/quotes.tstests/verbatim.test.ts
3. A score nobody can recompute is a number with a halo.
  Files README.mdsrc/cluster/score.ts
done when · tick each as it passes
watch out
- Copying a similarity threshold from somewhere gives you either one enormous cluster called 'problems with software' or four hundred clusters of one item each, depending on which embedding model you happened to use, and both look like a working pipeline until you read the output. Sweep the threshold, print the distribution, and pick the value where the clusters mean something to you.
5
The tenth run, not the firstproduct builder
A later scan shows what is new and what moved, rather than presenting last month's themes again.
1. By shared items and embedding similarity, not by name.
  Files src/cluster/match.tstests/cross-run-match.test.ts
2. The reason to open it a second time.
  Files src/views/changes.tsx
3. Manual community selection is the real limitation here.
  Files README.md
done when · tick each as it passes
watch out
- A tool that produces the same top ten themes every week is one somebody opens twice, because the first run is interesting and the second is identical, which is exactly why the paid products in this category compete on the corpus rather than on the clustering. Match clusters across runs, default the view to what changed, and make an unchanged theme quiet rather than prominent.

what this build does not replace

Sources beyond the first platform.
Automatic community discovery.
Idea generation and validation.
Logged-in scraping or limit evasion.
community discovery, you can only scan subreddits you already thought of
cross-scan dedupe, so repeat runs resurface the same pains as if they were new
a warmed corpus, every fresh scan pays the full ingestion wait
cost control, naive LLM classification of a busy subreddit gets expensive fast
the idea generation and validation layer on top of the raw clusters

after v1, if you want it

A second platform added as an ingest module, once the first has run for a month without a 429
Per-cluster alerting when a theme grows past a threshold, using data the compare step already produces
Storing embeddings for the raw items rather than the summaries, which changes clustering quality and needs the sweep repeated

Need the files? The project pack on the verdict page hands your agent the whole brief · more user research.