Build 100 Questions

KINDAreplaces $9/mosaves $108/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A brand visibility benchmark whose hard problems are that every provider prices, caps and licences its web search differently, that answers are not reproducible so one run proves little, and that every published figure has to be traceable to the answer text it came from.

estimated effort multi-daythe files for this build are in the project pack

RuntimeA Node command line tool with no server and no accountsStorageSQLite, one file per run, written once and never editedProvidersOne adapter per provider behind one ask function, each with its own limitsCostA ceiling checked before the run and enforced during itReportingA self-contained static HTML file with the raw answers embedded

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the command line tool and the report renderer.
Get it nodejs.org, or your version manager. open ↗
Verify node --version prints v22 or higher
API keypay per use; a hundred questions across three providers with repeats is single-digit dollars in search fees plus tokens, checked against published prices on the day you build
Why Each provider bills its web search separately from its tokens, and the keys are the only thing in this project that costs money.
Get it Create a key in each provider's console and give it a spend limit where one is offered.
Verify A one-question run against each provider returns an answer and a search count
decidefree
Why Whether you may store, analyse and redisplay a grounded answer differs per provider, and it decides which providers this tool can include at all.
Get it Open each provider's API terms and its search tool documentation, note what is permitted for storage, analysis and display, and write down the date.
Verify A file lists each provider, the decision, the clause and the date read
have readyfree
Why Reducing a citation to a comparable source needs the registrable domain, and splitting on dots gets a long list of real sites wrong.
Get it Use a maintained library that bundles the list and updates it. open ↗
Verify A known multi-part suffix resolves correctly in a test
decidefree
Why Detection quality is mostly this file, and writing it after seeing results means tuning until the number looks good.
Get it List every spelling, abbreviation and product name that counts as your brand, and the competitors you expect.
Verify The list is in configuration and twenty highlighted answers have been read by hand

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- runs(id, started_at, finished_at, brand, domain, aliases_json, question_set_id, repeats, cost_ceiling, cost_actual, terms_note)
- providers_used(run_id, provider, model_id, search_tool, max_searches, price_note, terms_checked_on) · what was asked, and under which terms
- questions(id, question_set_id, ordinal, text, source) · generated or supplied, frozen with the run
- answers(id, run_id, question_id, provider, repeat_index, requested_at, latency_ms, raw_text, finish_reason, error, tokens_in, tokens_out, searches_used)
- citations(id, answer_id, ordinal, raw_url, registrable_domain, canonical_url, title) · normalised alongside the original, never instead of it
- mentions(id, answer_id, kind, matched_text, start_offset, end_offset, alias_used) · kind is brand or competitor, and the offsets are what the report highlights
- costs(run_id, provider, calls, searches, unit_note, amount_minor) · so the invoice can be checked against the run
- Nothing in this schema is updated after the run finishes. A rerun is a new run, and comparison is a join between two of them.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`OPENAI_API_KEY`secret	optional	`sk-...redacted`	Only if you included this provider. Give the key a spend limit.
`ANTHROPIC_API_KEY`secret	optional	`sk-ant-...redacted`	Only if you included this provider. Its search tool bills per search on top of tokens.
`XAI_API_KEY`secret	optional	`xai-...redacted`	Only if you included this provider. Check its current search pricing, which has been revised.
`RUN_DB_DIR`	required	`./runs`	One SQLite file per run, written once. Never edited afterwards.
`COST_CEILING_USD`	required	`15`	The run refuses to start above this and aborts if actual spend reaches it.
`REPEATS`	required	`3`	How many times each question is asked per provider. One is allowed and is labelled as one sample.
`MAX_SEARCHES_PER_QUESTION`	required	`4`	Passed to providers that support a cap, and used as the worst case in the estimate.

The build, in order

1
A run that knows what it will cost before it starts
The tool refuses to begin a run whose worst case exceeds the ceiling you set.
1. They change, and a number in code is a number nobody rechecks.
  Files config/pricing.jsonsrc/cost.ts
2. One comparative question can be ten searches.
  Files src/cost.tstests/cost.test.ts
3. A ceiling that is only checked at the start is a wish.
  Files src/runner.ts
done when · tick each as it passes
watch out
- Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort.
next phase
2
Asking everyone the same thing, and keeping the failures
Every question reaches every included provider, and a provider that fails leaves a visible hole rather than a gap.
1. The differences between providers belong in the adapters, not in the runner.
  Files src/providers/openai.tssrc/providers/anthropic.tssrc/providers/xai.ts
2. A hole you can see beats a number you cannot check.
  Files src/runner.tstests/failures.test.ts
3. Some providers reject a continued conversation whose result blocks were touched.
  Files src/providers/anthropic.ts
done when · tick each as it passes
watch out
- Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident.
next phase
3
Finding your name without fooling yourself
A mention is a match you can point at in the text, with its position.
1. Substring matching measures English, not your brand.
  Files src/mentions.tstests/mentions.test.ts
2. So the report can show the sentence rather than assert a count.
  Files src/mentions.ts
3. Detection is cheap to redo; answers are not.
  Files src/competitors.ts
done when · tick each as it passes
watch out
- Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate.
next phase
4
Citations reduced to something comparable
Two answers citing the same site count as the same source.
1. Splitting on dots is wrong for a long list of real suffixes.
  Files src/urls.tstests/urls.test.ts
2. A named list you can read beats a clever guess.
  Files config/tracking-params.jsonsrc/urls.ts
3. Normalisation is an opinion and you may want to change it.
  Files src/urls.ts
done when · tick each as it passes
watch out
- Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original.
5
A report that shows its workingproduct builder
Every figure in the report can be opened to the answers behind it, and repeats are visible as a range.
1. A report that needs the database is a report that dies with it.
  Files src/report.tssrc/report-template.html
2. One run is a sample, and the report should say so itself.
  Files src/report.ts
3. Different questions is the most common reason a number moved.
  Files src/compare.tstests/compare.test.ts
done when · tick each as it passes
watch out
- Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top.

what this build does not replace

Providers whose terms forbid storing or analysing grounded results.
Scheduled monitoring and alerting.
Generated recommendations.
Accounts, teams and billing.
reliable orchestration and retries across four providers
normalized citations and evidence-linked metrics
competitor and missed-question extraction
stored point-in-time reports and comparisons
polished exports and action recommendations

after v1, if you want it

A second question set written by hand, since generated questions measure the generator as much as the brand
A sentiment pass over stored answers, run separately and reported with its own uncertainty
A small local model for mention detection, which removes one provider from the loop entirely

Need the files? The project pack on the verdict page hands your agent the whole brief · more seo & marketing.