Vibecode 100 Questions
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
A personal CLI that asks the same questions across four model APIs and compares the answers is weekend-buildable, but matching the product's web-grounded runs, source normalization, failure handling, durable evidence, scoring, and polished reports takes substantially more work.
the project pack5 files · written for this build, step by step
You are building a lean indie version of 100 Questions. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # 100 Questions · indie build A brand visibility benchmark whose hard problems are that every provider prices, caps and licences its web search differently, that answers are not reproducible so one run proves little, and that every published figure has to be traceable to the answer text it came from. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Node command line tool with no server and no accounts | this is a measurement instrument, and instruments should be runnable from a script | | Storage | SQLite, one file per run, written once and never edited | a benchmark you can rewrite after the fact is not a benchmark | | Providers | One adapter per provider behind one ask function, each with its own limits | search pricing, caps and terms differ per provider and the differences are load-bearing | | Cost | A ceiling checked before the run and enforced during it | search tools bill per call, and one question can fan out into many | | Reporting | A self-contained static HTML file with the raw answers embedded | a report that needs a server to read is a report nobody opens in six months | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the command line tool and the report renderer. - Get it: nodejs.org, or your version manager. - Verify: node --version prints v22 or higher - [ ] **API keys for the providers you decided to include** · pay per use; a hundred questions across three providers with repeats is single-digit dollars in search fees plus tokens, checked against published prices on the day you build - Why: Each provider bills its web search separately from its tokens, and the keys are the only thing in this project that costs money. - Get it: Create a key in each provider's console and give it a spend limit where one is offered. - Verify: A one-question run against each provider returns an answer and a search count - [ ] **A reading of each provider's current terms** · free - Why: Whether you may store, analyse and redisplay a grounded answer differs per provider, and it decides which providers this tool can include at all. - Get it: Open each provider's API terms and its search tool documentation, note what is permitted for storage, analysis and display, and write down the date. - Verify: A file lists each provider, the decision, the clause and the date read - [ ] **The public suffix list** · free - Why: Reducing a citation to a comparable source needs the registrable domain, and splitting on dots gets a long list of real sites wrong. - Get it: Use a maintained library that bundles the list and updates it. - Verify: A known multi-part suffix resolves correctly in a test - [ ] **Your brand aliases and competitor list** · free - Why: Detection quality is mostly this file, and writing it after seeing results means tuning until the number looks good. - Get it: List every spelling, abbreviation and product name that counts as your brand, and the competitors you expect. - Verify: The list is in configuration and twenty highlighted answers have been read by hand ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Providers whose terms forbid storing or analysing grounded results. - Scheduled monitoring and alerting. - Generated recommendations. - Accounts, teams and billing. - reliable orchestration and retries across four providers - normalized citations and evidence-linked metrics - competitor and missed-question extraction - stored point-in-time reports and comparisons - polished exports and action recommendations If one of those is essential to you, that is the reason to keep paying for 100 Questions, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · 100 Questions The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local benchmark that asks the same buyer questions of several model APIs and reports where your brand appears, with the evidence attached to replace 100 Questions. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and both are about not lying to yourself. One run is a sample, not a measurement. A web-grounded answer varies between runs because the model is non-deterministic by default (OpenAI says so in writing) and because the live index it searched changed overnight. So a single percentage is a snapshot with a timestamp, never a score. Either repeat each question a stated number of times and report the range and the number of runs beside every figure, or label the whole report as one sample. Never print a bare number that reads like a measurement of how visible you are. Second, the provider's terms decide what you may store and show, and they are not all the same. Anthropic prices its web search tool per thousand searches, requires citations when output is displayed, and requires search result blocks to be sent back byte-exact if a conversation continues. Google's Gemini API terms for Grounding with Google Search are stricter: they forbid caching, analysing or learning from Grounded Results, require them to be shown only to the end user who submitted the prompt, and forbid interspersing other content with them. A tool whose whole purpose is to store answers and compare them side by side does not fit inside that, so read each provider's current terms yourself, record the date, and write down which providers you included and which you left out and why. This is the part that decides the design, not the code. ### Stack (fixed, do not substitute) - Runtime: A Node command line tool with no server and no accounts · this is a measurement instrument, and instruments should be runnable from a script - Storage: SQLite, one file per run, written once and never edited · a benchmark you can rewrite after the fact is not a benchmark - Providers: One adapter per provider behind one ask function, each with its own limits · search pricing, caps and terms differ per provider and the differences are load-bearing - Cost: A ceiling checked before the run and enforced during it · search tools bill per call, and one question can fan out into many - Reporting: A self-contained static HTML file with the raw answers embedded · a report that needs a server to read is a report nobody opens in six months ### Data model (create this before Phase 1) - runs(id, started_at, finished_at, brand, domain, aliases_json, question_set_id, repeats, cost_ceiling, cost_actual, terms_note) - providers_used(run_id, provider, model_id, search_tool, max_searches, price_note, terms_checked_on) · what was asked, and under which terms - questions(id, question_set_id, ordinal, text, source) · generated or supplied, frozen with the run - answers(id, run_id, question_id, provider, repeat_index, requested_at, latency_ms, raw_text, finish_reason, error, tokens_in, tokens_out, searches_used) - citations(id, answer_id, ordinal, raw_url, registrable_domain, canonical_url, title) · normalised alongside the original, never instead of it - mentions(id, answer_id, kind, matched_text, start_offset, end_offset, alias_used) · kind is brand or competitor, and the offsets are what the report highlights - costs(run_id, provider, calls, searches, unit_note, amount_minor) · so the invoice can be checked against the run - Nothing in this schema is updated after the run finishes. A rerun is a new run, and comparison is a join between two of them. ### Phase 1 · A run that knows what it will cost before it starts Build: Freeze the questions, the aliases, the model identifiers and the repeat count into a run row. Before any call, compute the worst case from the current published prices per provider: at the time of writing OpenAI charges per thousand web search calls with search content tokens billed at model rates, Anthropic ten dollars per thousand searches plus tokens, xAI five dollars per thousand web search calls, and Google's grounding is billed per thousand requests above a free allowance that differs by model family. Read every one of these from the vendor's pricing page on the day you build, because they change. Then multiply by questions, repeats and the maximum searches per question, print the figure, and stop the run if it passes the ceiling. Trap: Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. Done when: A dry run prints a worst-case cost per provider and a total, computed from prices recorded in configuration with the date they were checked. A run whose worst case exceeds the ceiling refuses to start and says by how much. Actual calls and searches are counted per provider during the run and stored, so the figures can be compared against the invoice. The run row records model identifiers, search settings and the terms check date for every provider used. Do not build yet: No question generation, mention detection, normalisation or reporting yet. ### Phase 2 · Asking everyone the same thing, and keeping the failures Build: One adapter per provider behind a single ask function returning text, citations, search count and an error. Limit concurrency per provider separately. Retry only failures that could not have been billed, with backoff. Store the raw response text exactly as returned, including any opaque blocks a provider requires to be sent back unmodified if a conversation continues. Trap: Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. Done when: Every question and provider pair has a row, either an answer or an error, with no missing combinations, asserted in the suite. A provider returning errors for half the run produces a report showing those cells as failed and denominators that state the count they used. Raw answer text is stored unmodified, verified by comparing a stored row against a recorded response. Concurrency limits are per provider and a slow provider does not stall the others. Do not build yet: No mention detection, citation normalisation or report yet. ### Phase 3 · Finding your name without fooling yourself Build: Match brand and competitor names on word boundaries, case-insensitively, against an alias list in configuration, and store the offsets of every match. Extract competitor names in a separate structured pass over answers already stored, never during the run, so the extraction can be rerun without paying for the answers again. Trap: Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. Done when: A brand name appearing inside a longer word is not counted, tested with a crafted answer. Every mention row carries offsets that highlight correctly in the report, verified by opening one. Competitor extraction runs over stored answers and can be rerun without new API calls, tested by rerunning with the network disabled. Aliases are configuration, and adding one and rerunning detection changes counts without changing any stored answer. Do not build yet: No citation normalisation or report yet. ### Phase 4 · Citations reduced to something comparable Build: Normalise each citation by resolving the registrable domain with the public suffix list, stripping known tracking parameters, and keeping the canonical form beside the original URL. Compute how often your own domain is cited and which sources appear most, per provider. Trap: Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. Done when: Two forms of the same site, with and without a leading www and with tracking parameters, normalise to one source, tested. A hostname on a multi-part public suffix resolves to the correct registrable domain, tested with a known example. The original URL is retrievable for every citation. Owned-domain citation rate is computed per provider and links to the answers it came from. Do not build yet: No report, export or comparison yet. ### Phase 5 · A report that shows its working Build: Render one self-contained HTML file with the answers embedded, every aggregate expandable to the rows it used, failed cells shown as failures, and each figure carrying the number of runs it averages and the spread across them. Export questions, answers, mentions and citations as CSV. Compare two runs by identifier and show what moved and by how much, alongside the question text so a change in wording is visible as a reason. Trap: Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top. Done when: Every aggregate in the report expands to the answer rows it was computed from, checked for each figure type. Figures show the repeat count and the spread, and a single-repeat run is labelled as one sample throughout. Comparing two runs with different question sets or models shows a warning naming the difference before any number. The report opens from the file system with no server and no network, tested by disconnecting. Do not build yet: Nothing after this. ### Out of scope (and why) - Any provider whose terms forbid storing or analysing its grounded results, which on a reading of the current Gemini API terms includes Grounding with Google Search; check the terms yourself and record what you decided. - Scheduled monitoring and alerting, which turns a measurement instrument into a service with a bill that arrives whether you read the report or not. - Recommendations about what to change, since the tool has no way to know whether a change caused a movement. - Accounts, teams and billing, which is the product being replaced rather than the measurement. ### README must contain - That a single run is one sample of a non-deterministic system searching a changing index, and what that means for acting on a figure - Which providers were included, which were excluded on terms grounds, and the date those terms were read - The current price per search or per grounded request for each provider used, with the date checked and the page it came from - How the cost ceiling works, and that estimates assume the maximum searches per question rather than the average - That every figure in the report expands to the answers behind it, and that failed cells are shown rather than dropped ===== AGENTS.md ===== # Agent instructions · 100 Questions indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Node command line tool with no server and no accounts, SQLite, one file per run, written once and never edited, One adapter per provider behind one ask function, each with its own limits, A ceiling checked before the run and enforced during it, A self-contained static HTML file with the raw answers embedded. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. ===== BUILD_PLAN.md ===== # Build plan · 100 Questions A brand visibility benchmark whose hard problems are that every provider prices, caps and licences its web search differently, that answers are not reproducible so one run proves little, and that every published figure has to be traceable to the answer text it came from. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A run that knows what it will cost before it starts The tool refuses to begin a run whose worst case exceeds the ceiling you set. ### Steps 1. Record prices in configuration with a checked date They change, and a number in code is a number nobody rechecks. Files: `config/pricing.json`, `src/cost.ts` 2. Estimate on the cap, not the average One comparative question can be ten searches. Files: `src/cost.ts`, `tests/cost.test.ts` 3. Abort mid-run when spend passes the ceiling A ceiling that is only checked at the start is a wish. Files: `src/runner.ts` ### Done when - [ ] A dry run prints worst-case cost per provider - [ ] A run over the ceiling refuses to start with the figure - [ ] Calls and searches are counted and stored per provider - [ ] Model identifiers and the terms date are in the run row ### Watch out - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. ## Phase 2 · Asking everyone the same thing, and keeping the failures Every question reaches every included provider, and a provider that fails leaves a visible hole rather than a gap. ### Steps 1. One adapter per provider, one ask function The differences between providers belong in the adapters, not in the runner. Files: `src/providers/openai.ts`, `src/providers/anthropic.ts`, `src/providers/xai.ts` 2. Store failures as rows A hole you can see beats a number you cannot check. Files: `src/runner.ts`, `tests/failures.test.ts` 3. Keep provider blocks byte-exact Some providers reject a continued conversation whose result blocks were touched. Files: `src/providers/anthropic.ts` ### Done when - [ ] No question and provider combination is missing - [ ] Failed cells appear in the report with their error - [ ] Denominators state how many answers they used - [ ] Raw answer text is stored unmodified ### Watch out - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. ## Phase 3 · Finding your name without fooling yourself A mention is a match you can point at in the text, with its position. ### Steps 1. Match on word boundaries with an alias list Substring matching measures English, not your brand. Files: `src/mentions.ts`, `tests/mentions.test.ts` 2. Store the offsets of every match So the report can show the sentence rather than assert a count. Files: `src/mentions.ts` 3. Extract competitors from stored answers only Detection is cheap to redo; answers are not. Files: `src/competitors.ts` ### Done when - [ ] A name inside a longer word is not counted - [ ] Every mention highlights at its stored offset - [ ] Detection reruns with no network calls - [ ] Adding an alias changes counts without new answers ### Watch out - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. ## Phase 4 · Citations reduced to something comparable Two answers citing the same site count as the same source. ### Steps 1. Resolve domains with the public suffix list Splitting on dots is wrong for a long list of real suffixes. Files: `src/urls.ts`, `tests/urls.test.ts` 2. Strip tracking parameters from a named list A named list you can read beats a clever guess. Files: `config/tracking-params.json`, `src/urls.ts` 3. Keep the raw URL beside the normalised one Normalisation is an opinion and you may want to change it. Files: `src/urls.ts` ### Done when - [ ] www and tracking variants collapse to one source - [ ] A multi-part public suffix resolves correctly - [ ] The original URL is retrievable for every citation - [ ] Owned-domain rate links to its answers ### Watch out - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. ## Not in this build - Providers whose terms forbid storing or analysing grounded results. - Scheduled monitoring and alerting. - Generated recommendations. - Accounts, teams and billing. ## After v1, if you want it - A second question set written by hand, since generated questions measure the generator as much as the brand - A sentiment pass over stored answers, run separately and reported with its own uncertainty - A small local model for mention detection, which removes one provider from the loop entirely ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. Only if you included this provider. Give the key a spend limit. OPENAI_API_KEY=sk-...redacted # Optional · secret. Only if you included this provider. Its search tool bills per search on top of tokens. ANTHROPIC_API_KEY=sk-ant-...redacted # Optional · secret. Only if you included this provider. Check its current search pricing, which has been revised. XAI_API_KEY=xai-...redacted # Required. One SQLite file per run, written once. Never edited afterwards. RUN_DB_DIR=./runs # Required. The run refuses to start above this and aborts if actual spend reaches it. COST_CEILING_USD=15 # Required. How many times each question is asked per provider. One is allowed and is labelled as one sample. REPEATS=3 # Required. Passed to providers that support a cap, and used as the worst case in the estimate. MAX_SEARCHES_PER_QUESTION=4
You are building a lean indie version of 100 Questions. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # 100 Questions · indie build A brand visibility benchmark whose hard problems are that every provider prices, caps and licences its web search differently, that answers are not reproducible so one run proves little, and that every published figure has to be traceable to the answer text it came from. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Node command line tool with no server and no accounts | this is a measurement instrument, and instruments should be runnable from a script | | Storage | SQLite, one file per run, written once and never edited | a benchmark you can rewrite after the fact is not a benchmark | | Providers | One adapter per provider behind one ask function, each with its own limits | search pricing, caps and terms differ per provider and the differences are load-bearing | | Cost | A ceiling checked before the run and enforced during it | search tools bill per call, and one question can fan out into many | | Reporting | A self-contained static HTML file with the raw answers embedded | a report that needs a server to read is a report nobody opens in six months | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the command line tool and the report renderer. - Get it: nodejs.org, or your version manager. - Verify: node --version prints v22 or higher - [ ] **API keys for the providers you decided to include** · pay per use; a hundred questions across three providers with repeats is single-digit dollars in search fees plus tokens, checked against published prices on the day you build - Why: Each provider bills its web search separately from its tokens, and the keys are the only thing in this project that costs money. - Get it: Create a key in each provider's console and give it a spend limit where one is offered. - Verify: A one-question run against each provider returns an answer and a search count - [ ] **A reading of each provider's current terms** · free - Why: Whether you may store, analyse and redisplay a grounded answer differs per provider, and it decides which providers this tool can include at all. - Get it: Open each provider's API terms and its search tool documentation, note what is permitted for storage, analysis and display, and write down the date. - Verify: A file lists each provider, the decision, the clause and the date read - [ ] **The public suffix list** · free - Why: Reducing a citation to a comparable source needs the registrable domain, and splitting on dots gets a long list of real sites wrong. - Get it: Use a maintained library that bundles the list and updates it. - Verify: A known multi-part suffix resolves correctly in a test - [ ] **Your brand aliases and competitor list** · free - Why: Detection quality is mostly this file, and writing it after seeing results means tuning until the number looks good. - Get it: List every spelling, abbreviation and product name that counts as your brand, and the competitors you expect. - Verify: The list is in configuration and twenty highlighted answers have been read by hand ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Providers whose terms forbid storing or analysing grounded results. - Scheduled monitoring and alerting. - Generated recommendations. - Accounts, teams and billing. - reliable orchestration and retries across four providers - normalized citations and evidence-linked metrics - competitor and missed-question extraction - stored point-in-time reports and comparisons - polished exports and action recommendations If one of those is essential to you, that is the reason to keep paying for 100 Questions, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · 100 Questions The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local benchmark that asks the same buyer questions of several model APIs and reports where your brand appears, with the evidence attached to replace 100 Questions. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and both are about not lying to yourself. One run is a sample, not a measurement. A web-grounded answer varies between runs because the model is non-deterministic by default (OpenAI says so in writing) and because the live index it searched changed overnight. So a single percentage is a snapshot with a timestamp, never a score. Either repeat each question a stated number of times and report the range and the number of runs beside every figure, or label the whole report as one sample. Never print a bare number that reads like a measurement of how visible you are. Second, the provider's terms decide what you may store and show, and they are not all the same. Anthropic prices its web search tool per thousand searches, requires citations when output is displayed, and requires search result blocks to be sent back byte-exact if a conversation continues. Google's Gemini API terms for Grounding with Google Search are stricter: they forbid caching, analysing or learning from Grounded Results, require them to be shown only to the end user who submitted the prompt, and forbid interspersing other content with them. A tool whose whole purpose is to store answers and compare them side by side does not fit inside that, so read each provider's current terms yourself, record the date, and write down which providers you included and which you left out and why. This is the part that decides the design, not the code. ### Stack (fixed, do not substitute) - Runtime: A Node command line tool with no server and no accounts · this is a measurement instrument, and instruments should be runnable from a script - Storage: SQLite, one file per run, written once and never edited · a benchmark you can rewrite after the fact is not a benchmark - Providers: One adapter per provider behind one ask function, each with its own limits · search pricing, caps and terms differ per provider and the differences are load-bearing - Cost: A ceiling checked before the run and enforced during it · search tools bill per call, and one question can fan out into many - Reporting: A self-contained static HTML file with the raw answers embedded · a report that needs a server to read is a report nobody opens in six months ### Data model (create this before Phase 1) - runs(id, started_at, finished_at, brand, domain, aliases_json, question_set_id, repeats, cost_ceiling, cost_actual, terms_note) - providers_used(run_id, provider, model_id, search_tool, max_searches, price_note, terms_checked_on) · what was asked, and under which terms - questions(id, question_set_id, ordinal, text, source) · generated or supplied, frozen with the run - answers(id, run_id, question_id, provider, repeat_index, requested_at, latency_ms, raw_text, finish_reason, error, tokens_in, tokens_out, searches_used) - citations(id, answer_id, ordinal, raw_url, registrable_domain, canonical_url, title) · normalised alongside the original, never instead of it - mentions(id, answer_id, kind, matched_text, start_offset, end_offset, alias_used) · kind is brand or competitor, and the offsets are what the report highlights - costs(run_id, provider, calls, searches, unit_note, amount_minor) · so the invoice can be checked against the run - Nothing in this schema is updated after the run finishes. A rerun is a new run, and comparison is a join between two of them. ### Phase 1 · A run that knows what it will cost before it starts Build: Freeze the questions, the aliases, the model identifiers and the repeat count into a run row. Before any call, compute the worst case from the current published prices per provider: at the time of writing OpenAI charges per thousand web search calls with search content tokens billed at model rates, Anthropic ten dollars per thousand searches plus tokens, xAI five dollars per thousand web search calls, and Google's grounding is billed per thousand requests above a free allowance that differs by model family. Read every one of these from the vendor's pricing page on the day you build, because they change. Then multiply by questions, repeats and the maximum searches per question, print the figure, and stop the run if it passes the ceiling. Trap: Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. Done when: A dry run prints a worst-case cost per provider and a total, computed from prices recorded in configuration with the date they were checked. A run whose worst case exceeds the ceiling refuses to start and says by how much. Actual calls and searches are counted per provider during the run and stored, so the figures can be compared against the invoice. The run row records model identifiers, search settings and the terms check date for every provider used. Do not build yet: No question generation, mention detection, normalisation or reporting yet. ### Phase 2 · Asking everyone the same thing, and keeping the failures Build: One adapter per provider behind a single ask function returning text, citations, search count and an error. Limit concurrency per provider separately. Retry only failures that could not have been billed, with backoff. Store the raw response text exactly as returned, including any opaque blocks a provider requires to be sent back unmodified if a conversation continues. Trap: Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. Done when: Every question and provider pair has a row, either an answer or an error, with no missing combinations, asserted in the suite. A provider returning errors for half the run produces a report showing those cells as failed and denominators that state the count they used. Raw answer text is stored unmodified, verified by comparing a stored row against a recorded response. Concurrency limits are per provider and a slow provider does not stall the others. Do not build yet: No mention detection, citation normalisation or report yet. ### Phase 3 · Finding your name without fooling yourself Build: Match brand and competitor names on word boundaries, case-insensitively, against an alias list in configuration, and store the offsets of every match. Extract competitor names in a separate structured pass over answers already stored, never during the run, so the extraction can be rerun without paying for the answers again. Trap: Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. Done when: A brand name appearing inside a longer word is not counted, tested with a crafted answer. Every mention row carries offsets that highlight correctly in the report, verified by opening one. Competitor extraction runs over stored answers and can be rerun without new API calls, tested by rerunning with the network disabled. Aliases are configuration, and adding one and rerunning detection changes counts without changing any stored answer. Do not build yet: No citation normalisation or report yet. ### Phase 4 · Citations reduced to something comparable Build: Normalise each citation by resolving the registrable domain with the public suffix list, stripping known tracking parameters, and keeping the canonical form beside the original URL. Compute how often your own domain is cited and which sources appear most, per provider. Trap: Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. Done when: Two forms of the same site, with and without a leading www and with tracking parameters, normalise to one source, tested. A hostname on a multi-part public suffix resolves to the correct registrable domain, tested with a known example. The original URL is retrievable for every citation. Owned-domain citation rate is computed per provider and links to the answers it came from. Do not build yet: No report, export or comparison yet. ### Phase 5 · A report that shows its working Build: Render one self-contained HTML file with the answers embedded, every aggregate expandable to the rows it used, failed cells shown as failures, and each figure carrying the number of runs it averages and the spread across them. Export questions, answers, mentions and citations as CSV. Compare two runs by identifier and show what moved and by how much, alongside the question text so a change in wording is visible as a reason. Trap: Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top. Done when: Every aggregate in the report expands to the answer rows it was computed from, checked for each figure type. Figures show the repeat count and the spread, and a single-repeat run is labelled as one sample throughout. Comparing two runs with different question sets or models shows a warning naming the difference before any number. The report opens from the file system with no server and no network, tested by disconnecting. Do not build yet: Nothing after this. ### Out of scope (and why) - Any provider whose terms forbid storing or analysing its grounded results, which on a reading of the current Gemini API terms includes Grounding with Google Search; check the terms yourself and record what you decided. - Scheduled monitoring and alerting, which turns a measurement instrument into a service with a bill that arrives whether you read the report or not. - Recommendations about what to change, since the tool has no way to know whether a change caused a movement. - Accounts, teams and billing, which is the product being replaced rather than the measurement. ### README must contain - That a single run is one sample of a non-deterministic system searching a changing index, and what that means for acting on a figure - Which providers were included, which were excluded on terms grounds, and the date those terms were read - The current price per search or per grounded request for each provider used, with the date checked and the page it came from - How the cost ceiling works, and that estimates assume the maximum searches per question rather than the average - That every figure in the report expands to the answers behind it, and that failed cells are shown rather than dropped ===== AGENTS.md ===== # Agent instructions · 100 Questions indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Node command line tool with no server and no accounts, SQLite, one file per run, written once and never edited, One adapter per provider behind one ask function, each with its own limits, A ceiling checked before the run and enforced during it, A self-contained static HTML file with the raw answers embedded. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. ===== BUILD_PLAN.md ===== # Build plan · 100 Questions A brand visibility benchmark whose hard problems are that every provider prices, caps and licences its web search differently, that answers are not reproducible so one run proves little, and that every published figure has to be traceable to the answer text it came from. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A run that knows what it will cost before it starts The tool refuses to begin a run whose worst case exceeds the ceiling you set. ### Steps 1. Record prices in configuration with a checked date They change, and a number in code is a number nobody rechecks. Files: `config/pricing.json`, `src/cost.ts` 2. Estimate on the cap, not the average One comparative question can be ten searches. Files: `src/cost.ts`, `tests/cost.test.ts` 3. Abort mid-run when spend passes the ceiling A ceiling that is only checked at the start is a wish. Files: `src/runner.ts` ### Done when - [ ] A dry run prints worst-case cost per provider - [ ] A run over the ceiling refuses to start with the figure - [ ] Calls and searches are counted and stored per provider - [ ] Model identifiers and the terms date are in the run row ### Watch out - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. ## Phase 2 · Asking everyone the same thing, and keeping the failures Every question reaches every included provider, and a provider that fails leaves a visible hole rather than a gap. ### Steps 1. One adapter per provider, one ask function The differences between providers belong in the adapters, not in the runner. Files: `src/providers/openai.ts`, `src/providers/anthropic.ts`, `src/providers/xai.ts` 2. Store failures as rows A hole you can see beats a number you cannot check. Files: `src/runner.ts`, `tests/failures.test.ts` 3. Keep provider blocks byte-exact Some providers reject a continued conversation whose result blocks were touched. Files: `src/providers/anthropic.ts` ### Done when - [ ] No question and provider combination is missing - [ ] Failed cells appear in the report with their error - [ ] Denominators state how many answers they used - [ ] Raw answer text is stored unmodified ### Watch out - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. ## Phase 3 · Finding your name without fooling yourself A mention is a match you can point at in the text, with its position. ### Steps 1. Match on word boundaries with an alias list Substring matching measures English, not your brand. Files: `src/mentions.ts`, `tests/mentions.test.ts` 2. Store the offsets of every match So the report can show the sentence rather than assert a count. Files: `src/mentions.ts` 3. Extract competitors from stored answers only Detection is cheap to redo; answers are not. Files: `src/competitors.ts` ### Done when - [ ] A name inside a longer word is not counted - [ ] Every mention highlights at its stored offset - [ ] Detection reruns with no network calls - [ ] Adding an alias changes counts without new answers ### Watch out - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. ## Phase 4 · Citations reduced to something comparable Two answers citing the same site count as the same source. ### Steps 1. Resolve domains with the public suffix list Splitting on dots is wrong for a long list of real suffixes. Files: `src/urls.ts`, `tests/urls.test.ts` 2. Strip tracking parameters from a named list A named list you can read beats a clever guess. Files: `config/tracking-params.json`, `src/urls.ts` 3. Keep the raw URL beside the normalised one Normalisation is an opinion and you may want to change it. Files: `src/urls.ts` ### Done when - [ ] www and tracking variants collapse to one source - [ ] A multi-part public suffix resolves correctly - [ ] The original URL is retrievable for every citation - [ ] Owned-domain rate links to its answers ### Watch out - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. ## Not in this build - Providers whose terms forbid storing or analysing grounded results. - Scheduled monitoring and alerting. - Generated recommendations. - Accounts, teams and billing. ## After v1, if you want it - A second question set written by hand, since generated questions measure the generator as much as the brand - A sentiment pass over stored answers, run separately and reported with its own uncertainty - A small local model for mention detection, which removes one provider from the loop entirely ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. Only if you included this provider. Give the key a spend limit. OPENAI_API_KEY=sk-...redacted # Optional · secret. Only if you included this provider. Its search tool bills per search on top of tokens. ANTHROPIC_API_KEY=sk-ant-...redacted # Optional · secret. Only if you included this provider. Check its current search pricing, which has been revised. XAI_API_KEY=xai-...redacted # Required. One SQLite file per run, written once. Never edited afterwards. RUN_DB_DIR=./runs # Required. The run refuses to start above this and aborts if actual spend reaches it. COST_CEILING_USD=15 # Required. How many times each question is asked per provider. One is allowed and is labelled as one sample. REPEATS=3 # Required. Passed to providers that support a cap, and used as the worst case in the estimate. MAX_SEARCHES_PER_QUESTION=4
You are building a production product version of 100 Questions. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # 100 Questions · product brief ## Problem A personal CLI that asks the same questions across four model APIs and compares the answers is weekend-buildable, but matching the product's web-grounded runs, source normalization, failure handling, durable evidence, scoring, and polished reports takes substantially more work. ## Product outcome A benchmark that tells you what it will cost before it runs, asks only the providers whose terms allow what you are doing, keeps every failure visible, and reports nothing you cannot open and read the evidence for. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - OpenAI, Anthropic, Google Gemini, and xAI API keys - provider-specific web search or grounding tools - durable run storage - URL and citation normalization - report generation ## Explicit non-goals for v1 - Providers whose terms forbid storing or analysing grounded results. - Scheduled monitoring and alerting. - Generated recommendations. - Accounts, teams and billing. - reliable orchestration and retries across four providers - normalized citations and evidence-linked metrics - competitor and missed-question extraction - stored point-in-time reports and comparisons - polished exports and action recommendations ## Success criteria - A dry run is confirmed to print worst-case cost per provider - A run over the ceiling is confirmed to refuse to start - Failed cells are confirmed visible with stated denominators - A brand name inside a longer word is confirmed not counted - Every report figure is confirmed to expand to its answer rows ===== BRIEF.md ===== # Build brief · 100 Questions The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local benchmark that asks the same buyer questions of several model APIs and reports where your brand appears, with the evidence attached to replace 100 Questions. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and both are about not lying to yourself. One run is a sample, not a measurement. A web-grounded answer varies between runs because the model is non-deterministic by default (OpenAI says so in writing) and because the live index it searched changed overnight. So a single percentage is a snapshot with a timestamp, never a score. Either repeat each question a stated number of times and report the range and the number of runs beside every figure, or label the whole report as one sample. Never print a bare number that reads like a measurement of how visible you are. Second, the provider's terms decide what you may store and show, and they are not all the same. Anthropic prices its web search tool per thousand searches, requires citations when output is displayed, and requires search result blocks to be sent back byte-exact if a conversation continues. Google's Gemini API terms for Grounding with Google Search are stricter: they forbid caching, analysing or learning from Grounded Results, require them to be shown only to the end user who submitted the prompt, and forbid interspersing other content with them. A tool whose whole purpose is to store answers and compare them side by side does not fit inside that, so read each provider's current terms yourself, record the date, and write down which providers you included and which you left out and why. This is the part that decides the design, not the code. ### Stack (fixed, do not substitute) - Runtime: A Node command line tool with no server and no accounts · this is a measurement instrument, and instruments should be runnable from a script - Storage: SQLite, one file per run, written once and never edited · a benchmark you can rewrite after the fact is not a benchmark - Providers: One adapter per provider behind one ask function, each with its own limits · search pricing, caps and terms differ per provider and the differences are load-bearing - Cost: A ceiling checked before the run and enforced during it · search tools bill per call, and one question can fan out into many - Reporting: A self-contained static HTML file with the raw answers embedded · a report that needs a server to read is a report nobody opens in six months ### Data model (create this before Phase 1) - runs(id, started_at, finished_at, brand, domain, aliases_json, question_set_id, repeats, cost_ceiling, cost_actual, terms_note) - providers_used(run_id, provider, model_id, search_tool, max_searches, price_note, terms_checked_on) · what was asked, and under which terms - questions(id, question_set_id, ordinal, text, source) · generated or supplied, frozen with the run - answers(id, run_id, question_id, provider, repeat_index, requested_at, latency_ms, raw_text, finish_reason, error, tokens_in, tokens_out, searches_used) - citations(id, answer_id, ordinal, raw_url, registrable_domain, canonical_url, title) · normalised alongside the original, never instead of it - mentions(id, answer_id, kind, matched_text, start_offset, end_offset, alias_used) · kind is brand or competitor, and the offsets are what the report highlights - costs(run_id, provider, calls, searches, unit_note, amount_minor) · so the invoice can be checked against the run - Nothing in this schema is updated after the run finishes. A rerun is a new run, and comparison is a join between two of them. ### Phase 1 · A run that knows what it will cost before it starts Build: Freeze the questions, the aliases, the model identifiers and the repeat count into a run row. Before any call, compute the worst case from the current published prices per provider: at the time of writing OpenAI charges per thousand web search calls with search content tokens billed at model rates, Anthropic ten dollars per thousand searches plus tokens, xAI five dollars per thousand web search calls, and Google's grounding is billed per thousand requests above a free allowance that differs by model family. Read every one of these from the vendor's pricing page on the day you build, because they change. Then multiply by questions, repeats and the maximum searches per question, print the figure, and stop the run if it passes the ceiling. Trap: Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. Done when: A dry run prints a worst-case cost per provider and a total, computed from prices recorded in configuration with the date they were checked. A run whose worst case exceeds the ceiling refuses to start and says by how much. Actual calls and searches are counted per provider during the run and stored, so the figures can be compared against the invoice. The run row records model identifiers, search settings and the terms check date for every provider used. Do not build yet: No question generation, mention detection, normalisation or reporting yet. ### Phase 2 · Asking everyone the same thing, and keeping the failures Build: One adapter per provider behind a single ask function returning text, citations, search count and an error. Limit concurrency per provider separately. Retry only failures that could not have been billed, with backoff. Store the raw response text exactly as returned, including any opaque blocks a provider requires to be sent back unmodified if a conversation continues. Trap: Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. Done when: Every question and provider pair has a row, either an answer or an error, with no missing combinations, asserted in the suite. A provider returning errors for half the run produces a report showing those cells as failed and denominators that state the count they used. Raw answer text is stored unmodified, verified by comparing a stored row against a recorded response. Concurrency limits are per provider and a slow provider does not stall the others. Do not build yet: No mention detection, citation normalisation or report yet. ### Phase 3 · Finding your name without fooling yourself Build: Match brand and competitor names on word boundaries, case-insensitively, against an alias list in configuration, and store the offsets of every match. Extract competitor names in a separate structured pass over answers already stored, never during the run, so the extraction can be rerun without paying for the answers again. Trap: Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. Done when: A brand name appearing inside a longer word is not counted, tested with a crafted answer. Every mention row carries offsets that highlight correctly in the report, verified by opening one. Competitor extraction runs over stored answers and can be rerun without new API calls, tested by rerunning with the network disabled. Aliases are configuration, and adding one and rerunning detection changes counts without changing any stored answer. Do not build yet: No citation normalisation or report yet. ### Phase 4 · Citations reduced to something comparable Build: Normalise each citation by resolving the registrable domain with the public suffix list, stripping known tracking parameters, and keeping the canonical form beside the original URL. Compute how often your own domain is cited and which sources appear most, per provider. Trap: Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. Done when: Two forms of the same site, with and without a leading www and with tracking parameters, normalise to one source, tested. A hostname on a multi-part public suffix resolves to the correct registrable domain, tested with a known example. The original URL is retrievable for every citation. Owned-domain citation rate is computed per provider and links to the answers it came from. Do not build yet: No report, export or comparison yet. ### Phase 5 · A report that shows its working Build: Render one self-contained HTML file with the answers embedded, every aggregate expandable to the rows it used, failed cells shown as failures, and each figure carrying the number of runs it averages and the spread across them. Export questions, answers, mentions and citations as CSV. Compare two runs by identifier and show what moved and by how much, alongside the question text so a change in wording is visible as a reason. Trap: Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top. Done when: Every aggregate in the report expands to the answer rows it was computed from, checked for each figure type. Figures show the repeat count and the spread, and a single-repeat run is labelled as one sample throughout. Comparing two runs with different question sets or models shows a warning naming the difference before any number. The report opens from the file system with no server and no network, tested by disconnecting. Do not build yet: Nothing after this. ### Out of scope (and why) - Any provider whose terms forbid storing or analysing its grounded results, which on a reading of the current Gemini API terms includes Grounding with Google Search; check the terms yourself and record what you decided. - Scheduled monitoring and alerting, which turns a measurement instrument into a service with a bill that arrives whether you read the report or not. - Recommendations about what to change, since the tool has no way to know whether a change caused a movement. - Accounts, teams and billing, which is the product being replaced rather than the measurement. ### README must contain - That a single run is one sample of a non-deterministic system searching a changing index, and what that means for acting on a figure - Which providers were included, which were excluded on terms grounds, and the date those terms were read - The current price per search or per grounded request for each provider used, with the date checked and the page it came from - How the cost ceiling works, and that estimates assume the maximum searches per question rather than the average - That every figure in the report expands to the answers behind it, and that failed cells are shown rather than dropped ===== ARCHITECTURE.md ===== # Architecture · 100 Questions ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Node command line tool with no server and no accounts | this is a measurement instrument, and instruments should be runnable from a script | | Storage | SQLite, one file per run, written once and never edited | a benchmark you can rewrite after the fact is not a benchmark | | Providers | One adapter per provider behind one ask function, each with its own limits | search pricing, caps and terms differ per provider and the differences are load-bearing | | Cost | A ceiling checked before the run and enforced during it | search tools bill per call, and one question can fan out into many | | Reporting | A self-contained static HTML file with the raw answers embedded | a report that needs a server to read is a report nobody opens in six months | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Cost | Estimation, ceiling and spend tracking. | Any pricing source, provided it carries the date it was checked. | | Providers | One adapter each, with its own limits and terms note. | Any provider whose terms permit storage and analysis, added as an adapter. | | Detection | Mentions, offsets and competitor extraction. | Any matcher, provided it stores offsets and never matches inside words. | | Citations | Registrable domains and canonical forms. | Any normaliser, provided the raw URL survives. | | Report | Self-contained HTML, spread and comparison. | Any renderer, provided every figure expands to its rows. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `OPENAI_API_KEY` · optional, secret · Only if you included this provider. Give the key a spend limit. - `ANTHROPIC_API_KEY` · optional, secret · Only if you included this provider. Its search tool bills per search on top of tokens. - `XAI_API_KEY` · optional, secret · Only if you included this provider. Check its current search pricing, which has been revised. - `RUN_DB_DIR` · required · One SQLite file per run, written once. Never edited afterwards. - `COST_CEILING_USD` · required · The run refuses to start above this and aborts if actual spend reaches it. - `REPEATS` · required · How many times each question is asked per provider. One is allowed and is labelled as one sample. - `MAX_SEARCHES_PER_QUESTION` · required · Passed to providers that support a cap, and used as the worst case in the estimate. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · 100 Questions product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A Node command line tool with no server and no accounts, SQLite, one file per run, written once and never edited, One adapter per provider behind one ask function, each with its own limits, A ceiling checked before the run and enforced during it, A self-contained static HTML file with the raw answers embedded. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. - Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top. ===== MILESTONES.md ===== # Delivery milestones · 100 Questions Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A run that knows what it will cost before it starts The tool refuses to begin a run whose worst case exceeds the ceiling you set. ### Steps 1. Record prices in configuration with a checked date They change, and a number in code is a number nobody rechecks. Files: `config/pricing.json`, `src/cost.ts` 2. Estimate on the cap, not the average One comparative question can be ten searches. Files: `src/cost.ts`, `tests/cost.test.ts` 3. Abort mid-run when spend passes the ceiling A ceiling that is only checked at the start is a wish. Files: `src/runner.ts` ### Done when - [ ] A dry run prints worst-case cost per provider - [ ] A run over the ceiling refuses to start with the figure - [ ] Calls and searches are counted and stored per provider - [ ] Model identifiers and the terms date are in the run row ### Watch out - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. ## M2 · Asking everyone the same thing, and keeping the failures Every question reaches every included provider, and a provider that fails leaves a visible hole rather than a gap. ### Steps 1. One adapter per provider, one ask function The differences between providers belong in the adapters, not in the runner. Files: `src/providers/openai.ts`, `src/providers/anthropic.ts`, `src/providers/xai.ts` 2. Store failures as rows A hole you can see beats a number you cannot check. Files: `src/runner.ts`, `tests/failures.test.ts` 3. Keep provider blocks byte-exact Some providers reject a continued conversation whose result blocks were touched. Files: `src/providers/anthropic.ts` ### Done when - [ ] No question and provider combination is missing - [ ] Failed cells appear in the report with their error - [ ] Denominators state how many answers they used - [ ] Raw answer text is stored unmodified ### Watch out - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. ## M3 · Finding your name without fooling yourself A mention is a match you can point at in the text, with its position. ### Steps 1. Match on word boundaries with an alias list Substring matching measures English, not your brand. Files: `src/mentions.ts`, `tests/mentions.test.ts` 2. Store the offsets of every match So the report can show the sentence rather than assert a count. Files: `src/mentions.ts` 3. Extract competitors from stored answers only Detection is cheap to redo; answers are not. Files: `src/competitors.ts` ### Done when - [ ] A name inside a longer word is not counted - [ ] Every mention highlights at its stored offset - [ ] Detection reruns with no network calls - [ ] Adding an alias changes counts without new answers ### Watch out - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. ## M4 · Citations reduced to something comparable Two answers citing the same site count as the same source. ### Steps 1. Resolve domains with the public suffix list Splitting on dots is wrong for a long list of real suffixes. Files: `src/urls.ts`, `tests/urls.test.ts` 2. Strip tracking parameters from a named list A named list you can read beats a clever guess. Files: `config/tracking-params.json`, `src/urls.ts` 3. Keep the raw URL beside the normalised one Normalisation is an opinion and you may want to change it. Files: `src/urls.ts` ### Done when - [ ] www and tracking variants collapse to one source - [ ] A multi-part public suffix resolves correctly - [ ] The original URL is retrievable for every citation - [ ] Owned-domain rate links to its answers ### Watch out - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. ## M5 · A report that shows its working (production only) Every figure in the report can be opened to the answers behind it, and repeats are visible as a range. ### Steps 1. Embed the evidence in the report file A report that needs the database is a report that dies with it. Files: `src/report.ts`, `src/report-template.html` 2. Show repeats and spread beside every figure One run is a sample, and the report should say so itself. Files: `src/report.ts` 3. Warn before comparing runs that differ Different questions is the most common reason a number moved. Files: `src/compare.ts`, `tests/compare.test.ts` ### Done when - [ ] Every figure expands to its answer rows - [ ] Repeat count and spread appear beside each figure - [ ] Comparing mismatched runs warns first - [ ] The report opens offline from the file system ### Watch out - Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top. ===== OPERATIONS.md ===== # Operations · 100 Questions ## Backup Run databases and rendered reports, copied off the machine. They are the only record of what a provider said on a date, and no provider will tell you again. ## Restore Restore a run file and open its report. If the report needs anything except the file itself, the report is wrong and the phase five test should have caught it. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Spend against ceiling per run, failed cells per provider, and the age of the recorded prices and terms dates. All three go stale quietly. ## Incident checklist The incidents are a surprise invoice and a figure somebody acted on. The costs table answers the first against the provider's own bill; the evidence links answer the second by showing the answers the figure came from. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A dry run is confirmed to print worst-case cost per provider - [ ] A run over the ceiling is confirmed to refuse to start - [ ] Failed cells are confirmed visible with stated denominators - [ ] A brand name inside a longer word is confirmed not counted - [ ] Every report figure is confirmed to expand to its answer rows ## Launch constraint Do not market omitted 100 Questions capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. Only if you included this provider. Give the key a spend limit. OPENAI_API_KEY=sk-...redacted # Optional · secret. Only if you included this provider. Its search tool bills per search on top of tokens. ANTHROPIC_API_KEY=sk-ant-...redacted # Optional · secret. Only if you included this provider. Check its current search pricing, which has been revised. XAI_API_KEY=xai-...redacted # Required. One SQLite file per run, written once. Never edited afterwards. RUN_DB_DIR=./runs # Required. The run refuses to start above this and aborts if actual spend reaches it. COST_CEILING_USD=15 # Required. How many times each question is asked per provider. One is allowed and is labelled as one sample. REPEATS=3 # Required. Passed to providers that support a cap, and used as the worst case in the estimate. MAX_SEARCHES_PER_QUESTION=4
README.md
# 100 Questions · indie build A brand visibility benchmark whose hard problems are that every provider prices, caps and licences its web search differently, that answers are not reproducible so one run proves little, and that every published figure has to be traceable to the answer text it came from. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Node command line tool with no server and no accounts | this is a measurement instrument, and instruments should be runnable from a script | | Storage | SQLite, one file per run, written once and never edited | a benchmark you can rewrite after the fact is not a benchmark | | Providers | One adapter per provider behind one ask function, each with its own limits | search pricing, caps and terms differ per provider and the differences are load-bearing | | Cost | A ceiling checked before the run and enforced during it | search tools bill per call, and one question can fan out into many | | Reporting | A self-contained static HTML file with the raw answers embedded | a report that needs a server to read is a report nobody opens in six months | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the command line tool and the report renderer. - Get it: nodejs.org, or your version manager. - Verify: node --version prints v22 or higher - [ ] **API keys for the providers you decided to include** · pay per use; a hundred questions across three providers with repeats is single-digit dollars in search fees plus tokens, checked against published prices on the day you build - Why: Each provider bills its web search separately from its tokens, and the keys are the only thing in this project that costs money. - Get it: Create a key in each provider's console and give it a spend limit where one is offered. - Verify: A one-question run against each provider returns an answer and a search count - [ ] **A reading of each provider's current terms** · free - Why: Whether you may store, analyse and redisplay a grounded answer differs per provider, and it decides which providers this tool can include at all. - Get it: Open each provider's API terms and its search tool documentation, note what is permitted for storage, analysis and display, and write down the date. - Verify: A file lists each provider, the decision, the clause and the date read - [ ] **The public suffix list** · free - Why: Reducing a citation to a comparable source needs the registrable domain, and splitting on dots gets a long list of real sites wrong. - Get it: Use a maintained library that bundles the list and updates it. - Verify: A known multi-part suffix resolves correctly in a test - [ ] **Your brand aliases and competitor list** · free - Why: Detection quality is mostly this file, and writing it after seeing results means tuning until the number looks good. - Get it: List every spelling, abbreviation and product name that counts as your brand, and the competitors you expect. - Verify: The list is in configuration and twenty highlighted answers have been read by hand ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Providers whose terms forbid storing or analysing grounded results. - Scheduled monitoring and alerting. - Generated recommendations. - Accounts, teams and billing. - reliable orchestration and retries across four providers - normalized citations and evidence-linked metrics - competitor and missed-question extraction - stored point-in-time reports and comparisons - polished exports and action recommendations If one of those is essential to you, that is the reason to keep paying for 100 Questions, and the README should say so rather than pretend.
BRIEF.md
# Build brief · 100 Questions The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local benchmark that asks the same buyer questions of several model APIs and reports where your brand appears, with the evidence attached to replace 100 Questions. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and both are about not lying to yourself. One run is a sample, not a measurement. A web-grounded answer varies between runs because the model is non-deterministic by default (OpenAI says so in writing) and because the live index it searched changed overnight. So a single percentage is a snapshot with a timestamp, never a score. Either repeat each question a stated number of times and report the range and the number of runs beside every figure, or label the whole report as one sample. Never print a bare number that reads like a measurement of how visible you are. Second, the provider's terms decide what you may store and show, and they are not all the same. Anthropic prices its web search tool per thousand searches, requires citations when output is displayed, and requires search result blocks to be sent back byte-exact if a conversation continues. Google's Gemini API terms for Grounding with Google Search are stricter: they forbid caching, analysing or learning from Grounded Results, require them to be shown only to the end user who submitted the prompt, and forbid interspersing other content with them. A tool whose whole purpose is to store answers and compare them side by side does not fit inside that, so read each provider's current terms yourself, record the date, and write down which providers you included and which you left out and why. This is the part that decides the design, not the code. ### Stack (fixed, do not substitute) - Runtime: A Node command line tool with no server and no accounts · this is a measurement instrument, and instruments should be runnable from a script - Storage: SQLite, one file per run, written once and never edited · a benchmark you can rewrite after the fact is not a benchmark - Providers: One adapter per provider behind one ask function, each with its own limits · search pricing, caps and terms differ per provider and the differences are load-bearing - Cost: A ceiling checked before the run and enforced during it · search tools bill per call, and one question can fan out into many - Reporting: A self-contained static HTML file with the raw answers embedded · a report that needs a server to read is a report nobody opens in six months ### Data model (create this before Phase 1) - runs(id, started_at, finished_at, brand, domain, aliases_json, question_set_id, repeats, cost_ceiling, cost_actual, terms_note) - providers_used(run_id, provider, model_id, search_tool, max_searches, price_note, terms_checked_on) · what was asked, and under which terms - questions(id, question_set_id, ordinal, text, source) · generated or supplied, frozen with the run - answers(id, run_id, question_id, provider, repeat_index, requested_at, latency_ms, raw_text, finish_reason, error, tokens_in, tokens_out, searches_used) - citations(id, answer_id, ordinal, raw_url, registrable_domain, canonical_url, title) · normalised alongside the original, never instead of it - mentions(id, answer_id, kind, matched_text, start_offset, end_offset, alias_used) · kind is brand or competitor, and the offsets are what the report highlights - costs(run_id, provider, calls, searches, unit_note, amount_minor) · so the invoice can be checked against the run - Nothing in this schema is updated after the run finishes. A rerun is a new run, and comparison is a join between two of them. ### Phase 1 · A run that knows what it will cost before it starts Build: Freeze the questions, the aliases, the model identifiers and the repeat count into a run row. Before any call, compute the worst case from the current published prices per provider: at the time of writing OpenAI charges per thousand web search calls with search content tokens billed at model rates, Anthropic ten dollars per thousand searches plus tokens, xAI five dollars per thousand web search calls, and Google's grounding is billed per thousand requests above a free allowance that differs by model family. Read every one of these from the vendor's pricing page on the day you build, because they change. Then multiply by questions, repeats and the maximum searches per question, print the figure, and stop the run if it passes the ceiling. Trap: Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. Done when: A dry run prints a worst-case cost per provider and a total, computed from prices recorded in configuration with the date they were checked. A run whose worst case exceeds the ceiling refuses to start and says by how much. Actual calls and searches are counted per provider during the run and stored, so the figures can be compared against the invoice. The run row records model identifiers, search settings and the terms check date for every provider used. Do not build yet: No question generation, mention detection, normalisation or reporting yet. ### Phase 2 · Asking everyone the same thing, and keeping the failures Build: One adapter per provider behind a single ask function returning text, citations, search count and an error. Limit concurrency per provider separately. Retry only failures that could not have been billed, with backoff. Store the raw response text exactly as returned, including any opaque blocks a provider requires to be sent back unmodified if a conversation continues. Trap: Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. Done when: Every question and provider pair has a row, either an answer or an error, with no missing combinations, asserted in the suite. A provider returning errors for half the run produces a report showing those cells as failed and denominators that state the count they used. Raw answer text is stored unmodified, verified by comparing a stored row against a recorded response. Concurrency limits are per provider and a slow provider does not stall the others. Do not build yet: No mention detection, citation normalisation or report yet. ### Phase 3 · Finding your name without fooling yourself Build: Match brand and competitor names on word boundaries, case-insensitively, against an alias list in configuration, and store the offsets of every match. Extract competitor names in a separate structured pass over answers already stored, never during the run, so the extraction can be rerun without paying for the answers again. Trap: Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. Done when: A brand name appearing inside a longer word is not counted, tested with a crafted answer. Every mention row carries offsets that highlight correctly in the report, verified by opening one. Competitor extraction runs over stored answers and can be rerun without new API calls, tested by rerunning with the network disabled. Aliases are configuration, and adding one and rerunning detection changes counts without changing any stored answer. Do not build yet: No citation normalisation or report yet. ### Phase 4 · Citations reduced to something comparable Build: Normalise each citation by resolving the registrable domain with the public suffix list, stripping known tracking parameters, and keeping the canonical form beside the original URL. Compute how often your own domain is cited and which sources appear most, per provider. Trap: Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. Done when: Two forms of the same site, with and without a leading www and with tracking parameters, normalise to one source, tested. A hostname on a multi-part public suffix resolves to the correct registrable domain, tested with a known example. The original URL is retrievable for every citation. Owned-domain citation rate is computed per provider and links to the answers it came from. Do not build yet: No report, export or comparison yet. ### Phase 5 · A report that shows its working Build: Render one self-contained HTML file with the answers embedded, every aggregate expandable to the rows it used, failed cells shown as failures, and each figure carrying the number of runs it averages and the spread across them. Export questions, answers, mentions and citations as CSV. Compare two runs by identifier and show what moved and by how much, alongside the question text so a change in wording is visible as a reason. Trap: Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top. Done when: Every aggregate in the report expands to the answer rows it was computed from, checked for each figure type. Figures show the repeat count and the spread, and a single-repeat run is labelled as one sample throughout. Comparing two runs with different question sets or models shows a warning naming the difference before any number. The report opens from the file system with no server and no network, tested by disconnecting. Do not build yet: Nothing after this. ### Out of scope (and why) - Any provider whose terms forbid storing or analysing its grounded results, which on a reading of the current Gemini API terms includes Grounding with Google Search; check the terms yourself and record what you decided. - Scheduled monitoring and alerting, which turns a measurement instrument into a service with a bill that arrives whether you read the report or not. - Recommendations about what to change, since the tool has no way to know whether a change caused a movement. - Accounts, teams and billing, which is the product being replaced rather than the measurement. ### README must contain - That a single run is one sample of a non-deterministic system searching a changing index, and what that means for acting on a figure - Which providers were included, which were excluded on terms grounds, and the date those terms were read - The current price per search or per grounded request for each provider used, with the date checked and the page it came from - How the cost ceiling works, and that estimates assume the maximum searches per question rather than the average - That every figure in the report expands to the answers behind it, and that failed cells are shown rather than dropped
AGENTS.md
# Agent instructions · 100 Questions indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Node command line tool with no server and no accounts, SQLite, one file per run, written once and never edited, One adapter per provider behind one ask function, each with its own limits, A ceiling checked before the run and enforced during it, A self-contained static HTML file with the raw answers embedded. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original.
BUILD_PLAN.md
# Build plan · 100 Questions A brand visibility benchmark whose hard problems are that every provider prices, caps and licences its web search differently, that answers are not reproducible so one run proves little, and that every published figure has to be traceable to the answer text it came from. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A run that knows what it will cost before it starts The tool refuses to begin a run whose worst case exceeds the ceiling you set. ### Steps 1. Record prices in configuration with a checked date They change, and a number in code is a number nobody rechecks. Files: `config/pricing.json`, `src/cost.ts` 2. Estimate on the cap, not the average One comparative question can be ten searches. Files: `src/cost.ts`, `tests/cost.test.ts` 3. Abort mid-run when spend passes the ceiling A ceiling that is only checked at the start is a wish. Files: `src/runner.ts` ### Done when - [ ] A dry run prints worst-case cost per provider - [ ] A run over the ceiling refuses to start with the figure - [ ] Calls and searches are counted and stored per provider - [ ] Model identifiers and the terms date are in the run row ### Watch out - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. ## Phase 2 · Asking everyone the same thing, and keeping the failures Every question reaches every included provider, and a provider that fails leaves a visible hole rather than a gap. ### Steps 1. One adapter per provider, one ask function The differences between providers belong in the adapters, not in the runner. Files: `src/providers/openai.ts`, `src/providers/anthropic.ts`, `src/providers/xai.ts` 2. Store failures as rows A hole you can see beats a number you cannot check. Files: `src/runner.ts`, `tests/failures.test.ts` 3. Keep provider blocks byte-exact Some providers reject a continued conversation whose result blocks were touched. Files: `src/providers/anthropic.ts` ### Done when - [ ] No question and provider combination is missing - [ ] Failed cells appear in the report with their error - [ ] Denominators state how many answers they used - [ ] Raw answer text is stored unmodified ### Watch out - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. ## Phase 3 · Finding your name without fooling yourself A mention is a match you can point at in the text, with its position. ### Steps 1. Match on word boundaries with an alias list Substring matching measures English, not your brand. Files: `src/mentions.ts`, `tests/mentions.test.ts` 2. Store the offsets of every match So the report can show the sentence rather than assert a count. Files: `src/mentions.ts` 3. Extract competitors from stored answers only Detection is cheap to redo; answers are not. Files: `src/competitors.ts` ### Done when - [ ] A name inside a longer word is not counted - [ ] Every mention highlights at its stored offset - [ ] Detection reruns with no network calls - [ ] Adding an alias changes counts without new answers ### Watch out - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. ## Phase 4 · Citations reduced to something comparable Two answers citing the same site count as the same source. ### Steps 1. Resolve domains with the public suffix list Splitting on dots is wrong for a long list of real suffixes. Files: `src/urls.ts`, `tests/urls.test.ts` 2. Strip tracking parameters from a named list A named list you can read beats a clever guess. Files: `config/tracking-params.json`, `src/urls.ts` 3. Keep the raw URL beside the normalised one Normalisation is an opinion and you may want to change it. Files: `src/urls.ts` ### Done when - [ ] www and tracking variants collapse to one source - [ ] A multi-part public suffix resolves correctly - [ ] The original URL is retrievable for every citation - [ ] Owned-domain rate links to its answers ### Watch out - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. ## Not in this build - Providers whose terms forbid storing or analysing grounded results. - Scheduled monitoring and alerting. - Generated recommendations. - Accounts, teams and billing. ## After v1, if you want it - A second question set written by hand, since generated questions measure the generator as much as the brand - A sentiment pass over stored answers, run separately and reported with its own uncertainty - A small local model for mention detection, which removes one provider from the loop entirely
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. Only if you included this provider. Give the key a spend limit. OPENAI_API_KEY=sk-...redacted # Optional · secret. Only if you included this provider. Its search tool bills per search on top of tokens. ANTHROPIC_API_KEY=sk-ant-...redacted # Optional · secret. Only if you included this provider. Check its current search pricing, which has been revised. XAI_API_KEY=xai-...redacted # Required. One SQLite file per run, written once. Never edited afterwards. RUN_DB_DIR=./runs # Required. The run refuses to start above this and aborts if actual spend reaches it. COST_CEILING_USD=15 # Required. How many times each question is asked per provider. One is allowed and is labelled as one sample. REPEATS=3 # Required. Passed to providers that support a cap, and used as the worst case in the estimate. MAX_SEARCHES_PER_QUESTION=4
PRODUCT.md
# 100 Questions · product brief ## Problem A personal CLI that asks the same questions across four model APIs and compares the answers is weekend-buildable, but matching the product's web-grounded runs, source normalization, failure handling, durable evidence, scoring, and polished reports takes substantially more work. ## Product outcome A benchmark that tells you what it will cost before it runs, asks only the providers whose terms allow what you are doing, keeps every failure visible, and reports nothing you cannot open and read the evidence for. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - OpenAI, Anthropic, Google Gemini, and xAI API keys - provider-specific web search or grounding tools - durable run storage - URL and citation normalization - report generation ## Explicit non-goals for v1 - Providers whose terms forbid storing or analysing grounded results. - Scheduled monitoring and alerting. - Generated recommendations. - Accounts, teams and billing. - reliable orchestration and retries across four providers - normalized citations and evidence-linked metrics - competitor and missed-question extraction - stored point-in-time reports and comparisons - polished exports and action recommendations ## Success criteria - A dry run is confirmed to print worst-case cost per provider - A run over the ceiling is confirmed to refuse to start - Failed cells are confirmed visible with stated denominators - A brand name inside a longer word is confirmed not counted - Every report figure is confirmed to expand to its answer rows
BRIEF.md
# Build brief · 100 Questions The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local benchmark that asks the same buyer questions of several model APIs and reports where your brand appears, with the evidence attached to replace 100 Questions. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two rules, and both are about not lying to yourself. One run is a sample, not a measurement. A web-grounded answer varies between runs because the model is non-deterministic by default (OpenAI says so in writing) and because the live index it searched changed overnight. So a single percentage is a snapshot with a timestamp, never a score. Either repeat each question a stated number of times and report the range and the number of runs beside every figure, or label the whole report as one sample. Never print a bare number that reads like a measurement of how visible you are. Second, the provider's terms decide what you may store and show, and they are not all the same. Anthropic prices its web search tool per thousand searches, requires citations when output is displayed, and requires search result blocks to be sent back byte-exact if a conversation continues. Google's Gemini API terms for Grounding with Google Search are stricter: they forbid caching, analysing or learning from Grounded Results, require them to be shown only to the end user who submitted the prompt, and forbid interspersing other content with them. A tool whose whole purpose is to store answers and compare them side by side does not fit inside that, so read each provider's current terms yourself, record the date, and write down which providers you included and which you left out and why. This is the part that decides the design, not the code. ### Stack (fixed, do not substitute) - Runtime: A Node command line tool with no server and no accounts · this is a measurement instrument, and instruments should be runnable from a script - Storage: SQLite, one file per run, written once and never edited · a benchmark you can rewrite after the fact is not a benchmark - Providers: One adapter per provider behind one ask function, each with its own limits · search pricing, caps and terms differ per provider and the differences are load-bearing - Cost: A ceiling checked before the run and enforced during it · search tools bill per call, and one question can fan out into many - Reporting: A self-contained static HTML file with the raw answers embedded · a report that needs a server to read is a report nobody opens in six months ### Data model (create this before Phase 1) - runs(id, started_at, finished_at, brand, domain, aliases_json, question_set_id, repeats, cost_ceiling, cost_actual, terms_note) - providers_used(run_id, provider, model_id, search_tool, max_searches, price_note, terms_checked_on) · what was asked, and under which terms - questions(id, question_set_id, ordinal, text, source) · generated or supplied, frozen with the run - answers(id, run_id, question_id, provider, repeat_index, requested_at, latency_ms, raw_text, finish_reason, error, tokens_in, tokens_out, searches_used) - citations(id, answer_id, ordinal, raw_url, registrable_domain, canonical_url, title) · normalised alongside the original, never instead of it - mentions(id, answer_id, kind, matched_text, start_offset, end_offset, alias_used) · kind is brand or competitor, and the offsets are what the report highlights - costs(run_id, provider, calls, searches, unit_note, amount_minor) · so the invoice can be checked against the run - Nothing in this schema is updated after the run finishes. A rerun is a new run, and comparison is a join between two of them. ### Phase 1 · A run that knows what it will cost before it starts Build: Freeze the questions, the aliases, the model identifiers and the repeat count into a run row. Before any call, compute the worst case from the current published prices per provider: at the time of writing OpenAI charges per thousand web search calls with search content tokens billed at model rates, Anthropic ten dollars per thousand searches plus tokens, xAI five dollars per thousand web search calls, and Google's grounding is billed per thousand requests above a free allowance that differs by model family. Read every one of these from the vendor's pricing page on the day you build, because they change. Then multiply by questions, repeats and the maximum searches per question, print the figure, and stop the run if it passes the ceiling. Trap: Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. Done when: A dry run prints a worst-case cost per provider and a total, computed from prices recorded in configuration with the date they were checked. A run whose worst case exceeds the ceiling refuses to start and says by how much. Actual calls and searches are counted per provider during the run and stored, so the figures can be compared against the invoice. The run row records model identifiers, search settings and the terms check date for every provider used. Do not build yet: No question generation, mention detection, normalisation or reporting yet. ### Phase 2 · Asking everyone the same thing, and keeping the failures Build: One adapter per provider behind a single ask function returning text, citations, search count and an error. Limit concurrency per provider separately. Retry only failures that could not have been billed, with backoff. Store the raw response text exactly as returned, including any opaque blocks a provider requires to be sent back unmodified if a conversation continues. Trap: Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. Done when: Every question and provider pair has a row, either an answer or an error, with no missing combinations, asserted in the suite. A provider returning errors for half the run produces a report showing those cells as failed and denominators that state the count they used. Raw answer text is stored unmodified, verified by comparing a stored row against a recorded response. Concurrency limits are per provider and a slow provider does not stall the others. Do not build yet: No mention detection, citation normalisation or report yet. ### Phase 3 · Finding your name without fooling yourself Build: Match brand and competitor names on word boundaries, case-insensitively, against an alias list in configuration, and store the offsets of every match. Extract competitor names in a separate structured pass over answers already stored, never during the run, so the extraction can be rerun without paying for the answers again. Trap: Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. Done when: A brand name appearing inside a longer word is not counted, tested with a crafted answer. Every mention row carries offsets that highlight correctly in the report, verified by opening one. Competitor extraction runs over stored answers and can be rerun without new API calls, tested by rerunning with the network disabled. Aliases are configuration, and adding one and rerunning detection changes counts without changing any stored answer. Do not build yet: No citation normalisation or report yet. ### Phase 4 · Citations reduced to something comparable Build: Normalise each citation by resolving the registrable domain with the public suffix list, stripping known tracking parameters, and keeping the canonical form beside the original URL. Compute how often your own domain is cited and which sources appear most, per provider. Trap: Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. Done when: Two forms of the same site, with and without a leading www and with tracking parameters, normalise to one source, tested. A hostname on a multi-part public suffix resolves to the correct registrable domain, tested with a known example. The original URL is retrievable for every citation. Owned-domain citation rate is computed per provider and links to the answers it came from. Do not build yet: No report, export or comparison yet. ### Phase 5 · A report that shows its working Build: Render one self-contained HTML file with the answers embedded, every aggregate expandable to the rows it used, failed cells shown as failures, and each figure carrying the number of runs it averages and the spread across them. Export questions, answers, mentions and citations as CSV. Compare two runs by identifier and show what moved and by how much, alongside the question text so a change in wording is visible as a reason. Trap: Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top. Done when: Every aggregate in the report expands to the answer rows it was computed from, checked for each figure type. Figures show the repeat count and the spread, and a single-repeat run is labelled as one sample throughout. Comparing two runs with different question sets or models shows a warning naming the difference before any number. The report opens from the file system with no server and no network, tested by disconnecting. Do not build yet: Nothing after this. ### Out of scope (and why) - Any provider whose terms forbid storing or analysing its grounded results, which on a reading of the current Gemini API terms includes Grounding with Google Search; check the terms yourself and record what you decided. - Scheduled monitoring and alerting, which turns a measurement instrument into a service with a bill that arrives whether you read the report or not. - Recommendations about what to change, since the tool has no way to know whether a change caused a movement. - Accounts, teams and billing, which is the product being replaced rather than the measurement. ### README must contain - That a single run is one sample of a non-deterministic system searching a changing index, and what that means for acting on a figure - Which providers were included, which were excluded on terms grounds, and the date those terms were read - The current price per search or per grounded request for each provider used, with the date checked and the page it came from - How the cost ceiling works, and that estimates assume the maximum searches per question rather than the average - That every figure in the report expands to the answers behind it, and that failed cells are shown rather than dropped
ARCHITECTURE.md
# Architecture · 100 Questions ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Node command line tool with no server and no accounts | this is a measurement instrument, and instruments should be runnable from a script | | Storage | SQLite, one file per run, written once and never edited | a benchmark you can rewrite after the fact is not a benchmark | | Providers | One adapter per provider behind one ask function, each with its own limits | search pricing, caps and terms differ per provider and the differences are load-bearing | | Cost | A ceiling checked before the run and enforced during it | search tools bill per call, and one question can fan out into many | | Reporting | A self-contained static HTML file with the raw answers embedded | a report that needs a server to read is a report nobody opens in six months | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Cost | Estimation, ceiling and spend tracking. | Any pricing source, provided it carries the date it was checked. | | Providers | One adapter each, with its own limits and terms note. | Any provider whose terms permit storage and analysis, added as an adapter. | | Detection | Mentions, offsets and competitor extraction. | Any matcher, provided it stores offsets and never matches inside words. | | Citations | Registrable domains and canonical forms. | Any normaliser, provided the raw URL survives. | | Report | Self-contained HTML, spread and comparison. | Any renderer, provided every figure expands to its rows. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `OPENAI_API_KEY` · optional, secret · Only if you included this provider. Give the key a spend limit. - `ANTHROPIC_API_KEY` · optional, secret · Only if you included this provider. Its search tool bills per search on top of tokens. - `XAI_API_KEY` · optional, secret · Only if you included this provider. Check its current search pricing, which has been revised. - `RUN_DB_DIR` · required · One SQLite file per run, written once. Never edited afterwards. - `COST_CEILING_USD` · required · The run refuses to start above this and aborts if actual spend reaches it. - `REPEATS` · required · How many times each question is asked per provider. One is allowed and is labelled as one sample. - `MAX_SEARCHES_PER_QUESTION` · required · Passed to providers that support a cap, and used as the worst case in the estimate. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · 100 Questions product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A Node command line tool with no server and no accounts, SQLite, one file per run, written once and never edited, One adapter per provider behind one ask function, each with its own limits, A ceiling checked before the run and enforced during it, A self-contained static HTML file with the raw answers embedded. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. - Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top.
MILESTONES.md
# Delivery milestones · 100 Questions Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A run that knows what it will cost before it starts The tool refuses to begin a run whose worst case exceeds the ceiling you set. ### Steps 1. Record prices in configuration with a checked date They change, and a number in code is a number nobody rechecks. Files: `config/pricing.json`, `src/cost.ts` 2. Estimate on the cap, not the average One comparative question can be ten searches. Files: `src/cost.ts`, `tests/cost.test.ts` 3. Abort mid-run when spend passes the ceiling A ceiling that is only checked at the start is a wish. Files: `src/runner.ts` ### Done when - [ ] A dry run prints worst-case cost per provider - [ ] A run over the ceiling refuses to start with the figure - [ ] Calls and searches are counted and stored per provider - [ ] Model identifiers and the terms date are in the run row ### Watch out - Estimating cost as one search per question is wrong in the direction that hurts: Anthropic's own documentation says a simple factual query takes one to three searches while a comparative one can take ten or more, and OpenAI documents no per-call cap at all, so a hundred questions can quietly become a thousand billable searches. Cap searches per request where the provider offers a cap, estimate on the cap rather than the hope, and track spend as the run proceeds so it can abort. ## M2 · Asking everyone the same thing, and keeping the failures Every question reaches every included provider, and a provider that fails leaves a visible hole rather than a gap. ### Steps 1. One adapter per provider, one ask function The differences between providers belong in the adapters, not in the runner. Files: `src/providers/openai.ts`, `src/providers/anthropic.ts`, `src/providers/xai.ts` 2. Store failures as rows A hole you can see beats a number you cannot check. Files: `src/runner.ts`, `tests/failures.test.ts` 3. Keep provider blocks byte-exact Some providers reject a continued conversation whose result blocks were touched. Files: `src/providers/anthropic.ts` ### Done when - [ ] No question and provider combination is missing - [ ] Failed cells appear in the report with their error - [ ] Denominators state how many answers they used - [ ] Raw answer text is stored unmodified ### Watch out - Dropping a failed cell and computing the percentages over whatever came back silently inflates a provider's score in exactly the runs where it was rate limited, and the report reads as if that provider simply mentions you less. Store the failure as a row with its error, show it as a failed cell in the report, and exclude it from denominators explicitly rather than by accident. ## M3 · Finding your name without fooling yourself A mention is a match you can point at in the text, with its position. ### Steps 1. Match on word boundaries with an alias list Substring matching measures English, not your brand. Files: `src/mentions.ts`, `tests/mentions.test.ts` 2. Store the offsets of every match So the report can show the sentence rather than assert a count. Files: `src/mentions.ts` 3. Extract competitors from stored answers only Detection is cheap to redo; answers are not. Files: `src/competitors.ts` ### Done when - [ ] A name inside a longer word is not counted - [ ] Every mention highlights at its stored offset - [ ] Detection reruns with no network calls - [ ] Adding an alias changes counts without new answers ### Watch out - Substring matching reports a mention every time a short brand name appears inside an ordinary word, and the visibility figure rises with how common that string is in English rather than with anything about your brand. Match on word boundaries against an explicit alias list, store the offset of every match, and read twenty highlighted answers by hand before trusting any aggregate. ## M4 · Citations reduced to something comparable Two answers citing the same site count as the same source. ### Steps 1. Resolve domains with the public suffix list Splitting on dots is wrong for a long list of real suffixes. Files: `src/urls.ts`, `tests/urls.test.ts` 2. Strip tracking parameters from a named list A named list you can read beats a clever guess. Files: `config/tracking-params.json`, `src/urls.ts` 3. Keep the raw URL beside the normalised one Normalisation is an opinion and you may want to change it. Files: `src/urls.ts` ### Done when - [ ] www and tracking variants collapse to one source - [ ] A multi-part public suffix resolves correctly - [ ] The original URL is retrievable for every citation - [ ] Owned-domain rate links to its answers ### Watch out - Taking the hostname by splitting on dots treats a site on a multi-part public suffix as a different source per subdomain, so a blog platform appears fifty times as fifty sources and your own site splits into www and bare forms. Use the public suffix list, keep the raw URL alongside the normalised one, and never overwrite the original. ## M5 · A report that shows its working (production only) Every figure in the report can be opened to the answers behind it, and repeats are visible as a range. ### Steps 1. Embed the evidence in the report file A report that needs the database is a report that dies with it. Files: `src/report.ts`, `src/report-template.html` 2. Show repeats and spread beside every figure One run is a sample, and the report should say so itself. Files: `src/report.ts` 3. Warn before comparing runs that differ Different questions is the most common reason a number moved. Files: `src/compare.ts`, `tests/compare.test.ts` ### Done when - [ ] Every figure expands to its answer rows - [ ] Repeat count and spread appear beside each figure - [ ] Comparing mismatched runs warns first - [ ] The report opens offline from the file system ### Watch out - Reporting a single visibility percentage from one run invites someone to act on a three point change that is entirely noise, because the same questions asked an hour later against a changed index give a different answer. Print the number of repeats and the range next to every figure, and refuse to display a comparison between two runs whose question sets or model identifiers differ without saying so at the top.
OPERATIONS.md
# Operations · 100 Questions ## Backup Run databases and rendered reports, copied off the machine. They are the only record of what a provider said on a date, and no provider will tell you again. ## Restore Restore a run file and open its report. If the report needs anything except the file itself, the report is wrong and the phase five test should have caught it. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Spend against ceiling per run, failed cells per provider, and the age of the recorded prices and terms dates. All three go stale quietly. ## Incident checklist The incidents are a surprise invoice and a figure somebody acted on. The costs table answers the first against the provider's own bill; the evidence links answer the second by showing the answers the figure came from. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A dry run is confirmed to print worst-case cost per provider - [ ] A run over the ceiling is confirmed to refuse to start - [ ] Failed cells are confirmed visible with stated denominators - [ ] A brand name inside a longer word is confirmed not counted - [ ] Every report figure is confirmed to expand to its answer rows ## Launch constraint Do not market omitted 100 Questions capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. Only if you included this provider. Give the key a spend limit. OPENAI_API_KEY=sk-...redacted # Optional · secret. Only if you included this provider. Its search tool bills per search on top of tokens. ANTHROPIC_API_KEY=sk-ant-...redacted # Optional · secret. Only if you included this provider. Check its current search pricing, which has been revised. XAI_API_KEY=xai-...redacted # Required. One SQLite file per run, written once. Never edited afterwards. RUN_DB_DIR=./runs # Required. The run refuses to start above this and aborts if actual spend reaches it. COST_CEILING_USD=15 # Required. How many times each question is asked per provider. One is allowed and is labelled as one sample. REPEATS=3 # Required. Passed to providers that support a cap, and used as the worst case in the estimate. MAX_SEARCHES_PER_QUESTION=4
$ choose a build depth, inspect the files, then open the complete pack in your agent
They pay for a repeatable, frozen benchmark with provider failures handled, citations normalized, every metric tied to evidence, and a report that is ready to act on.
xreliable orchestration and retries across four providers
xnormalized citations and evidence-linked metrics
xcompetitor and missed-question extraction
xstored point-in-time reports and comparisons
xpolished exports and action recommendations
share on X ↗"I just replaced 100 Questions ($9/mo) with one prompt"
Don't feel like building it? These folks already made it free.
ElmoPrompt-by-prompt AI visibility with citations and competitors across the major engines; the queries still need model or scraper credentials.open source↗no votes, no pay-to-list · just what's real
Vibecode 100 Questions
Kinda. The core of 100 Questions is buildable in a weekend with the prompt on this page, but there are real gaps: reliable orchestration and retries across four providers, normalized citations and evidence-linked metrics. Read the honest list above before committing.
How much does 100 Questions cost?
100 Questions costs about $9/month (First benchmark, checked 2026-07-31), which is $108 per year.
What do I lose by replacing 100 Questions?
Honestly: reliable orchestration and retries across four providers; normalized citations and evidence-linked metrics; competitor and missed-question extraction; stored point-in-time reports and comparisons; polished exports and action recommendations. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to 100 Questions?
Yes: Elmo (Prompt-by-prompt AI visibility with citations and competitors across the major engines; the queries still need model or scraper credentials.) The prompt is for when you want it exactly your way.