Vibecode BrandGEO
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
Running a fixed battery of brand questions through five model APIs and scoring the answers with a second LLM pass is a genuine weekend build, and for one brand it answers the headline question: what does AI say about us. The gaps are the ones every tracker in this category shares. API answers approximate but do not equal the consumer apps, a score with no trend history behind it is a screenshot rather than a signal, and a rubric only becomes comparable after it has scored many brands.
the project pack5 files · written for this build, step by step
You are building a lean indie version of BrandGEO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # BrandGEO · indie build A local auditor whose hard problems are treating non-deterministic model output as a sample rather than a measurement, keeping every run immutable so a second week means something, and scoring answers against a rubric where each score quotes the sentence behind it. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript and a command line interface, local only | an audit is a script you run, not a service somebody logs into | | Storage | SQLite with one immutable row per run, question, engine and sample | overwriting a previous run destroys the only part that gets more valuable | | Engines | One module per provider behind a single call interface, every key optional | a missing key should narrow the report, not fail the run | | Sampling | Several samples per question per engine, the count recorded on the run | the providers document that identical calls need not return identical text | | Scoring | A second pass against a rubric file, each score quoting a sentence | an unquoted score is a number somebody will put in a slide | | Cost | An estimate printed and confirmed before any call, and a hard ceiling enforced | questions times engines times samples is larger than it looks | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the command line tool. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **API keys for at least two providers** · pennies per run at small sample counts, and it multiplies - Why: One engine is an anecdote; the disagreement between engines is the actual finding, and every key stays optional so a run narrows rather than fails. - Get it: Create a key in each provider's console and put it in .env; leave the rest unset. - Verify: A single cheap question returns an answer from each configured engine - [ ] **A written rubric with per dimension definitions** · free - Why: The rubric is hashed onto every run, so writing it after seeing the first scores means writing whatever the scores already said. - Get it: Write rubric.md defining each of the six dimensions and what a 0, a 5 and a 10 look like. - Verify: Two people scoring the same answer by hand land within two points on each dimension - [ ] **A brand you know well enough to catch errors** · free - Why: The most useful output is the list of things the models got wrong, and you can only produce it about a brand whose facts you already know. - Get it: Your own product, or one whose history you can check quickly. - Verify: You can confirm or refute every factual claim in one sample answer - [ ] **A cost ceiling you are willing to lose** · free - Why: Questions times engines times samples is the arithmetic that surprises people, and the ceiling is enforced in code rather than remembered. - Get it: Pick a figure per run, put it in the environment, and check the printed estimate against it before confirming. - Verify: A run configured to exceed it stops and reports what it completed ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - White label PDF reports. - Multi brand management. - Scheduled monitoring. - Any driving of the consumer chat interfaces. - white-label PDF reports an agency can hand to a client - weekly monitoring that keeps running when nobody is thinking about it - a rubric calibrated across many brands, so scores are comparable - competitor benchmarks per brand - the consumer app surfaces, which no API exactly reproduces If one of those is essential to you, that is the reason to keep paying for BrandGEO, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · BrandGEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local brand visibility auditor that reports the spread of what model APIs say, not a single score to replace BrandGEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three things this tool must never pretend to be. It is not a measurement of one number. The providers say so themselves: Anthropic's API reference states that even with a temperature of 0.0 the results will not be fully deterministic, and that models released after Claude Opus 4.6 reject any temperature other than 1.0 outright. One answer to one question is a sample. Ask each question several times per engine and report the range you got back. It is not what users see. A consumer chat product wraps retrieval, tools and product surfaces around the model you are calling, so the report says what the API returned on a given date and makes no claim about the app. And it does not close that gap by driving the chat interfaces. That breaks their terms, breaks every few weeks, and produces a number worth less than an honestly stated gap. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript and a command line interface, local only · an audit is a script you run, not a service somebody logs into - Storage: SQLite with one immutable row per run, question, engine and sample · overwriting a previous run destroys the only part that gets more valuable - Engines: One module per provider behind a single call interface, every key optional · a missing key should narrow the report, not fail the run - Sampling: Several samples per question per engine, the count recorded on the run · the providers document that identical calls need not return identical text - Scoring: A second pass against a rubric file, each score quoting a sentence · an unquoted score is a number somebody will put in a slide - Cost: An estimate printed and confirmed before any call, and a hard ceiling enforced · questions times engines times samples is larger than it looks ### Data model (create this before Phase 1) - brands(id, name, aliases_json, domain, competitors_json) - questions(id, brand_id, text, category, active) · fixed between runs, because an edited question starts a new series - runs(id, brand_id, started_at, finished_at, samples_per_question, rubric_sha256, engines_json, estimated_cost, actual_cost) - answers(id, run_id, question_id, engine, model_id, sample_index, raw_text, latency_ms, error_text, at) · never updated and never deleted - scores(answer_id, dimension, value, quote, scored_by_model, at) · every score carries the sentence that justifies it - aggregates(run_id, engine, dimension, median, low, high, n) · what the report prints, because the spread is the finding - A run pins the rubric checksum and the question set, so changing either visibly starts a new series rather than quietly rewriting the old one. ### Phase 1 · A brand file and a store that never forgets Build: A brand file holding the name, aliases, domain, competitors and up to twenty five questions. A run captures the brand, the question set, the rubric checksum and which engines were available. Answers are insert only, with the model id and any error text kept beside them. Trap: Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. Done when: No code path updates or deletes an answer row, verified by review and by a trigger that rejects it. A run records the rubric checksum and the exact question set it used. An engine with a missing key is recorded as skipped with a reason rather than omitted silently. Re-running the same questions creates a second run and leaves the first byte identical. Do not build yet: No sampling, scoring, reports or history. ### Phase 2 · Asking several times, and knowing the bill first Build: Ask each question a configured number of times per engine. Cap concurrency per engine, retry rate limit and server errors with backoff, and record failures as rows. Print a cost estimate and require confirmation before the first call, then stop hard at the ceiling. Trap: A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. Done when: A run killed halfway resumes and asks only for missing samples, tested against a fake engine. The estimate is printed with the arithmetic shown, and no call is made until it is confirmed. Reaching the cost ceiling stops the run and reports what was completed. Rate limited and failed calls appear as rows with their error text rather than disappearing. Do not build yet: No scoring, reports or history yet. ### Phase 3 · Scores that quote the sentence Build: Score each stored answer against the rubric file on six dimensions. Every score must return a quote, and the quote is checked for presence in that answer before the score is saved. Scores that fail the check are discarded and counted, not repaired. Trap: A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. Done when: Every saved score has a quote confirmed present in its answer, asserted in the suite. A deliberately fabricated quote is rejected by the check, tested with a fixture. The rubric lives in a file whose checksum is stored on the run, and editing it changes the checksum. The count of discarded scores appears in the report rather than in a log. Do not build yet: No reports, history or exports yet. ### Phase 4 · A report that shows the spread Build: Render a Markdown report: per engine and dimension, the median with the low and high across samples and the sample count. List the wrong claims models made about the brand, quoted. Name every engine that was skipped and why. No single overall score anywhere. Trap: Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. Done when: No single overall score is produced by any code path, verified by review. Every figure in the report is accompanied by its sample count and its low and high. Wrong claims are quoted verbatim with the engine and the sample index. Skipped engines are listed with the reason in the report body, not a footnote. Do not build yet: No trend history or exports yet. ### Phase 5 · A series that stays comparable Build: A history command plotting each engine and dimension across runs. Break the line visibly when the rubric checksum, the question set or an engine's model id changes. Export every run as JSON and CSV. Print total spend per run and per engine. Trap: Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes. Done when: A change of rubric checksum, question set or model id breaks the series visibly in the history output, tested by changing each. Exports contain every raw answer, not only the scores. Spend is reported per run and per engine against the estimate. The report states its own limitations in the body, including that API answers are not what the consumer apps show. Do not build yet: Nothing after this. ### Out of scope (and why) - White label PDF reports for clients, which is the part agencies actually pay for and is a document production problem rather than an audit one. - Managing many brands, which turns a script into a product with tenancy, quotas and a support burden. - Scheduled monitoring, which needs a machine that is always on and an alerting story, and is better added once one run has proven useful. - Driving the consumer chat interfaces to see what users see, which breaks their terms and breaks weekly. ### README must contain - That model output is not deterministic, that this is why each question is sampled several times, and where the providers state it - That API answers are not what the consumer apps show, stated plainly and not buried - The per run cost arithmetic by engine, and where the ceiling is enforced - That every score quotes a sentence verified to appear in the answer, and how many were discarded - What breaks a trend series: a rubric change, a question change or a new model id ===== AGENTS.md ===== # Agent instructions · BrandGEO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript and a command line interface, local only, SQLite with one immutable row per run, question, engine and sample, One module per provider behind a single call interface, every key optional, Several samples per question per engine, the count recorded on the run, A second pass against a rubric file, each score quoting a sentence, An estimate printed and confirmed before any call, and a hard ceiling enforced. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. ===== BUILD_PLAN.md ===== # Build plan · BrandGEO A local auditor whose hard problems are treating non-deterministic model output as a sample rather than a measurement, keeping every run immutable so a second week means something, and scoring answers against a rubric where each score quotes the sentence behind it. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A brand file and a store that never forgets Every answer ever received is kept, attributed and immutable. ### Steps 1. Make answers insert only in the schema Enforce it in the database, not in the discipline of the caller. Files: `src/db/schema.sql`, `tests/immutable-answers.test.ts` 2. Pin the rubric and question set to the run Two runs scored differently are not two points on one line. Files: `src/run.ts` 3. Record skipped engines with reasons A narrower report is fine; a silently narrower one is not. Files: `src/engines/registry.ts` ### Done when - [ ] Answer rows cannot be updated or deleted - [ ] Runs pin the rubric checksum and question set - [ ] Missing keys produce a recorded skip with a reason - [ ] A second run leaves the first unchanged ### Watch out - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. ## Phase 2 · Asking several times, and knowing the bill first Each question is sampled repeatedly within a cost ceiling you agreed to. ### Steps 1. Write each answer as it arrives So money already spent is money already stored. Files: `src/sample.ts` 2. Estimate and confirm before the first call Show the multiplication, because that is where the surprise lives. Files: `src/cost.ts` 3. Back off on rate limits, record every failure A failure row is data about the engine, not an absence of data. Files: `src/engines/call.ts`, `tests/resume.test.ts` ### Done when - [ ] A killed run resumes without re-asking - [ ] The estimate is shown and confirmed before calling - [ ] The cost ceiling stops the run and reports progress - [ ] Failures are stored as rows with error text ### Watch out - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. ## Phase 3 · Scores that quote the sentence No dimension score exists without the text that justifies it. ### Steps 1. Verify the quote before saving the score A substring check catches nearly all of it and costs nothing. Files: `src/score/verify.ts`, `tests/quote-check.test.ts` 2. Keep the rubric in a file, hashed per run Rubric drift is the quietest way to ruin a trend line. Files: `rubric.md`, `src/rubric.ts` 3. Count and report discards The discard rate tells you when a scoring model has changed under you. Files: `src/score/index.ts` ### Done when - [ ] Every score has a verified quote - [ ] Fabricated quotes are rejected - [ ] The rubric checksum is stored per run - [ ] Discard counts appear in the report ### Watch out - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. ## Phase 4 · A report that shows the spread The output tells you how much the engines disagreed, including with themselves. ### Steps 1. Report median with low and high Three numbers and an n say what one number cannot. Files: `src/report/aggregate.ts` 2. Quote wrong claims verbatim This is the part a founder actually reads and acts on. Files: `src/report/errors.ts` 3. Refuse to compute one headline number Not a setting, not a flag; there is no code that does it. Files: `src/report/index.ts` ### Done when - [ ] No overall single score exists anywhere - [ ] Every figure shows n, low and high - [ ] Wrong claims are quoted with engine and sample - [ ] Skipped engines appear in the report body ### Watch out - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. ## Not in this build - White label PDF reports. - Multi brand management. - Scheduled monitoring. - Any driving of the consumer chat interfaces. ## After v1, if you want it - A second brand as a comparison, once one brand has produced three runs worth trusting - Prompt variants per question, to see how much the wording rather than the brand moves the answer - A weekly scheduled run, once you have a machine that is always on and a reason to look at it ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, answers and scores. Answers are insert only. DATABASE_PATH=./data/audits.db # Required. Name, aliases, domain, competitors and the question set. BRAND_FILE=./brand.json # Required. Hashed onto every run. Editing it starts a new series. RUBRIC_PATH=./rubric.md # Required. Per engine. One is a sample and it will read like a measurement. SAMPLES_PER_QUESTION=5 # Required. Keeps you inside published rate limits without needing to know them exactly. MAX_CONCURRENCY_PER_ENGINE=2 # Required. In your billing currency. Enforced in code, not in memory. COST_CEILING=10.00 # Optional · secret. Optional. Unset means that engine is recorded as skipped. OPENAI_API_KEY=sk-...redacted # Optional · secret. Optional, as are the other provider keys, which follow the same pattern. ANTHROPIC_API_KEY=sk-ant-...redacted
You are building a lean indie version of BrandGEO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # BrandGEO · indie build A local auditor whose hard problems are treating non-deterministic model output as a sample rather than a measurement, keeping every run immutable so a second week means something, and scoring answers against a rubric where each score quotes the sentence behind it. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript and a command line interface, local only | an audit is a script you run, not a service somebody logs into | | Storage | SQLite with one immutable row per run, question, engine and sample | overwriting a previous run destroys the only part that gets more valuable | | Engines | One module per provider behind a single call interface, every key optional | a missing key should narrow the report, not fail the run | | Sampling | Several samples per question per engine, the count recorded on the run | the providers document that identical calls need not return identical text | | Scoring | A second pass against a rubric file, each score quoting a sentence | an unquoted score is a number somebody will put in a slide | | Cost | An estimate printed and confirmed before any call, and a hard ceiling enforced | questions times engines times samples is larger than it looks | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the command line tool. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **API keys for at least two providers** · pennies per run at small sample counts, and it multiplies - Why: One engine is an anecdote; the disagreement between engines is the actual finding, and every key stays optional so a run narrows rather than fails. - Get it: Create a key in each provider's console and put it in .env; leave the rest unset. - Verify: A single cheap question returns an answer from each configured engine - [ ] **A written rubric with per dimension definitions** · free - Why: The rubric is hashed onto every run, so writing it after seeing the first scores means writing whatever the scores already said. - Get it: Write rubric.md defining each of the six dimensions and what a 0, a 5 and a 10 look like. - Verify: Two people scoring the same answer by hand land within two points on each dimension - [ ] **A brand you know well enough to catch errors** · free - Why: The most useful output is the list of things the models got wrong, and you can only produce it about a brand whose facts you already know. - Get it: Your own product, or one whose history you can check quickly. - Verify: You can confirm or refute every factual claim in one sample answer - [ ] **A cost ceiling you are willing to lose** · free - Why: Questions times engines times samples is the arithmetic that surprises people, and the ceiling is enforced in code rather than remembered. - Get it: Pick a figure per run, put it in the environment, and check the printed estimate against it before confirming. - Verify: A run configured to exceed it stops and reports what it completed ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - White label PDF reports. - Multi brand management. - Scheduled monitoring. - Any driving of the consumer chat interfaces. - white-label PDF reports an agency can hand to a client - weekly monitoring that keeps running when nobody is thinking about it - a rubric calibrated across many brands, so scores are comparable - competitor benchmarks per brand - the consumer app surfaces, which no API exactly reproduces If one of those is essential to you, that is the reason to keep paying for BrandGEO, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · BrandGEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local brand visibility auditor that reports the spread of what model APIs say, not a single score to replace BrandGEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three things this tool must never pretend to be. It is not a measurement of one number. The providers say so themselves: Anthropic's API reference states that even with a temperature of 0.0 the results will not be fully deterministic, and that models released after Claude Opus 4.6 reject any temperature other than 1.0 outright. One answer to one question is a sample. Ask each question several times per engine and report the range you got back. It is not what users see. A consumer chat product wraps retrieval, tools and product surfaces around the model you are calling, so the report says what the API returned on a given date and makes no claim about the app. And it does not close that gap by driving the chat interfaces. That breaks their terms, breaks every few weeks, and produces a number worth less than an honestly stated gap. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript and a command line interface, local only · an audit is a script you run, not a service somebody logs into - Storage: SQLite with one immutable row per run, question, engine and sample · overwriting a previous run destroys the only part that gets more valuable - Engines: One module per provider behind a single call interface, every key optional · a missing key should narrow the report, not fail the run - Sampling: Several samples per question per engine, the count recorded on the run · the providers document that identical calls need not return identical text - Scoring: A second pass against a rubric file, each score quoting a sentence · an unquoted score is a number somebody will put in a slide - Cost: An estimate printed and confirmed before any call, and a hard ceiling enforced · questions times engines times samples is larger than it looks ### Data model (create this before Phase 1) - brands(id, name, aliases_json, domain, competitors_json) - questions(id, brand_id, text, category, active) · fixed between runs, because an edited question starts a new series - runs(id, brand_id, started_at, finished_at, samples_per_question, rubric_sha256, engines_json, estimated_cost, actual_cost) - answers(id, run_id, question_id, engine, model_id, sample_index, raw_text, latency_ms, error_text, at) · never updated and never deleted - scores(answer_id, dimension, value, quote, scored_by_model, at) · every score carries the sentence that justifies it - aggregates(run_id, engine, dimension, median, low, high, n) · what the report prints, because the spread is the finding - A run pins the rubric checksum and the question set, so changing either visibly starts a new series rather than quietly rewriting the old one. ### Phase 1 · A brand file and a store that never forgets Build: A brand file holding the name, aliases, domain, competitors and up to twenty five questions. A run captures the brand, the question set, the rubric checksum and which engines were available. Answers are insert only, with the model id and any error text kept beside them. Trap: Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. Done when: No code path updates or deletes an answer row, verified by review and by a trigger that rejects it. A run records the rubric checksum and the exact question set it used. An engine with a missing key is recorded as skipped with a reason rather than omitted silently. Re-running the same questions creates a second run and leaves the first byte identical. Do not build yet: No sampling, scoring, reports or history. ### Phase 2 · Asking several times, and knowing the bill first Build: Ask each question a configured number of times per engine. Cap concurrency per engine, retry rate limit and server errors with backoff, and record failures as rows. Print a cost estimate and require confirmation before the first call, then stop hard at the ceiling. Trap: A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. Done when: A run killed halfway resumes and asks only for missing samples, tested against a fake engine. The estimate is printed with the arithmetic shown, and no call is made until it is confirmed. Reaching the cost ceiling stops the run and reports what was completed. Rate limited and failed calls appear as rows with their error text rather than disappearing. Do not build yet: No scoring, reports or history yet. ### Phase 3 · Scores that quote the sentence Build: Score each stored answer against the rubric file on six dimensions. Every score must return a quote, and the quote is checked for presence in that answer before the score is saved. Scores that fail the check are discarded and counted, not repaired. Trap: A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. Done when: Every saved score has a quote confirmed present in its answer, asserted in the suite. A deliberately fabricated quote is rejected by the check, tested with a fixture. The rubric lives in a file whose checksum is stored on the run, and editing it changes the checksum. The count of discarded scores appears in the report rather than in a log. Do not build yet: No reports, history or exports yet. ### Phase 4 · A report that shows the spread Build: Render a Markdown report: per engine and dimension, the median with the low and high across samples and the sample count. List the wrong claims models made about the brand, quoted. Name every engine that was skipped and why. No single overall score anywhere. Trap: Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. Done when: No single overall score is produced by any code path, verified by review. Every figure in the report is accompanied by its sample count and its low and high. Wrong claims are quoted verbatim with the engine and the sample index. Skipped engines are listed with the reason in the report body, not a footnote. Do not build yet: No trend history or exports yet. ### Phase 5 · A series that stays comparable Build: A history command plotting each engine and dimension across runs. Break the line visibly when the rubric checksum, the question set or an engine's model id changes. Export every run as JSON and CSV. Print total spend per run and per engine. Trap: Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes. Done when: A change of rubric checksum, question set or model id breaks the series visibly in the history output, tested by changing each. Exports contain every raw answer, not only the scores. Spend is reported per run and per engine against the estimate. The report states its own limitations in the body, including that API answers are not what the consumer apps show. Do not build yet: Nothing after this. ### Out of scope (and why) - White label PDF reports for clients, which is the part agencies actually pay for and is a document production problem rather than an audit one. - Managing many brands, which turns a script into a product with tenancy, quotas and a support burden. - Scheduled monitoring, which needs a machine that is always on and an alerting story, and is better added once one run has proven useful. - Driving the consumer chat interfaces to see what users see, which breaks their terms and breaks weekly. ### README must contain - That model output is not deterministic, that this is why each question is sampled several times, and where the providers state it - That API answers are not what the consumer apps show, stated plainly and not buried - The per run cost arithmetic by engine, and where the ceiling is enforced - That every score quotes a sentence verified to appear in the answer, and how many were discarded - What breaks a trend series: a rubric change, a question change or a new model id ===== AGENTS.md ===== # Agent instructions · BrandGEO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript and a command line interface, local only, SQLite with one immutable row per run, question, engine and sample, One module per provider behind a single call interface, every key optional, Several samples per question per engine, the count recorded on the run, A second pass against a rubric file, each score quoting a sentence, An estimate printed and confirmed before any call, and a hard ceiling enforced. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. ===== BUILD_PLAN.md ===== # Build plan · BrandGEO A local auditor whose hard problems are treating non-deterministic model output as a sample rather than a measurement, keeping every run immutable so a second week means something, and scoring answers against a rubric where each score quotes the sentence behind it. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A brand file and a store that never forgets Every answer ever received is kept, attributed and immutable. ### Steps 1. Make answers insert only in the schema Enforce it in the database, not in the discipline of the caller. Files: `src/db/schema.sql`, `tests/immutable-answers.test.ts` 2. Pin the rubric and question set to the run Two runs scored differently are not two points on one line. Files: `src/run.ts` 3. Record skipped engines with reasons A narrower report is fine; a silently narrower one is not. Files: `src/engines/registry.ts` ### Done when - [ ] Answer rows cannot be updated or deleted - [ ] Runs pin the rubric checksum and question set - [ ] Missing keys produce a recorded skip with a reason - [ ] A second run leaves the first unchanged ### Watch out - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. ## Phase 2 · Asking several times, and knowing the bill first Each question is sampled repeatedly within a cost ceiling you agreed to. ### Steps 1. Write each answer as it arrives So money already spent is money already stored. Files: `src/sample.ts` 2. Estimate and confirm before the first call Show the multiplication, because that is where the surprise lives. Files: `src/cost.ts` 3. Back off on rate limits, record every failure A failure row is data about the engine, not an absence of data. Files: `src/engines/call.ts`, `tests/resume.test.ts` ### Done when - [ ] A killed run resumes without re-asking - [ ] The estimate is shown and confirmed before calling - [ ] The cost ceiling stops the run and reports progress - [ ] Failures are stored as rows with error text ### Watch out - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. ## Phase 3 · Scores that quote the sentence No dimension score exists without the text that justifies it. ### Steps 1. Verify the quote before saving the score A substring check catches nearly all of it and costs nothing. Files: `src/score/verify.ts`, `tests/quote-check.test.ts` 2. Keep the rubric in a file, hashed per run Rubric drift is the quietest way to ruin a trend line. Files: `rubric.md`, `src/rubric.ts` 3. Count and report discards The discard rate tells you when a scoring model has changed under you. Files: `src/score/index.ts` ### Done when - [ ] Every score has a verified quote - [ ] Fabricated quotes are rejected - [ ] The rubric checksum is stored per run - [ ] Discard counts appear in the report ### Watch out - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. ## Phase 4 · A report that shows the spread The output tells you how much the engines disagreed, including with themselves. ### Steps 1. Report median with low and high Three numbers and an n say what one number cannot. Files: `src/report/aggregate.ts` 2. Quote wrong claims verbatim This is the part a founder actually reads and acts on. Files: `src/report/errors.ts` 3. Refuse to compute one headline number Not a setting, not a flag; there is no code that does it. Files: `src/report/index.ts` ### Done when - [ ] No overall single score exists anywhere - [ ] Every figure shows n, low and high - [ ] Wrong claims are quoted with engine and sample - [ ] Skipped engines appear in the report body ### Watch out - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. ## Not in this build - White label PDF reports. - Multi brand management. - Scheduled monitoring. - Any driving of the consumer chat interfaces. ## After v1, if you want it - A second brand as a comparison, once one brand has produced three runs worth trusting - Prompt variants per question, to see how much the wording rather than the brand moves the answer - A weekly scheduled run, once you have a machine that is always on and a reason to look at it ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, answers and scores. Answers are insert only. DATABASE_PATH=./data/audits.db # Required. Name, aliases, domain, competitors and the question set. BRAND_FILE=./brand.json # Required. Hashed onto every run. Editing it starts a new series. RUBRIC_PATH=./rubric.md # Required. Per engine. One is a sample and it will read like a measurement. SAMPLES_PER_QUESTION=5 # Required. Keeps you inside published rate limits without needing to know them exactly. MAX_CONCURRENCY_PER_ENGINE=2 # Required. In your billing currency. Enforced in code, not in memory. COST_CEILING=10.00 # Optional · secret. Optional. Unset means that engine is recorded as skipped. OPENAI_API_KEY=sk-...redacted # Optional · secret. Optional, as are the other provider keys, which follow the same pattern. ANTHROPIC_API_KEY=sk-ant-...redacted
You are building a production product version of BrandGEO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # BrandGEO · product brief ## Problem Running a fixed battery of brand questions through five model APIs and scoring the answers with a second LLM pass is a genuine weekend build, and for one brand it answers the headline question: what does AI say about us. The gaps are the ones every tracker in this category shares. API answers approximate but do not equal the consumer apps, a score with no trend history behind it is a screenshot rather than a signal, and a rubric only becomes comparable after it has scored many brands. ## Product outcome An auditor that samples each question several times per engine, keeps every raw answer forever, scores only with quotes it verified, reports a spread instead of a headline number, and says out loud where it differs from what users actually see. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - API keys for OpenAI, Anthropic, Gemini, xAI (Grok), and DeepSeek - a written scoring rubric with per-dimension definitions - durable per-run storage for trend history - a scheduler for recurring audits - an API budget that scales with prompts x engines ## Explicit non-goals for v1 - White label PDF reports. - Multi brand management. - Scheduled monitoring. - Any driving of the consumer chat interfaces. - white-label PDF reports an agency can hand to a client - weekly monitoring that keeps running when nobody is thinking about it - a rubric calibrated across many brands, so scores are comparable - competitor benchmarks per brand - the consumer app surfaces, which no API exactly reproduces ## Success criteria - Answers are confirmed insert only - A killed run is confirmed to resume without re-asking - Every score is confirmed to carry a verified quote - No single overall score is confirmed to exist - A model id change is confirmed to break the series ===== BRIEF.md ===== # Build brief · BrandGEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local brand visibility auditor that reports the spread of what model APIs say, not a single score to replace BrandGEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three things this tool must never pretend to be. It is not a measurement of one number. The providers say so themselves: Anthropic's API reference states that even with a temperature of 0.0 the results will not be fully deterministic, and that models released after Claude Opus 4.6 reject any temperature other than 1.0 outright. One answer to one question is a sample. Ask each question several times per engine and report the range you got back. It is not what users see. A consumer chat product wraps retrieval, tools and product surfaces around the model you are calling, so the report says what the API returned on a given date and makes no claim about the app. And it does not close that gap by driving the chat interfaces. That breaks their terms, breaks every few weeks, and produces a number worth less than an honestly stated gap. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript and a command line interface, local only · an audit is a script you run, not a service somebody logs into - Storage: SQLite with one immutable row per run, question, engine and sample · overwriting a previous run destroys the only part that gets more valuable - Engines: One module per provider behind a single call interface, every key optional · a missing key should narrow the report, not fail the run - Sampling: Several samples per question per engine, the count recorded on the run · the providers document that identical calls need not return identical text - Scoring: A second pass against a rubric file, each score quoting a sentence · an unquoted score is a number somebody will put in a slide - Cost: An estimate printed and confirmed before any call, and a hard ceiling enforced · questions times engines times samples is larger than it looks ### Data model (create this before Phase 1) - brands(id, name, aliases_json, domain, competitors_json) - questions(id, brand_id, text, category, active) · fixed between runs, because an edited question starts a new series - runs(id, brand_id, started_at, finished_at, samples_per_question, rubric_sha256, engines_json, estimated_cost, actual_cost) - answers(id, run_id, question_id, engine, model_id, sample_index, raw_text, latency_ms, error_text, at) · never updated and never deleted - scores(answer_id, dimension, value, quote, scored_by_model, at) · every score carries the sentence that justifies it - aggregates(run_id, engine, dimension, median, low, high, n) · what the report prints, because the spread is the finding - A run pins the rubric checksum and the question set, so changing either visibly starts a new series rather than quietly rewriting the old one. ### Phase 1 · A brand file and a store that never forgets Build: A brand file holding the name, aliases, domain, competitors and up to twenty five questions. A run captures the brand, the question set, the rubric checksum and which engines were available. Answers are insert only, with the model id and any error text kept beside them. Trap: Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. Done when: No code path updates or deletes an answer row, verified by review and by a trigger that rejects it. A run records the rubric checksum and the exact question set it used. An engine with a missing key is recorded as skipped with a reason rather than omitted silently. Re-running the same questions creates a second run and leaves the first byte identical. Do not build yet: No sampling, scoring, reports or history. ### Phase 2 · Asking several times, and knowing the bill first Build: Ask each question a configured number of times per engine. Cap concurrency per engine, retry rate limit and server errors with backoff, and record failures as rows. Print a cost estimate and require confirmation before the first call, then stop hard at the ceiling. Trap: A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. Done when: A run killed halfway resumes and asks only for missing samples, tested against a fake engine. The estimate is printed with the arithmetic shown, and no call is made until it is confirmed. Reaching the cost ceiling stops the run and reports what was completed. Rate limited and failed calls appear as rows with their error text rather than disappearing. Do not build yet: No scoring, reports or history yet. ### Phase 3 · Scores that quote the sentence Build: Score each stored answer against the rubric file on six dimensions. Every score must return a quote, and the quote is checked for presence in that answer before the score is saved. Scores that fail the check are discarded and counted, not repaired. Trap: A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. Done when: Every saved score has a quote confirmed present in its answer, asserted in the suite. A deliberately fabricated quote is rejected by the check, tested with a fixture. The rubric lives in a file whose checksum is stored on the run, and editing it changes the checksum. The count of discarded scores appears in the report rather than in a log. Do not build yet: No reports, history or exports yet. ### Phase 4 · A report that shows the spread Build: Render a Markdown report: per engine and dimension, the median with the low and high across samples and the sample count. List the wrong claims models made about the brand, quoted. Name every engine that was skipped and why. No single overall score anywhere. Trap: Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. Done when: No single overall score is produced by any code path, verified by review. Every figure in the report is accompanied by its sample count and its low and high. Wrong claims are quoted verbatim with the engine and the sample index. Skipped engines are listed with the reason in the report body, not a footnote. Do not build yet: No trend history or exports yet. ### Phase 5 · A series that stays comparable Build: A history command plotting each engine and dimension across runs. Break the line visibly when the rubric checksum, the question set or an engine's model id changes. Export every run as JSON and CSV. Print total spend per run and per engine. Trap: Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes. Done when: A change of rubric checksum, question set or model id breaks the series visibly in the history output, tested by changing each. Exports contain every raw answer, not only the scores. Spend is reported per run and per engine against the estimate. The report states its own limitations in the body, including that API answers are not what the consumer apps show. Do not build yet: Nothing after this. ### Out of scope (and why) - White label PDF reports for clients, which is the part agencies actually pay for and is a document production problem rather than an audit one. - Managing many brands, which turns a script into a product with tenancy, quotas and a support burden. - Scheduled monitoring, which needs a machine that is always on and an alerting story, and is better added once one run has proven useful. - Driving the consumer chat interfaces to see what users see, which breaks their terms and breaks weekly. ### README must contain - That model output is not deterministic, that this is why each question is sampled several times, and where the providers state it - That API answers are not what the consumer apps show, stated plainly and not buried - The per run cost arithmetic by engine, and where the ceiling is enforced - That every score quotes a sentence verified to appear in the answer, and how many were discarded - What breaks a trend series: a rubric change, a question change or a new model id ===== ARCHITECTURE.md ===== # Architecture · BrandGEO ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript and a command line interface, local only | an audit is a script you run, not a service somebody logs into | | Storage | SQLite with one immutable row per run, question, engine and sample | overwriting a previous run destroys the only part that gets more valuable | | Engines | One module per provider behind a single call interface, every key optional | a missing key should narrow the report, not fail the run | | Sampling | Several samples per question per engine, the count recorded on the run | the providers document that identical calls need not return identical text | | Scoring | A second pass against a rubric file, each score quoting a sentence | an unquoted score is a number somebody will put in a slide | | Cost | An estimate printed and confirmed before any call, and a hard ceiling enforced | questions times engines times samples is larger than it looks | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Store | Immutable answers and pinned run metadata. | Any database, provided answers stay insert only. | | Engines | One module per provider, all keys optional. | Any provider added as a module, none of them required. | | Sampling | Repetition, concurrency, backoff and resume. | Not swappable for one call per question, which is the core error. | | Scoring | Rubric application with verified quotes. | Any scoring model, provided the quote check stays in front of the write. | | Report | Spread, wrong claims and stated limits. | Any format, provided no single overall score appears. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Runs, answers and scores. Answers are insert only. - `BRAND_FILE` · required · Name, aliases, domain, competitors and the question set. - `RUBRIC_PATH` · required · Hashed onto every run. Editing it starts a new series. - `SAMPLES_PER_QUESTION` · required · Per engine. One is a sample and it will read like a measurement. - `MAX_CONCURRENCY_PER_ENGINE` · required · Keeps you inside published rate limits without needing to know them exactly. - `COST_CEILING` · required · In your billing currency. Enforced in code, not in memory. - `OPENAI_API_KEY` · optional, secret · Optional. Unset means that engine is recorded as skipped. - `ANTHROPIC_API_KEY` · optional, secret · Optional, as are the other provider keys, which follow the same pattern. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · BrandGEO product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with TypeScript and a command line interface, local only, SQLite with one immutable row per run, question, engine and sample, One module per provider behind a single call interface, every key optional, Several samples per question per engine, the count recorded on the run, A second pass against a rubric file, each score quoting a sentence, An estimate printed and confirmed before any call, and a hard ceiling enforced. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. - Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes. ===== MILESTONES.md ===== # Delivery milestones · BrandGEO Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A brand file and a store that never forgets Every answer ever received is kept, attributed and immutable. ### Steps 1. Make answers insert only in the schema Enforce it in the database, not in the discipline of the caller. Files: `src/db/schema.sql`, `tests/immutable-answers.test.ts` 2. Pin the rubric and question set to the run Two runs scored differently are not two points on one line. Files: `src/run.ts` 3. Record skipped engines with reasons A narrower report is fine; a silently narrower one is not. Files: `src/engines/registry.ts` ### Done when - [ ] Answer rows cannot be updated or deleted - [ ] Runs pin the rubric checksum and question set - [ ] Missing keys produce a recorded skip with a reason - [ ] A second run leaves the first unchanged ### Watch out - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. ## M2 · Asking several times, and knowing the bill first Each question is sampled repeatedly within a cost ceiling you agreed to. ### Steps 1. Write each answer as it arrives So money already spent is money already stored. Files: `src/sample.ts` 2. Estimate and confirm before the first call Show the multiplication, because that is where the surprise lives. Files: `src/cost.ts` 3. Back off on rate limits, record every failure A failure row is data about the engine, not an absence of data. Files: `src/engines/call.ts`, `tests/resume.test.ts` ### Done when - [ ] A killed run resumes without re-asking - [ ] The estimate is shown and confirmed before calling - [ ] The cost ceiling stops the run and reports progress - [ ] Failures are stored as rows with error text ### Watch out - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. ## M3 · Scores that quote the sentence No dimension score exists without the text that justifies it. ### Steps 1. Verify the quote before saving the score A substring check catches nearly all of it and costs nothing. Files: `src/score/verify.ts`, `tests/quote-check.test.ts` 2. Keep the rubric in a file, hashed per run Rubric drift is the quietest way to ruin a trend line. Files: `rubric.md`, `src/rubric.ts` 3. Count and report discards The discard rate tells you when a scoring model has changed under you. Files: `src/score/index.ts` ### Done when - [ ] Every score has a verified quote - [ ] Fabricated quotes are rejected - [ ] The rubric checksum is stored per run - [ ] Discard counts appear in the report ### Watch out - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. ## M4 · A report that shows the spread The output tells you how much the engines disagreed, including with themselves. ### Steps 1. Report median with low and high Three numbers and an n say what one number cannot. Files: `src/report/aggregate.ts` 2. Quote wrong claims verbatim This is the part a founder actually reads and acts on. Files: `src/report/errors.ts` 3. Refuse to compute one headline number Not a setting, not a flag; there is no code that does it. Files: `src/report/index.ts` ### Done when - [ ] No overall single score exists anywhere - [ ] Every figure shows n, low and high - [ ] Wrong claims are quoted with engine and sample - [ ] Skipped engines appear in the report body ### Watch out - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. ## M5 · A series that stays comparable (production only) Week two means something, or the tool says why it does not. ### Steps 1. Record the model id on every answer It is the field that explains a mysterious jump three weeks later. Files: `src/engines/call.ts` 2. Break the series when the basis changes A visible break is honest; a continuous line is not. Files: `src/history.ts`, `tests/series-break.test.ts` 3. Export raw answers, not just scores The raw text is the evidence and it is the part you cannot regenerate. Files: `src/export.ts` ### Done when - [ ] Rubric, question or model changes break the series - [ ] Exports include raw answers - [ ] Spend is reported per run and per engine - [ ] The report states its own limitations ### Watch out - Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes. ===== OPERATIONS.md ===== # Operations · BrandGEO ## Backup The database, because the raw answers cannot be regenerated: the same question asked next month reaches a different model. Copy it off the machine after every run. ## Restore Restore and run the history command; if a series breaks where you did not expect it, the restore is fine and the basis changed. Re-running to fill a gap creates a new run rather than repairing an old one. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Spend against the estimate, the discard rate from quote verification, and the model ids returned per engine. A model id change is the single most common cause of a trend that looks like news. ## Incident checklist The reportable incident is a number that left the tool without its sample count, because that is the one somebody will quote back at you. The report format exists to make it impossible. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Answers are confirmed insert only - [ ] A killed run is confirmed to resume without re-asking - [ ] Every score is confirmed to carry a verified quote - [ ] No single overall score is confirmed to exist - [ ] A model id change is confirmed to break the series ## Launch constraint Do not market omitted BrandGEO capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, answers and scores. Answers are insert only. DATABASE_PATH=./data/audits.db # Required. Name, aliases, domain, competitors and the question set. BRAND_FILE=./brand.json # Required. Hashed onto every run. Editing it starts a new series. RUBRIC_PATH=./rubric.md # Required. Per engine. One is a sample and it will read like a measurement. SAMPLES_PER_QUESTION=5 # Required. Keeps you inside published rate limits without needing to know them exactly. MAX_CONCURRENCY_PER_ENGINE=2 # Required. In your billing currency. Enforced in code, not in memory. COST_CEILING=10.00 # Optional · secret. Optional. Unset means that engine is recorded as skipped. OPENAI_API_KEY=sk-...redacted # Optional · secret. Optional, as are the other provider keys, which follow the same pattern. ANTHROPIC_API_KEY=sk-ant-...redacted
README.md
# BrandGEO · indie build A local auditor whose hard problems are treating non-deterministic model output as a sample rather than a measurement, keeping every run immutable so a second week means something, and scoring answers against a rubric where each score quotes the sentence behind it. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript and a command line interface, local only | an audit is a script you run, not a service somebody logs into | | Storage | SQLite with one immutable row per run, question, engine and sample | overwriting a previous run destroys the only part that gets more valuable | | Engines | One module per provider behind a single call interface, every key optional | a missing key should narrow the report, not fail the run | | Sampling | Several samples per question per engine, the count recorded on the run | the providers document that identical calls need not return identical text | | Scoring | A second pass against a rubric file, each score quoting a sentence | an unquoted score is a number somebody will put in a slide | | Cost | An estimate printed and confirmed before any call, and a hard ceiling enforced | questions times engines times samples is larger than it looks | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the command line tool. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **API keys for at least two providers** · pennies per run at small sample counts, and it multiplies - Why: One engine is an anecdote; the disagreement between engines is the actual finding, and every key stays optional so a run narrows rather than fails. - Get it: Create a key in each provider's console and put it in .env; leave the rest unset. - Verify: A single cheap question returns an answer from each configured engine - [ ] **A written rubric with per dimension definitions** · free - Why: The rubric is hashed onto every run, so writing it after seeing the first scores means writing whatever the scores already said. - Get it: Write rubric.md defining each of the six dimensions and what a 0, a 5 and a 10 look like. - Verify: Two people scoring the same answer by hand land within two points on each dimension - [ ] **A brand you know well enough to catch errors** · free - Why: The most useful output is the list of things the models got wrong, and you can only produce it about a brand whose facts you already know. - Get it: Your own product, or one whose history you can check quickly. - Verify: You can confirm or refute every factual claim in one sample answer - [ ] **A cost ceiling you are willing to lose** · free - Why: Questions times engines times samples is the arithmetic that surprises people, and the ceiling is enforced in code rather than remembered. - Get it: Pick a figure per run, put it in the environment, and check the printed estimate against it before confirming. - Verify: A run configured to exceed it stops and reports what it completed ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - White label PDF reports. - Multi brand management. - Scheduled monitoring. - Any driving of the consumer chat interfaces. - white-label PDF reports an agency can hand to a client - weekly monitoring that keeps running when nobody is thinking about it - a rubric calibrated across many brands, so scores are comparable - competitor benchmarks per brand - the consumer app surfaces, which no API exactly reproduces If one of those is essential to you, that is the reason to keep paying for BrandGEO, and the README should say so rather than pretend.
BRIEF.md
# Build brief · BrandGEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local brand visibility auditor that reports the spread of what model APIs say, not a single score to replace BrandGEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three things this tool must never pretend to be. It is not a measurement of one number. The providers say so themselves: Anthropic's API reference states that even with a temperature of 0.0 the results will not be fully deterministic, and that models released after Claude Opus 4.6 reject any temperature other than 1.0 outright. One answer to one question is a sample. Ask each question several times per engine and report the range you got back. It is not what users see. A consumer chat product wraps retrieval, tools and product surfaces around the model you are calling, so the report says what the API returned on a given date and makes no claim about the app. And it does not close that gap by driving the chat interfaces. That breaks their terms, breaks every few weeks, and produces a number worth less than an honestly stated gap. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript and a command line interface, local only · an audit is a script you run, not a service somebody logs into - Storage: SQLite with one immutable row per run, question, engine and sample · overwriting a previous run destroys the only part that gets more valuable - Engines: One module per provider behind a single call interface, every key optional · a missing key should narrow the report, not fail the run - Sampling: Several samples per question per engine, the count recorded on the run · the providers document that identical calls need not return identical text - Scoring: A second pass against a rubric file, each score quoting a sentence · an unquoted score is a number somebody will put in a slide - Cost: An estimate printed and confirmed before any call, and a hard ceiling enforced · questions times engines times samples is larger than it looks ### Data model (create this before Phase 1) - brands(id, name, aliases_json, domain, competitors_json) - questions(id, brand_id, text, category, active) · fixed between runs, because an edited question starts a new series - runs(id, brand_id, started_at, finished_at, samples_per_question, rubric_sha256, engines_json, estimated_cost, actual_cost) - answers(id, run_id, question_id, engine, model_id, sample_index, raw_text, latency_ms, error_text, at) · never updated and never deleted - scores(answer_id, dimension, value, quote, scored_by_model, at) · every score carries the sentence that justifies it - aggregates(run_id, engine, dimension, median, low, high, n) · what the report prints, because the spread is the finding - A run pins the rubric checksum and the question set, so changing either visibly starts a new series rather than quietly rewriting the old one. ### Phase 1 · A brand file and a store that never forgets Build: A brand file holding the name, aliases, domain, competitors and up to twenty five questions. A run captures the brand, the question set, the rubric checksum and which engines were available. Answers are insert only, with the model id and any error text kept beside them. Trap: Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. Done when: No code path updates or deletes an answer row, verified by review and by a trigger that rejects it. A run records the rubric checksum and the exact question set it used. An engine with a missing key is recorded as skipped with a reason rather than omitted silently. Re-running the same questions creates a second run and leaves the first byte identical. Do not build yet: No sampling, scoring, reports or history. ### Phase 2 · Asking several times, and knowing the bill first Build: Ask each question a configured number of times per engine. Cap concurrency per engine, retry rate limit and server errors with backoff, and record failures as rows. Print a cost estimate and require confirmation before the first call, then stop hard at the ceiling. Trap: A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. Done when: A run killed halfway resumes and asks only for missing samples, tested against a fake engine. The estimate is printed with the arithmetic shown, and no call is made until it is confirmed. Reaching the cost ceiling stops the run and reports what was completed. Rate limited and failed calls appear as rows with their error text rather than disappearing. Do not build yet: No scoring, reports or history yet. ### Phase 3 · Scores that quote the sentence Build: Score each stored answer against the rubric file on six dimensions. Every score must return a quote, and the quote is checked for presence in that answer before the score is saved. Scores that fail the check are discarded and counted, not repaired. Trap: A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. Done when: Every saved score has a quote confirmed present in its answer, asserted in the suite. A deliberately fabricated quote is rejected by the check, tested with a fixture. The rubric lives in a file whose checksum is stored on the run, and editing it changes the checksum. The count of discarded scores appears in the report rather than in a log. Do not build yet: No reports, history or exports yet. ### Phase 4 · A report that shows the spread Build: Render a Markdown report: per engine and dimension, the median with the low and high across samples and the sample count. List the wrong claims models made about the brand, quoted. Name every engine that was skipped and why. No single overall score anywhere. Trap: Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. Done when: No single overall score is produced by any code path, verified by review. Every figure in the report is accompanied by its sample count and its low and high. Wrong claims are quoted verbatim with the engine and the sample index. Skipped engines are listed with the reason in the report body, not a footnote. Do not build yet: No trend history or exports yet. ### Phase 5 · A series that stays comparable Build: A history command plotting each engine and dimension across runs. Break the line visibly when the rubric checksum, the question set or an engine's model id changes. Export every run as JSON and CSV. Print total spend per run and per engine. Trap: Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes. Done when: A change of rubric checksum, question set or model id breaks the series visibly in the history output, tested by changing each. Exports contain every raw answer, not only the scores. Spend is reported per run and per engine against the estimate. The report states its own limitations in the body, including that API answers are not what the consumer apps show. Do not build yet: Nothing after this. ### Out of scope (and why) - White label PDF reports for clients, which is the part agencies actually pay for and is a document production problem rather than an audit one. - Managing many brands, which turns a script into a product with tenancy, quotas and a support burden. - Scheduled monitoring, which needs a machine that is always on and an alerting story, and is better added once one run has proven useful. - Driving the consumer chat interfaces to see what users see, which breaks their terms and breaks weekly. ### README must contain - That model output is not deterministic, that this is why each question is sampled several times, and where the providers state it - That API answers are not what the consumer apps show, stated plainly and not buried - The per run cost arithmetic by engine, and where the ceiling is enforced - That every score quotes a sentence verified to appear in the answer, and how many were discarded - What breaks a trend series: a rubric change, a question change or a new model id
AGENTS.md
# Agent instructions · BrandGEO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with TypeScript and a command line interface, local only, SQLite with one immutable row per run, question, engine and sample, One module per provider behind a single call interface, every key optional, Several samples per question per engine, the count recorded on the run, A second pass against a rubric file, each score quoting a sentence, An estimate printed and confirmed before any call, and a hard ceiling enforced. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number.
BUILD_PLAN.md
# Build plan · BrandGEO A local auditor whose hard problems are treating non-deterministic model output as a sample rather than a measurement, keeping every run immutable so a second week means something, and scoring answers against a rubric where each score quotes the sentence behind it. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A brand file and a store that never forgets Every answer ever received is kept, attributed and immutable. ### Steps 1. Make answers insert only in the schema Enforce it in the database, not in the discipline of the caller. Files: `src/db/schema.sql`, `tests/immutable-answers.test.ts` 2. Pin the rubric and question set to the run Two runs scored differently are not two points on one line. Files: `src/run.ts` 3. Record skipped engines with reasons A narrower report is fine; a silently narrower one is not. Files: `src/engines/registry.ts` ### Done when - [ ] Answer rows cannot be updated or deleted - [ ] Runs pin the rubric checksum and question set - [ ] Missing keys produce a recorded skip with a reason - [ ] A second run leaves the first unchanged ### Watch out - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. ## Phase 2 · Asking several times, and knowing the bill first Each question is sampled repeatedly within a cost ceiling you agreed to. ### Steps 1. Write each answer as it arrives So money already spent is money already stored. Files: `src/sample.ts` 2. Estimate and confirm before the first call Show the multiplication, because that is where the surprise lives. Files: `src/cost.ts` 3. Back off on rate limits, record every failure A failure row is data about the engine, not an absence of data. Files: `src/engines/call.ts`, `tests/resume.test.ts` ### Done when - [ ] A killed run resumes without re-asking - [ ] The estimate is shown and confirmed before calling - [ ] The cost ceiling stops the run and reports progress - [ ] Failures are stored as rows with error text ### Watch out - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. ## Phase 3 · Scores that quote the sentence No dimension score exists without the text that justifies it. ### Steps 1. Verify the quote before saving the score A substring check catches nearly all of it and costs nothing. Files: `src/score/verify.ts`, `tests/quote-check.test.ts` 2. Keep the rubric in a file, hashed per run Rubric drift is the quietest way to ruin a trend line. Files: `rubric.md`, `src/rubric.ts` 3. Count and report discards The discard rate tells you when a scoring model has changed under you. Files: `src/score/index.ts` ### Done when - [ ] Every score has a verified quote - [ ] Fabricated quotes are rejected - [ ] The rubric checksum is stored per run - [ ] Discard counts appear in the report ### Watch out - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. ## Phase 4 · A report that shows the spread The output tells you how much the engines disagreed, including with themselves. ### Steps 1. Report median with low and high Three numbers and an n say what one number cannot. Files: `src/report/aggregate.ts` 2. Quote wrong claims verbatim This is the part a founder actually reads and acts on. Files: `src/report/errors.ts` 3. Refuse to compute one headline number Not a setting, not a flag; there is no code that does it. Files: `src/report/index.ts` ### Done when - [ ] No overall single score exists anywhere - [ ] Every figure shows n, low and high - [ ] Wrong claims are quoted with engine and sample - [ ] Skipped engines appear in the report body ### Watch out - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. ## Not in this build - White label PDF reports. - Multi brand management. - Scheduled monitoring. - Any driving of the consumer chat interfaces. ## After v1, if you want it - A second brand as a comparison, once one brand has produced three runs worth trusting - Prompt variants per question, to see how much the wording rather than the brand moves the answer - A weekly scheduled run, once you have a machine that is always on and a reason to look at it
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, answers and scores. Answers are insert only. DATABASE_PATH=./data/audits.db # Required. Name, aliases, domain, competitors and the question set. BRAND_FILE=./brand.json # Required. Hashed onto every run. Editing it starts a new series. RUBRIC_PATH=./rubric.md # Required. Per engine. One is a sample and it will read like a measurement. SAMPLES_PER_QUESTION=5 # Required. Keeps you inside published rate limits without needing to know them exactly. MAX_CONCURRENCY_PER_ENGINE=2 # Required. In your billing currency. Enforced in code, not in memory. COST_CEILING=10.00 # Optional · secret. Optional. Unset means that engine is recorded as skipped. OPENAI_API_KEY=sk-...redacted # Optional · secret. Optional, as are the other provider keys, which follow the same pattern. ANTHROPIC_API_KEY=sk-ant-...redacted
PRODUCT.md
# BrandGEO · product brief ## Problem Running a fixed battery of brand questions through five model APIs and scoring the answers with a second LLM pass is a genuine weekend build, and for one brand it answers the headline question: what does AI say about us. The gaps are the ones every tracker in this category shares. API answers approximate but do not equal the consumer apps, a score with no trend history behind it is a screenshot rather than a signal, and a rubric only becomes comparable after it has scored many brands. ## Product outcome An auditor that samples each question several times per engine, keeps every raw answer forever, scores only with quotes it verified, reports a spread instead of a headline number, and says out loud where it differs from what users actually see. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - API keys for OpenAI, Anthropic, Gemini, xAI (Grok), and DeepSeek - a written scoring rubric with per-dimension definitions - durable per-run storage for trend history - a scheduler for recurring audits - an API budget that scales with prompts x engines ## Explicit non-goals for v1 - White label PDF reports. - Multi brand management. - Scheduled monitoring. - Any driving of the consumer chat interfaces. - white-label PDF reports an agency can hand to a client - weekly monitoring that keeps running when nobody is thinking about it - a rubric calibrated across many brands, so scores are comparable - competitor benchmarks per brand - the consumer app surfaces, which no API exactly reproduces ## Success criteria - Answers are confirmed insert only - A killed run is confirmed to resume without re-asking - Every score is confirmed to carry a verified quote - No single overall score is confirmed to exist - A model id change is confirmed to break the series
BRIEF.md
# Build brief · BrandGEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local brand visibility auditor that reports the spread of what model APIs say, not a single score to replace BrandGEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three things this tool must never pretend to be. It is not a measurement of one number. The providers say so themselves: Anthropic's API reference states that even with a temperature of 0.0 the results will not be fully deterministic, and that models released after Claude Opus 4.6 reject any temperature other than 1.0 outright. One answer to one question is a sample. Ask each question several times per engine and report the range you got back. It is not what users see. A consumer chat product wraps retrieval, tools and product surfaces around the model you are calling, so the report says what the API returned on a given date and makes no claim about the app. And it does not close that gap by driving the chat interfaces. That breaks their terms, breaks every few weeks, and produces a number worth less than an honestly stated gap. ### Stack (fixed, do not substitute) - Runtime: Node 22 with TypeScript and a command line interface, local only · an audit is a script you run, not a service somebody logs into - Storage: SQLite with one immutable row per run, question, engine and sample · overwriting a previous run destroys the only part that gets more valuable - Engines: One module per provider behind a single call interface, every key optional · a missing key should narrow the report, not fail the run - Sampling: Several samples per question per engine, the count recorded on the run · the providers document that identical calls need not return identical text - Scoring: A second pass against a rubric file, each score quoting a sentence · an unquoted score is a number somebody will put in a slide - Cost: An estimate printed and confirmed before any call, and a hard ceiling enforced · questions times engines times samples is larger than it looks ### Data model (create this before Phase 1) - brands(id, name, aliases_json, domain, competitors_json) - questions(id, brand_id, text, category, active) · fixed between runs, because an edited question starts a new series - runs(id, brand_id, started_at, finished_at, samples_per_question, rubric_sha256, engines_json, estimated_cost, actual_cost) - answers(id, run_id, question_id, engine, model_id, sample_index, raw_text, latency_ms, error_text, at) · never updated and never deleted - scores(answer_id, dimension, value, quote, scored_by_model, at) · every score carries the sentence that justifies it - aggregates(run_id, engine, dimension, median, low, high, n) · what the report prints, because the spread is the finding - A run pins the rubric checksum and the question set, so changing either visibly starts a new series rather than quietly rewriting the old one. ### Phase 1 · A brand file and a store that never forgets Build: A brand file holding the name, aliases, domain, competitors and up to twenty five questions. A run captures the brand, the question set, the rubric checksum and which engines were available. Answers are insert only, with the model id and any error text kept beside them. Trap: Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. Done when: No code path updates or deletes an answer row, verified by review and by a trigger that rejects it. A run records the rubric checksum and the exact question set it used. An engine with a missing key is recorded as skipped with a reason rather than omitted silently. Re-running the same questions creates a second run and leaves the first byte identical. Do not build yet: No sampling, scoring, reports or history. ### Phase 2 · Asking several times, and knowing the bill first Build: Ask each question a configured number of times per engine. Cap concurrency per engine, retry rate limit and server errors with backoff, and record failures as rows. Print a cost estimate and require confirmation before the first call, then stop hard at the ceiling. Trap: A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. Done when: A run killed halfway resumes and asks only for missing samples, tested against a fake engine. The estimate is printed with the arithmetic shown, and no call is made until it is confirmed. Reaching the cost ceiling stops the run and reports what was completed. Rate limited and failed calls appear as rows with their error text rather than disappearing. Do not build yet: No scoring, reports or history yet. ### Phase 3 · Scores that quote the sentence Build: Score each stored answer against the rubric file on six dimensions. Every score must return a quote, and the quote is checked for presence in that answer before the score is saved. Scores that fail the check are discarded and counted, not repaired. Trap: A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. Done when: Every saved score has a quote confirmed present in its answer, asserted in the suite. A deliberately fabricated quote is rejected by the check, tested with a fixture. The rubric lives in a file whose checksum is stored on the run, and editing it changes the checksum. The count of discarded scores appears in the report rather than in a log. Do not build yet: No reports, history or exports yet. ### Phase 4 · A report that shows the spread Build: Render a Markdown report: per engine and dimension, the median with the low and high across samples and the sample count. List the wrong claims models made about the brand, quoted. Name every engine that was skipped and why. No single overall score anywhere. Trap: Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. Done when: No single overall score is produced by any code path, verified by review. Every figure in the report is accompanied by its sample count and its low and high. Wrong claims are quoted verbatim with the engine and the sample index. Skipped engines are listed with the reason in the report body, not a footnote. Do not build yet: No trend history or exports yet. ### Phase 5 · A series that stays comparable Build: A history command plotting each engine and dimension across runs. Break the line visibly when the rubric checksum, the question set or an engine's model id changes. Export every run as JSON and CSV. Print total spend per run and per engine. Trap: Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes. Done when: A change of rubric checksum, question set or model id breaks the series visibly in the history output, tested by changing each. Exports contain every raw answer, not only the scores. Spend is reported per run and per engine against the estimate. The report states its own limitations in the body, including that API answers are not what the consumer apps show. Do not build yet: Nothing after this. ### Out of scope (and why) - White label PDF reports for clients, which is the part agencies actually pay for and is a document production problem rather than an audit one. - Managing many brands, which turns a script into a product with tenancy, quotas and a support burden. - Scheduled monitoring, which needs a machine that is always on and an alerting story, and is better added once one run has proven useful. - Driving the consumer chat interfaces to see what users see, which breaks their terms and breaks weekly. ### README must contain - That model output is not deterministic, that this is why each question is sampled several times, and where the providers state it - That API answers are not what the consumer apps show, stated plainly and not buried - The per run cost arithmetic by engine, and where the ceiling is enforced - That every score quotes a sentence verified to appear in the answer, and how many were discarded - What breaks a trend series: a rubric change, a question change or a new model id
ARCHITECTURE.md
# Architecture · BrandGEO ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with TypeScript and a command line interface, local only | an audit is a script you run, not a service somebody logs into | | Storage | SQLite with one immutable row per run, question, engine and sample | overwriting a previous run destroys the only part that gets more valuable | | Engines | One module per provider behind a single call interface, every key optional | a missing key should narrow the report, not fail the run | | Sampling | Several samples per question per engine, the count recorded on the run | the providers document that identical calls need not return identical text | | Scoring | A second pass against a rubric file, each score quoting a sentence | an unquoted score is a number somebody will put in a slide | | Cost | An estimate printed and confirmed before any call, and a hard ceiling enforced | questions times engines times samples is larger than it looks | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Store | Immutable answers and pinned run metadata. | Any database, provided answers stay insert only. | | Engines | One module per provider, all keys optional. | Any provider added as a module, none of them required. | | Sampling | Repetition, concurrency, backoff and resume. | Not swappable for one call per question, which is the core error. | | Scoring | Rubric application with verified quotes. | Any scoring model, provided the quote check stays in front of the write. | | Report | Spread, wrong claims and stated limits. | Any format, provided no single overall score appears. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Runs, answers and scores. Answers are insert only. - `BRAND_FILE` · required · Name, aliases, domain, competitors and the question set. - `RUBRIC_PATH` · required · Hashed onto every run. Editing it starts a new series. - `SAMPLES_PER_QUESTION` · required · Per engine. One is a sample and it will read like a measurement. - `MAX_CONCURRENCY_PER_ENGINE` · required · Keeps you inside published rate limits without needing to know them exactly. - `COST_CEILING` · required · In your billing currency. Enforced in code, not in memory. - `OPENAI_API_KEY` · optional, secret · Optional. Unset means that engine is recorded as skipped. - `ANTHROPIC_API_KEY` · optional, secret · Optional, as are the other provider keys, which follow the same pattern. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · BrandGEO product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with TypeScript and a command line interface, local only, SQLite with one immutable row per run, question, engine and sample, One module per provider behind a single call interface, every key optional, Several samples per question per engine, the count recorded on the run, A second pass against a rubric file, each score quoting a sentence, An estimate printed and confirmed before any call, and a hard ceiling enforced. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. - Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes.
MILESTONES.md
# Delivery milestones · BrandGEO Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A brand file and a store that never forgets Every answer ever received is kept, attributed and immutable. ### Steps 1. Make answers insert only in the schema Enforce it in the database, not in the discipline of the caller. Files: `src/db/schema.sql`, `tests/immutable-answers.test.ts` 2. Pin the rubric and question set to the run Two runs scored differently are not two points on one line. Files: `src/run.ts` 3. Record skipped engines with reasons A narrower report is fine; a silently narrower one is not. Files: `src/engines/registry.ts` ### Done when - [ ] Answer rows cannot be updated or deleted - [ ] Runs pin the rubric checksum and question set - [ ] Missing keys produce a recorded skip with a reason - [ ] A second run leaves the first unchanged ### Watch out - Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it. ## M2 · Asking several times, and knowing the bill first Each question is sampled repeatedly within a cost ceiling you agreed to. ### Steps 1. Write each answer as it arrives So money already spent is money already stored. Files: `src/sample.ts` 2. Estimate and confirm before the first call Show the multiplication, because that is where the surprise lives. Files: `src/cost.ts` 3. Back off on rate limits, record every failure A failure row is data about the engine, not an absence of data. Files: `src/engines/call.ts`, `tests/resume.test.ts` ### Done when - [ ] A killed run resumes without re-asking - [ ] The estimate is shown and confirmed before calling - [ ] The cost ceiling stops the run and reports progress - [ ] Failures are stored as rows with error text ### Watch out - A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing. ## M3 · Scores that quote the sentence No dimension score exists without the text that justifies it. ### Steps 1. Verify the quote before saving the score A substring check catches nearly all of it and costs nothing. Files: `src/score/verify.ts`, `tests/quote-check.test.ts` 2. Keep the rubric in a file, hashed per run Rubric drift is the quietest way to ruin a trend line. Files: `rubric.md`, `src/rubric.ts` 3. Count and report discards The discard rate tells you when a scoring model has changed under you. Files: `src/score/index.ts` ### Done when - [ ] Every score has a verified quote - [ ] Fabricated quotes are rejected - [ ] The rubric checksum is stored per run - [ ] Discard counts appear in the report ### Watch out - A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped. ## M4 · A report that shows the spread The output tells you how much the engines disagreed, including with themselves. ### Steps 1. Report median with low and high Three numbers and an n say what one number cannot. Files: `src/report/aggregate.ts` 2. Quote wrong claims verbatim This is the part a founder actually reads and acts on. Files: `src/report/errors.ts` 3. Refuse to compute one headline number Not a setting, not a flag; there is no code that does it. Files: `src/report/index.ts` ### Done when - [ ] No overall single score exists anywhere - [ ] Every figure shows n, low and high - [ ] Wrong claims are quoted with engine and sample - [ ] Skipped engines appear in the report body ### Watch out - Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number. ## M5 · A series that stays comparable (production only) Week two means something, or the tool says why it does not. ### Steps 1. Record the model id on every answer It is the field that explains a mysterious jump three weeks later. Files: `src/engines/call.ts` 2. Break the series when the basis changes A visible break is honest; a continuous line is not. Files: `src/history.ts`, `tests/series-break.test.ts` 3. Export raw answers, not just scores The raw text is the evidence and it is the part you cannot regenerate. Files: `src/export.ts` ### Done when - [ ] Rubric, question or model changes break the series - [ ] Exports include raw answers - [ ] Spend is reported per run and per engine - [ ] The report states its own limitations ### Watch out - Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes.
OPERATIONS.md
# Operations · BrandGEO ## Backup The database, because the raw answers cannot be regenerated: the same question asked next month reaches a different model. Copy it off the machine after every run. ## Restore Restore and run the history command; if a series breaks where you did not expect it, the restore is fine and the basis changed. Re-running to fill a gap creates a new run rather than repairing an old one. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Spend against the estimate, the discard rate from quote verification, and the model ids returned per engine. A model id change is the single most common cause of a trend that looks like news. ## Incident checklist The reportable incident is a number that left the tool without its sample count, because that is the one somebody will quote back at you. The report format exists to make it impossible. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Answers are confirmed insert only - [ ] A killed run is confirmed to resume without re-asking - [ ] Every score is confirmed to carry a verified quote - [ ] No single overall score is confirmed to exist - [ ] A model id change is confirmed to break the series ## Launch constraint Do not market omitted BrandGEO capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, answers and scores. Answers are insert only. DATABASE_PATH=./data/audits.db # Required. Name, aliases, domain, competitors and the question set. BRAND_FILE=./brand.json # Required. Hashed onto every run. Editing it starts a new series. RUBRIC_PATH=./rubric.md # Required. Per engine. One is a sample and it will read like a measurement. SAMPLES_PER_QUESTION=5 # Required. Keeps you inside published rate limits without needing to know them exactly. MAX_CONCURRENCY_PER_ENGINE=2 # Required. In your billing currency. Enforced in code, not in memory. COST_CEILING=10.00 # Optional · secret. Optional. Unset means that engine is recorded as skipped. OPENAI_API_KEY=sk-...redacted # Optional · secret. Optional, as are the other provider keys, which follow the same pattern. ANTHROPIC_API_KEY=sk-ant-...redacted
$ choose a build depth, inspect the files, then open the complete pack in your agent
Agencies are the tell: they pay for a report with someone else's methodology behind it that they can white-label and bill for, plus monitoring and trend history someone else keeps alive. A founder auditing one brand once is exactly who the free audit and a DIY script are for.
xwhite-label PDF reports an agency can hand to a client
xweekly monitoring that keeps running when nobody is thinking about it
xa rubric calibrated across many brands, so scores are comparable
xcompetitor benchmarks per brand
xthe consumer app surfaces, which no API exactly reproduces
share on X ↗"I just replaced BrandGEO ($79/mo) with one prompt"
Don't feel like building it? These folks already made it free.
ElmoA self-hosted AI visibility dashboard that runs your prompts across the major engines and records mentions and citations; the white-label reporting is the part you keep paying for.open source↗no votes, no pay-to-list · just what's real
BrandGEO pricing
starter$79/mo · monthly · $948/yr
free tierA free audit across all five engines without a credit card, plus a 7-day trial on paid plans.
verified 2026-08-10 · source ↗
Is BrandGEO free?
A free audit across all five engines without a credit card, plus a 7-day trial on paid plans. Paid is Starter at $79/mo (checked 2026-08-10).
Vibecode BrandGEO
Kinda. The core of BrandGEO is buildable in a weekend with the prompt on this page, but there are real gaps: white-label PDF reports an agency can hand to a client, weekly monitoring that keeps running when nobody is thinking about it. Read the honest list above before committing.
How much does BrandGEO cost?
BrandGEO costs about $79/month (Starter, checked 2026-08-10), which is $948 per year.
What do I lose by replacing BrandGEO?
Honestly: white-label PDF reports an agency can hand to a client; weekly monitoring that keeps running when nobody is thinking about it; a rubric calibrated across many brands, so scores are comparable; competitor benchmarks per brand; the consumer app surfaces, which no API exactly reproduces. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to BrandGEO?
Yes: Elmo (A self-hosted AI visibility dashboard that runs your prompts across the major engines and records mentions and citations; the white-label reporting is the part you keep paying for.) The prompt is for when you want it exactly your way.