Build BrandGEO

KINDAreplaces $79/mosaves $948/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A local auditor whose hard problems are treating non-deterministic model output as a sample rather than a measurement, keeping every run immutable so a second week means something, and scoring answers against a rubric where each score quotes the sentence behind it.

estimated effort weekendthe files for this build are in the project pack

RuntimeNode 22 with TypeScript and a command line interface, local onlyStorageSQLite with one immutable row per run, question, engine and sampleEnginesOne module per provider behind a single call interface, every key optionalSamplingSeveral samples per question per engine, the count recorded on the runScoringA second pass against a rubric file, each score quoting a sentenceCostAn estimate printed and confirmed before any call, and a hard ceiling enforced

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the command line tool.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
API keypennies per run at small sample counts, and it multiplies
Why One engine is an anecdote; the disagreement between engines is the actual finding, and every key stays optional so a run narrows rather than fails.
Get it Create a key in each provider's console and put it in .env; leave the rest unset.
Verify A single cheap question returns an answer from each configured engine
decidefree
Why The rubric is hashed onto every run, so writing it after seeing the first scores means writing whatever the scores already said.
Get it Write rubric.md defining each of the six dimensions and what a 0, a 5 and a 10 look like.
Verify Two people scoring the same answer by hand land within two points on each dimension
have readyfree
Why The most useful output is the list of things the models got wrong, and you can only produce it about a brand whose facts you already know.
Get it Your own product, or one whose history you can check quickly.
Verify You can confirm or refute every factual claim in one sample answer
decidefree
Why Questions times engines times samples is the arithmetic that surprises people, and the ceiling is enforced in code rather than remembered.
Get it Pick a figure per run, put it in the environment, and check the printed estimate against it before confirming.
Verify A run configured to exceed it stops and reports what it completed

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- brands(id, name, aliases_json, domain, competitors_json)
- questions(id, brand_id, text, category, active) · fixed between runs, because an edited question starts a new series
- runs(id, brand_id, started_at, finished_at, samples_per_question, rubric_sha256, engines_json, estimated_cost, actual_cost)
- answers(id, run_id, question_id, engine, model_id, sample_index, raw_text, latency_ms, error_text, at) · never updated and never deleted
- scores(answer_id, dimension, value, quote, scored_by_model, at) · every score carries the sentence that justifies it
- aggregates(run_id, engine, dimension, median, low, high, n) · what the report prints, because the spread is the finding
- A run pins the rubric checksum and the question set, so changing either visibly starts a new series rather than quietly rewriting the old one.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_PATH`	required	`./data/audits.db`	Runs, answers and scores. Answers are insert only.
`BRAND_FILE`	required	`./brand.json`	Name, aliases, domain, competitors and the question set.
`RUBRIC_PATH`	required	`./rubric.md`	Hashed onto every run. Editing it starts a new series.
`SAMPLES_PER_QUESTION`	required	`5`	Per engine. One is a sample and it will read like a measurement.
`MAX_CONCURRENCY_PER_ENGINE`	required	`2`	Keeps you inside published rate limits without needing to know them exactly.
`COST_CEILING`	required	`10.00`	In your billing currency. Enforced in code, not in memory.
`OPENAI_API_KEY`secret	optional	`sk-...redacted`	Optional. Unset means that engine is recorded as skipped.
`ANTHROPIC_API_KEY`secret	optional	`sk-ant-...redacted`	Optional, as are the other provider keys, which follow the same pattern.

The build, in order

1
A brand file and a store that never forgets
Every answer ever received is kept, attributed and immutable.
1. Enforce it in the database, not in the discipline of the caller.
  Files src/db/schema.sqltests/immutable-answers.test.ts
2. Two runs scored differently are not two points on one line.
  Files src/run.ts
3. A narrower report is fine; a silently narrower one is not.
  Files src/engines/registry.ts
done when · tick each as it passes
watch out
- Writing the newest scores into a column on the brand row makes the first run look tidy and makes the second run destroy the first, which is the only thing that would have made the tool worth keeping. Insert every answer as its own row and never update one, so history exists whether or not you planned for it.
next phase
2
Asking several times, and knowing the bill first
Each question is sampled repeatedly within a cost ceiling you agreed to.
1. So money already spent is money already stored.
  Files src/sample.ts
2. Show the multiplication, because that is where the surprise lives.
  Files src/cost.ts
3. A failure row is data about the engine, not an absence of data.
  Files src/engines/call.tstests/resume.test.ts
done when · tick each as it passes
watch out
- A run that dies at the fourth engine after paying for three, with nothing written until the end, costs real money and produces nothing, and the natural fix of restarting re-asks everything you already paid for. Write each answer as it arrives and make a resumed run ask only for the samples it is missing.
next phase
3
Scores that quote the sentence
No dimension score exists without the text that justifies it.
1. A substring check catches nearly all of it and costs nothing.
  Files src/score/verify.tstests/quote-check.test.ts
2. Rubric drift is the quietest way to ruin a trend line.
  Files rubric.mdsrc/rubric.ts
3. The discard rate tells you when a scoring model has changed under you.
  Files src/score/index.ts
done when · tick each as it passes
watch out
- A scoring pass asked for a justification will happily produce a fluent quote that does not appear in the answer it is scoring, and once saved it is indistinguishable from a real one. Verify the quote is a substring of the answer before writing the score, and report how many were dropped.
next phase
4
A report that shows the spread
The output tells you how much the engines disagreed, including with themselves.
1. Three numbers and an n say what one number cannot.
  Files src/report/aggregate.ts
2. This is the part a founder actually reads and acts on.
  Files src/report/errors.ts
3. Not a setting, not a flag; there is no code that does it.
  Files src/report/index.ts
done when · tick each as it passes
watch out
- Averaging across engines into one headline figure is exactly what a client will screenshot, and it hides both the disagreement between engines and the variation within one engine, which are the only two findings the tool actually produced. Print the spread per engine, and refuse to compute a single overall number.
5
A series that stays comparableproduct builder
Week two means something, or the tool says why it does not.
1. It is the field that explains a mysterious jump three weeks later.
  Files src/engines/call.ts
2. A visible break is honest; a continuous line is not.
  Files src/history.tstests/series-break.test.ts
3. The raw text is the evidence and it is the part you cannot regenerate.
  Files src/export.ts
done when · tick each as it passes
watch out
- Comparing this week's run to last week's when the provider quietly moved you to a new model version produces a trend line that measures the provider's release schedule rather than your brand. Record the model id returned on every answer, and break the series in the chart whenever it changes.

what this build does not replace

White label PDF reports.
Multi brand management.
Scheduled monitoring.
Any driving of the consumer chat interfaces.
white-label PDF reports an agency can hand to a client
weekly monitoring that keeps running when nobody is thinking about it
a rubric calibrated across many brands, so scores are comparable
competitor benchmarks per brand
the consumer app surfaces, which no API exactly reproduces

after v1, if you want it

A second brand as a comparison, once one brand has produced three runs worth trusting
Prompt variants per question, to see how much the wording rather than the brand moves the answer
A weekly scheduled run, once you have a machine that is always on and a reason to look at it

Need the files? The project pack on the verdict page hands your agent the whole brief · more seo & marketing.