Build LLM Pulse

KINDAreplaces $56.52/mosaves $678.24/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A measurement tool for brand presence in generated answers, where the central engineering problem is that the thing being measured is stochastic, so everything is a sampled distribution with a stated sample size rather than a position.

estimated effort multi-daythe files for this build are in the project pack

SamplingEvery prompt run repeatedly, results stored per runExtractionMentions and cited sources identified per response with the quote keptStatisticsProportions with intervals, sample size shown beside every figureCostPer-run cost tracked, since sampling multiplies itRuntimeTypeScript with PostgreSQL

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the sampler and the reporting.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
installfree locally
Why Runs, mentions and aggregates with enough volume that sampling adds up.
Get it Docker or your package manager. open ↗
Verify psql connects and creates a table
accountper token, multiplied by your sample size
Why Sampling means many calls per prompt, and the cost is the constraint that determines how many prompts you can track at all.
Get it An account with per-token billing, and calculate the cost of your intended sample size before starting.
Verify A sampling round of the intended size completes and its cost matches the projection
have readyfree
Why Measurement quality depends entirely on whether the prompts resemble what people actually type, and prompts written to include your brand name measure nothing.
Get it Write prompts describing the problem rather than naming your product, ideally taken from real questions your customers ask.
Verify Twenty prompts exist and none contains your brand name
decidefree to decide
Why Below a certain sample the figures are noise, and above a certain cost the measurement is not worth it, so both belong decided before the first round.
Get it Work out the sample size that gives an interval you would act on, and the cost of that across your prompts.
Verify Both numbers are written down and the cost has been calculated

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- prompts(id, text, category, added_at, retired_at)
- runs(id, prompt_id, model, model_version, at, response_text, tokens_in, tokens_out, cost_minor)
- mentions(run_id, entity_id, quote, char_start, char_end, sentiment, position_in_response)
- citations(run_id, url, domain, quote_context) · where the answer said it got something
- entities(id, name, aliases_json, kind) · kind is own_brand or competitor
- aggregates(prompt_id, entity_id, model, window_start, sample_size, mention_rate, interval_low, interval_high)
- There is deliberately no rank or position column, because a generated answer has no ranking and a stored number would be presented as one.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_URL`secret	required	`postgres://localhost:5432/pulse`	Sampled responses and extracted mentions.
`RUNS_PER_PROMPT`	required	`30`	The sample size. Below the minimum no figure is reported at all.
`MIN_SAMPLE_FOR_REPORT`	required	`20`	Below this a prompt produces no number rather than a provisional one.
`MONTHLY_BUDGET_MINOR`	required	`10000`	Sampling multiplies cost. The limit refuses rather than warns.
`REPORT_RANK`	required	`never`	Present as a constant. A generated answer has no ranking.
`MODEL_API_KEY`secret	required	`a provider key`	Billed per token across every run of every prompt.

The build, in order

1
Sampling rather than asking once
Every prompt is run enough times to say something.
1. The variation between runs is the finding, not noise to average away.
  Files src/sample.ts
2. A model update changes everything and must be visible in the series.
  Files src/runs.ts
3. A provisional figure is read as a figure.
  Files src/aggregate.ts
done when · tick each as it passes
watch out
- Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one.
next phase
2
Finding the mention, with the quote
Every recorded mention can be checked against the response.
1. So a claim about a mention can be checked.
  Files src/mentions.ts
2. Models get names slightly wrong constantly.
  Files src/entities.ts
3. Being dismissed is worse than being absent.
  Files src/sentiment.ts
done when · tick each as it passes
watch out
- Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal.
next phase
3
Numbers with their uncertainty attached
Every reported proportion shows its sample size and interval.
1. A proportion without one is a claim you cannot support.
  Files src/stats.tssrc/stats.test.ts
2. Otherwise only the point estimate is read.
  Files src/report.ts
3. It is the specific error this whole build exists to avoid.
  Files src/compare.ts
done when · tick each as it passes
watch out
- Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change.
next phase
4
Where the answer said it got that
Cited sources are recorded and their accuracy is checkable.
1. The surrounding sentence is what the citation was supporting.
  Files src/citations.ts
2. Generated citations are sometimes to pages that do not exist.
  Files src/verify.ts
3. A recurring source is the actionable finding.
  Files src/citations.ts
done when · tick each as it passes
watch out
- Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them.
5
Cost, comparison and what this cannot tell youproduct builder
The report is affordable, comparative and honest about its limits.
1. Sampling multiplies it, which is the price of an honest number.
  Files src/cost.ts
2. Different sample sizes make a comparison meaningless.
  Files src/compare.ts
3. Mention rate is not traffic and not influence.
  Files src/report.ts
done when · tick each as it passes
watch out
- Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read.

what this build does not replace

Any rank or position figure.
Attribution of traffic or revenue to a mention.
Advice on how to be mentioned more.
Coverage of every assistant.
managed execution across the full model set
long-term historical comparisons and evidence
reputation, source, traffic, and competitor workflows
team permissions, exports, alerts, and integrations
production monitoring and support

after v1, if you want it

Alerting on a mention rate change large enough to clear the interval, which is the only actionable signal here
Prompt clustering so similar questions are measured together with a larger effective sample
Recording the full response set for a period, which makes a qualitative read possible

Need the files? The project pack on the verdict page hands your agent the whole brief · more seo & marketing.