Vibecode LLM Pulse
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
price $56.52/moyou'd save $678.24/yrbuild time multi-daycategory 📈 seo & marketingreplaced by 0 people
The core loop is a realistic weekend build: run a fixed prompt set against a model API, detect brand and competitor mentions, collect citations, and chart the results. The gap appears when you need dependable runs across many models, long-term evidence, team access, exports, alerts, and the broader visibility and reputation workflow.
the project pack5 files · written for this build, step by step
You are building a lean indie version of LLM Pulse. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # LLM Pulse · indie build A measurement tool for brand presence in generated answers, where the central engineering problem is that the thing being measured is stochastic, so everything is a sampled distribution with a stated sample size rather than a position. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Sampling | Every prompt run repeatedly, results stored per run | one run is an anecdote and the variation is the finding | | Extraction | Mentions and cited sources identified per response with the quote kept | a claim about a mention should be checkable | | Statistics | Proportions with intervals, sample size shown beside every figure | a percentage from ten runs is not a percentage | | Cost | Per-run cost tracked, since sampling multiplies it | honest measurement is expensive and the budget must be visible | | Runtime | TypeScript with PostgreSQL | runs, extractions and aggregates are relational | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the sampler and the reporting. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally - Why: Runs, mentions and aggregates with enough volume that sampling adds up. - Get it: Docker or your package manager. - Verify: psql connects and creates a table - [ ] **API access to at least one assistant** · per token, multiplied by your sample size - Why: Sampling means many calls per prompt, and the cost is the constraint that determines how many prompts you can track at all. - Get it: An account with per-token billing, and calculate the cost of your intended sample size before starting. - Verify: A sampling round of the intended size completes and its cost matches the projection - [ ] **Twenty prompts a real customer might ask** · free - Why: Measurement quality depends entirely on whether the prompts resemble what people actually type, and prompts written to include your brand name measure nothing. - Get it: Write prompts describing the problem rather than naming your product, ideally taken from real questions your customers ask. - Verify: Twenty prompts exist and none contains your brand name - [ ] **Your minimum sample size and budget** · free to decide - Why: Below a certain sample the figures are noise, and above a certain cost the measurement is not worth it, so both belong decided before the first round. - Get it: Work out the sample size that gives an interval you would act on, and the cost of that across your prompts. - Verify: Both numbers are written down and the cost has been calculated ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any rank or position figure. - Attribution of traffic or revenue to a mention. - Advice on how to be mentioned more. - Coverage of every assistant. - managed execution across the full model set - long-term historical comparisons and evidence - reputation, source, traffic, and competitor workflows - team permissions, exports, alerts, and integrations - production monitoring and support If one of those is essential to you, that is the reason to keep paying for LLM Pulse, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · LLM Pulse The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me tracking how your brand appears in generated answers, without pretending it is a ranking to replace LLM Pulse. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This category borrows the language of search ranking for something that does not behave like one, and the borrowed language is what makes the numbers misleading. A generated answer is not a ranked list. Ask the same question twice and you can get different answers, different sources and a different mention of you, from the same model on the same day. There is no position to hold, no index to appear in, and no equivalent of a rank that persists between two identical queries. So the only honest measurement is a distribution. Run each prompt many times, report how often you were mentioned and with what variation, and show the sample size beside every figure. A single run is an anecdote, and a percentage from ten runs has an uncertainty wide enough that a change of ten points means nothing. The build therefore refuses to report a single-run result as a finding, refuses to call anything a rank, and shows the interval rather than the point estimate. That makes the numbers less exciting and considerably more useful. ### Stack (fixed, do not substitute) - Sampling: Every prompt run repeatedly, results stored per run · one run is an anecdote and the variation is the finding - Extraction: Mentions and cited sources identified per response with the quote kept · a claim about a mention should be checkable - Statistics: Proportions with intervals, sample size shown beside every figure · a percentage from ten runs is not a percentage - Cost: Per-run cost tracked, since sampling multiplies it · honest measurement is expensive and the budget must be visible - Runtime: TypeScript with PostgreSQL · runs, extractions and aggregates are relational ### Data model (create this before Phase 1) - prompts(id, text, category, added_at, retired_at) - runs(id, prompt_id, model, model_version, at, response_text, tokens_in, tokens_out, cost_minor) - mentions(run_id, entity_id, quote, char_start, char_end, sentiment, position_in_response) - citations(run_id, url, domain, quote_context) · where the answer said it got something - entities(id, name, aliases_json, kind) · kind is own_brand or competitor - aggregates(prompt_id, entity_id, model, window_start, sample_size, mention_rate, interval_low, interval_high) - There is deliberately no rank or position column, because a generated answer has no ranking and a stored number would be presented as one. ### Phase 1 · Sampling rather than asking once Build: Run each prompt a configured number of times per model, storing every response. Record the model version with each run. Refuse to report on a prompt with fewer runs than the minimum sample. Trap: Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. Done when: Each prompt runs the configured number of times and every response is stored. A prompt with fewer runs than the minimum produces no reported figure. The model version is recorded per run and a version change is visible in the data. Two sampling rounds on an unchanged prompt show the variation rather than hiding it. Do not build yet: No extraction, aggregation or reporting. ### Phase 2 · Finding the mention, with the quote Build: Detect mentions of your brand and competitors including aliases, storing the quote and position. Record where in the response the mention appeared. Handle a mention that is negative or that recommends a competitor instead. Trap: Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. Done when: Every mention stores the quote and its position, and the quote resolves to the response text. Aliases and common misspellings are matched. A mention that dismisses the brand is recorded distinctly from one that recommends it. A response mentioning nobody is recorded as such rather than being skipped. Do not build yet: No aggregation, citations or reporting yet. ### Phase 3 · Numbers with their uncertainty attached Build: Compute mention rates as proportions with a confidence interval from the sample size. Display the interval as prominently as the estimate. Refuse to report a change between two periods whose intervals overlap as a change. Trap: Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. Done when: Every proportion is displayed with its sample size and interval. A period-over-period change whose intervals overlap is reported as not distinguishable rather than as a change. Increasing the sample narrows the interval visibly. The interval calculation is unit tested against known values. Do not build yet: No citations, competitors or cost reporting yet. ### Phase 4 · Where the answer said it got that Build: Extract cited sources per response with the surrounding context. Verify that a cited page exists and, where possible, that it supports the claim. Record a citation that does not resolve as such. Trap: Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. Done when: Cited sources are extracted with context and their resolution is checked. A citation to a page that does not exist is recorded as unresolved rather than counted. Domains are aggregated so a recurring source is visible. A response with no citations is distinguished from one whose citations failed to resolve. Do not build yet: No competitor comparison or cost reporting yet. ### Phase 5 · Cost, comparison and what this cannot tell you Build: Track cost per prompt per round, since sampling multiplies it. Compare against competitors on the same samples. State plainly what the measurement does not establish, including that it does not measure traffic or influence. Trap: Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read. Done when: Cost per sampling round is tracked and projected before running. Competitors are compared on the same prompts and sample sizes rather than on separate ones. The report states what the measurement does not establish, prominently. A budget limit refuses further sampling rather than warning. Do not build yet: Nothing after this. ### Out of scope (and why) - Any rank or position figure, since a generated answer has no ranking and the word implies a stability that does not exist. - Attributing traffic or revenue to a mention, which is not observable from the answer side at all. - Advice on how to be mentioned more, which is speculation dressed as method and changes with every model update. - Coverage of every assistant, which multiplies sampling cost linearly and produces incomparable figures. ### README must contain - That a generated answer has no ranking, so nothing in this build reports a position - That every prompt is sampled repeatedly and a below-minimum sample produces no figure - That every proportion is shown with its sample size and interval - That a period-over-period change with overlapping intervals is reported as indistinguishable - That mention rate is not traffic and not influence, stated where the figures are read ===== AGENTS.md ===== # Agent instructions · LLM Pulse indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Every prompt run repeatedly, results stored per run, Mentions and cited sources identified per response with the quote kept, Proportions with intervals, sample size shown beside every figure, Per-run cost tracked, since sampling multiplies it, TypeScript with PostgreSQL. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. ===== BUILD_PLAN.md ===== # Build plan · LLM Pulse A measurement tool for brand presence in generated answers, where the central engineering problem is that the thing being measured is stochastic, so everything is a sampled distribution with a stated sample size rather than a position. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sampling rather than asking once Every prompt is run enough times to say something. ### Steps 1. Run repeatedly and store every response The variation between runs is the finding, not noise to average away. Files: `src/sample.ts` 2. Record the model version per run A model update changes everything and must be visible in the series. Files: `src/runs.ts` 3. Refuse below the minimum sample A provisional figure is read as a figure. Files: `src/aggregate.ts` ### Done when - [ ] Each prompt runs the configured number of times - [ ] Below-minimum samples produce no figure - [ ] Model versions are recorded per run - [ ] Variation between rounds is visible ### Watch out - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. ## Phase 2 · Finding the mention, with the quote Every recorded mention can be checked against the response. ### Steps 1. Keep the quote and position So a claim about a mention can be checked. Files: `src/mentions.ts` 2. Match aliases and misspellings Models get names slightly wrong constantly. Files: `src/entities.ts` 3. Record sentiment separately from presence Being dismissed is worse than being absent. Files: `src/sentiment.ts` ### Done when - [ ] Mentions store a resolvable quote and position - [ ] Aliases and misspellings match - [ ] Dismissals are distinct from recommendations - [ ] Responses mentioning nobody are recorded ### Watch out - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. ## Phase 3 · Numbers with their uncertainty attached Every reported proportion shows its sample size and interval. ### Steps 1. Compute intervals from the sample A proportion without one is a claim you cannot support. Files: `src/stats.ts`, `src/stats.test.ts` 2. Show the interval as prominently as the estimate Otherwise only the point estimate is read. Files: `src/report.ts` 3. Refuse to call an overlap a change It is the specific error this whole build exists to avoid. Files: `src/compare.ts` ### Done when - [ ] Every proportion shows sample size and interval - [ ] Overlapping changes are reported as indistinguishable - [ ] A larger sample visibly narrows the interval - [ ] Interval calculation matches known values ### Watch out - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. ## Phase 4 · Where the answer said it got that Cited sources are recorded and their accuracy is checkable. ### Steps 1. Extract citations with context The surrounding sentence is what the citation was supporting. Files: `src/citations.ts` 2. Verify the page resolves Generated citations are sometimes to pages that do not exist. Files: `src/verify.ts` 3. Aggregate by domain A recurring source is the actionable finding. Files: `src/citations.ts` ### Done when - [ ] Citations are extracted with context - [ ] Unresolvable citations are recorded as such - [ ] Recurring domains are visible - [ ] No citations and failed citations are distinguished ### Watch out - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. ## Not in this build - Any rank or position figure. - Attribution of traffic or revenue to a mention. - Advice on how to be mentioned more. - Coverage of every assistant. ## After v1, if you want it - Alerting on a mention rate change large enough to clear the interval, which is the only actionable signal here - Prompt clustering so similar questions are measured together with a larger effective sample - Recording the full response set for a period, which makes a qualitative read possible ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Sampled responses and extracted mentions. DATABASE_URL=postgres://localhost:5432/pulse # Required. The sample size. Below the minimum no figure is reported at all. RUNS_PER_PROMPT=30 # Required. Below this a prompt produces no number rather than a provisional one. MIN_SAMPLE_FOR_REPORT=20 # Required. Sampling multiplies cost. The limit refuses rather than warns. MONTHLY_BUDGET_MINOR=10000 # Required. Present as a constant. A generated answer has no ranking. REPORT_RANK=never # Required · secret. Billed per token across every run of every prompt. MODEL_API_KEY=a provider key
You are building a lean indie version of LLM Pulse. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # LLM Pulse · indie build A measurement tool for brand presence in generated answers, where the central engineering problem is that the thing being measured is stochastic, so everything is a sampled distribution with a stated sample size rather than a position. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Sampling | Every prompt run repeatedly, results stored per run | one run is an anecdote and the variation is the finding | | Extraction | Mentions and cited sources identified per response with the quote kept | a claim about a mention should be checkable | | Statistics | Proportions with intervals, sample size shown beside every figure | a percentage from ten runs is not a percentage | | Cost | Per-run cost tracked, since sampling multiplies it | honest measurement is expensive and the budget must be visible | | Runtime | TypeScript with PostgreSQL | runs, extractions and aggregates are relational | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the sampler and the reporting. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally - Why: Runs, mentions and aggregates with enough volume that sampling adds up. - Get it: Docker or your package manager. - Verify: psql connects and creates a table - [ ] **API access to at least one assistant** · per token, multiplied by your sample size - Why: Sampling means many calls per prompt, and the cost is the constraint that determines how many prompts you can track at all. - Get it: An account with per-token billing, and calculate the cost of your intended sample size before starting. - Verify: A sampling round of the intended size completes and its cost matches the projection - [ ] **Twenty prompts a real customer might ask** · free - Why: Measurement quality depends entirely on whether the prompts resemble what people actually type, and prompts written to include your brand name measure nothing. - Get it: Write prompts describing the problem rather than naming your product, ideally taken from real questions your customers ask. - Verify: Twenty prompts exist and none contains your brand name - [ ] **Your minimum sample size and budget** · free to decide - Why: Below a certain sample the figures are noise, and above a certain cost the measurement is not worth it, so both belong decided before the first round. - Get it: Work out the sample size that gives an interval you would act on, and the cost of that across your prompts. - Verify: Both numbers are written down and the cost has been calculated ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any rank or position figure. - Attribution of traffic or revenue to a mention. - Advice on how to be mentioned more. - Coverage of every assistant. - managed execution across the full model set - long-term historical comparisons and evidence - reputation, source, traffic, and competitor workflows - team permissions, exports, alerts, and integrations - production monitoring and support If one of those is essential to you, that is the reason to keep paying for LLM Pulse, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · LLM Pulse The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me tracking how your brand appears in generated answers, without pretending it is a ranking to replace LLM Pulse. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This category borrows the language of search ranking for something that does not behave like one, and the borrowed language is what makes the numbers misleading. A generated answer is not a ranked list. Ask the same question twice and you can get different answers, different sources and a different mention of you, from the same model on the same day. There is no position to hold, no index to appear in, and no equivalent of a rank that persists between two identical queries. So the only honest measurement is a distribution. Run each prompt many times, report how often you were mentioned and with what variation, and show the sample size beside every figure. A single run is an anecdote, and a percentage from ten runs has an uncertainty wide enough that a change of ten points means nothing. The build therefore refuses to report a single-run result as a finding, refuses to call anything a rank, and shows the interval rather than the point estimate. That makes the numbers less exciting and considerably more useful. ### Stack (fixed, do not substitute) - Sampling: Every prompt run repeatedly, results stored per run · one run is an anecdote and the variation is the finding - Extraction: Mentions and cited sources identified per response with the quote kept · a claim about a mention should be checkable - Statistics: Proportions with intervals, sample size shown beside every figure · a percentage from ten runs is not a percentage - Cost: Per-run cost tracked, since sampling multiplies it · honest measurement is expensive and the budget must be visible - Runtime: TypeScript with PostgreSQL · runs, extractions and aggregates are relational ### Data model (create this before Phase 1) - prompts(id, text, category, added_at, retired_at) - runs(id, prompt_id, model, model_version, at, response_text, tokens_in, tokens_out, cost_minor) - mentions(run_id, entity_id, quote, char_start, char_end, sentiment, position_in_response) - citations(run_id, url, domain, quote_context) · where the answer said it got something - entities(id, name, aliases_json, kind) · kind is own_brand or competitor - aggregates(prompt_id, entity_id, model, window_start, sample_size, mention_rate, interval_low, interval_high) - There is deliberately no rank or position column, because a generated answer has no ranking and a stored number would be presented as one. ### Phase 1 · Sampling rather than asking once Build: Run each prompt a configured number of times per model, storing every response. Record the model version with each run. Refuse to report on a prompt with fewer runs than the minimum sample. Trap: Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. Done when: Each prompt runs the configured number of times and every response is stored. A prompt with fewer runs than the minimum produces no reported figure. The model version is recorded per run and a version change is visible in the data. Two sampling rounds on an unchanged prompt show the variation rather than hiding it. Do not build yet: No extraction, aggregation or reporting. ### Phase 2 · Finding the mention, with the quote Build: Detect mentions of your brand and competitors including aliases, storing the quote and position. Record where in the response the mention appeared. Handle a mention that is negative or that recommends a competitor instead. Trap: Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. Done when: Every mention stores the quote and its position, and the quote resolves to the response text. Aliases and common misspellings are matched. A mention that dismisses the brand is recorded distinctly from one that recommends it. A response mentioning nobody is recorded as such rather than being skipped. Do not build yet: No aggregation, citations or reporting yet. ### Phase 3 · Numbers with their uncertainty attached Build: Compute mention rates as proportions with a confidence interval from the sample size. Display the interval as prominently as the estimate. Refuse to report a change between two periods whose intervals overlap as a change. Trap: Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. Done when: Every proportion is displayed with its sample size and interval. A period-over-period change whose intervals overlap is reported as not distinguishable rather than as a change. Increasing the sample narrows the interval visibly. The interval calculation is unit tested against known values. Do not build yet: No citations, competitors or cost reporting yet. ### Phase 4 · Where the answer said it got that Build: Extract cited sources per response with the surrounding context. Verify that a cited page exists and, where possible, that it supports the claim. Record a citation that does not resolve as such. Trap: Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. Done when: Cited sources are extracted with context and their resolution is checked. A citation to a page that does not exist is recorded as unresolved rather than counted. Domains are aggregated so a recurring source is visible. A response with no citations is distinguished from one whose citations failed to resolve. Do not build yet: No competitor comparison or cost reporting yet. ### Phase 5 · Cost, comparison and what this cannot tell you Build: Track cost per prompt per round, since sampling multiplies it. Compare against competitors on the same samples. State plainly what the measurement does not establish, including that it does not measure traffic or influence. Trap: Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read. Done when: Cost per sampling round is tracked and projected before running. Competitors are compared on the same prompts and sample sizes rather than on separate ones. The report states what the measurement does not establish, prominently. A budget limit refuses further sampling rather than warning. Do not build yet: Nothing after this. ### Out of scope (and why) - Any rank or position figure, since a generated answer has no ranking and the word implies a stability that does not exist. - Attributing traffic or revenue to a mention, which is not observable from the answer side at all. - Advice on how to be mentioned more, which is speculation dressed as method and changes with every model update. - Coverage of every assistant, which multiplies sampling cost linearly and produces incomparable figures. ### README must contain - That a generated answer has no ranking, so nothing in this build reports a position - That every prompt is sampled repeatedly and a below-minimum sample produces no figure - That every proportion is shown with its sample size and interval - That a period-over-period change with overlapping intervals is reported as indistinguishable - That mention rate is not traffic and not influence, stated where the figures are read ===== AGENTS.md ===== # Agent instructions · LLM Pulse indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Every prompt run repeatedly, results stored per run, Mentions and cited sources identified per response with the quote kept, Proportions with intervals, sample size shown beside every figure, Per-run cost tracked, since sampling multiplies it, TypeScript with PostgreSQL. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. ===== BUILD_PLAN.md ===== # Build plan · LLM Pulse A measurement tool for brand presence in generated answers, where the central engineering problem is that the thing being measured is stochastic, so everything is a sampled distribution with a stated sample size rather than a position. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sampling rather than asking once Every prompt is run enough times to say something. ### Steps 1. Run repeatedly and store every response The variation between runs is the finding, not noise to average away. Files: `src/sample.ts` 2. Record the model version per run A model update changes everything and must be visible in the series. Files: `src/runs.ts` 3. Refuse below the minimum sample A provisional figure is read as a figure. Files: `src/aggregate.ts` ### Done when - [ ] Each prompt runs the configured number of times - [ ] Below-minimum samples produce no figure - [ ] Model versions are recorded per run - [ ] Variation between rounds is visible ### Watch out - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. ## Phase 2 · Finding the mention, with the quote Every recorded mention can be checked against the response. ### Steps 1. Keep the quote and position So a claim about a mention can be checked. Files: `src/mentions.ts` 2. Match aliases and misspellings Models get names slightly wrong constantly. Files: `src/entities.ts` 3. Record sentiment separately from presence Being dismissed is worse than being absent. Files: `src/sentiment.ts` ### Done when - [ ] Mentions store a resolvable quote and position - [ ] Aliases and misspellings match - [ ] Dismissals are distinct from recommendations - [ ] Responses mentioning nobody are recorded ### Watch out - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. ## Phase 3 · Numbers with their uncertainty attached Every reported proportion shows its sample size and interval. ### Steps 1. Compute intervals from the sample A proportion without one is a claim you cannot support. Files: `src/stats.ts`, `src/stats.test.ts` 2. Show the interval as prominently as the estimate Otherwise only the point estimate is read. Files: `src/report.ts` 3. Refuse to call an overlap a change It is the specific error this whole build exists to avoid. Files: `src/compare.ts` ### Done when - [ ] Every proportion shows sample size and interval - [ ] Overlapping changes are reported as indistinguishable - [ ] A larger sample visibly narrows the interval - [ ] Interval calculation matches known values ### Watch out - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. ## Phase 4 · Where the answer said it got that Cited sources are recorded and their accuracy is checkable. ### Steps 1. Extract citations with context The surrounding sentence is what the citation was supporting. Files: `src/citations.ts` 2. Verify the page resolves Generated citations are sometimes to pages that do not exist. Files: `src/verify.ts` 3. Aggregate by domain A recurring source is the actionable finding. Files: `src/citations.ts` ### Done when - [ ] Citations are extracted with context - [ ] Unresolvable citations are recorded as such - [ ] Recurring domains are visible - [ ] No citations and failed citations are distinguished ### Watch out - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. ## Not in this build - Any rank or position figure. - Attribution of traffic or revenue to a mention. - Advice on how to be mentioned more. - Coverage of every assistant. ## After v1, if you want it - Alerting on a mention rate change large enough to clear the interval, which is the only actionable signal here - Prompt clustering so similar questions are measured together with a larger effective sample - Recording the full response set for a period, which makes a qualitative read possible ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Sampled responses and extracted mentions. DATABASE_URL=postgres://localhost:5432/pulse # Required. The sample size. Below the minimum no figure is reported at all. RUNS_PER_PROMPT=30 # Required. Below this a prompt produces no number rather than a provisional one. MIN_SAMPLE_FOR_REPORT=20 # Required. Sampling multiplies cost. The limit refuses rather than warns. MONTHLY_BUDGET_MINOR=10000 # Required. Present as a constant. A generated answer has no ranking. REPORT_RANK=never # Required · secret. Billed per token across every run of every prompt. MODEL_API_KEY=a provider key
You are building a production product version of LLM Pulse. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # LLM Pulse · product brief ## Problem The core loop is a realistic weekend build: run a fixed prompt set against a model API, detect brand and competitor mentions, collect citations, and chart the results. The gap appears when you need dependable runs across many models, long-term evidence, team access, exports, alerts, and the broader visibility and reputation workflow. ## Product outcome A measurement of how often you appear in generated answers, with the sample size and uncertainty attached, that refuses to describe noise as a trend. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - one compatible model API key - Node.js 22 - SQLite - a scheduled local process - a small API budget ## Explicit non-goals for v1 - Any rank or position figure. - Attribution of traffic or revenue to a mention. - Advice on how to be mentioned more. - Coverage of every assistant. - managed execution across the full model set - long-term historical comparisons and evidence - reputation, source, traffic, and competitor workflows - team permissions, exports, alerts, and integrations - production monitoring and support ## Success criteria - Below-minimum samples are confirmed to produce no figure - Every proportion is confirmed shown with sample size and interval - Overlapping changes are confirmed reported as indistinguishable - Unresolvable citations are confirmed not counted - No rank or position figure is confirmed to exist ===== BRIEF.md ===== # Build brief · LLM Pulse The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me tracking how your brand appears in generated answers, without pretending it is a ranking to replace LLM Pulse. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This category borrows the language of search ranking for something that does not behave like one, and the borrowed language is what makes the numbers misleading. A generated answer is not a ranked list. Ask the same question twice and you can get different answers, different sources and a different mention of you, from the same model on the same day. There is no position to hold, no index to appear in, and no equivalent of a rank that persists between two identical queries. So the only honest measurement is a distribution. Run each prompt many times, report how often you were mentioned and with what variation, and show the sample size beside every figure. A single run is an anecdote, and a percentage from ten runs has an uncertainty wide enough that a change of ten points means nothing. The build therefore refuses to report a single-run result as a finding, refuses to call anything a rank, and shows the interval rather than the point estimate. That makes the numbers less exciting and considerably more useful. ### Stack (fixed, do not substitute) - Sampling: Every prompt run repeatedly, results stored per run · one run is an anecdote and the variation is the finding - Extraction: Mentions and cited sources identified per response with the quote kept · a claim about a mention should be checkable - Statistics: Proportions with intervals, sample size shown beside every figure · a percentage from ten runs is not a percentage - Cost: Per-run cost tracked, since sampling multiplies it · honest measurement is expensive and the budget must be visible - Runtime: TypeScript with PostgreSQL · runs, extractions and aggregates are relational ### Data model (create this before Phase 1) - prompts(id, text, category, added_at, retired_at) - runs(id, prompt_id, model, model_version, at, response_text, tokens_in, tokens_out, cost_minor) - mentions(run_id, entity_id, quote, char_start, char_end, sentiment, position_in_response) - citations(run_id, url, domain, quote_context) · where the answer said it got something - entities(id, name, aliases_json, kind) · kind is own_brand or competitor - aggregates(prompt_id, entity_id, model, window_start, sample_size, mention_rate, interval_low, interval_high) - There is deliberately no rank or position column, because a generated answer has no ranking and a stored number would be presented as one. ### Phase 1 · Sampling rather than asking once Build: Run each prompt a configured number of times per model, storing every response. Record the model version with each run. Refuse to report on a prompt with fewer runs than the minimum sample. Trap: Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. Done when: Each prompt runs the configured number of times and every response is stored. A prompt with fewer runs than the minimum produces no reported figure. The model version is recorded per run and a version change is visible in the data. Two sampling rounds on an unchanged prompt show the variation rather than hiding it. Do not build yet: No extraction, aggregation or reporting. ### Phase 2 · Finding the mention, with the quote Build: Detect mentions of your brand and competitors including aliases, storing the quote and position. Record where in the response the mention appeared. Handle a mention that is negative or that recommends a competitor instead. Trap: Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. Done when: Every mention stores the quote and its position, and the quote resolves to the response text. Aliases and common misspellings are matched. A mention that dismisses the brand is recorded distinctly from one that recommends it. A response mentioning nobody is recorded as such rather than being skipped. Do not build yet: No aggregation, citations or reporting yet. ### Phase 3 · Numbers with their uncertainty attached Build: Compute mention rates as proportions with a confidence interval from the sample size. Display the interval as prominently as the estimate. Refuse to report a change between two periods whose intervals overlap as a change. Trap: Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. Done when: Every proportion is displayed with its sample size and interval. A period-over-period change whose intervals overlap is reported as not distinguishable rather than as a change. Increasing the sample narrows the interval visibly. The interval calculation is unit tested against known values. Do not build yet: No citations, competitors or cost reporting yet. ### Phase 4 · Where the answer said it got that Build: Extract cited sources per response with the surrounding context. Verify that a cited page exists and, where possible, that it supports the claim. Record a citation that does not resolve as such. Trap: Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. Done when: Cited sources are extracted with context and their resolution is checked. A citation to a page that does not exist is recorded as unresolved rather than counted. Domains are aggregated so a recurring source is visible. A response with no citations is distinguished from one whose citations failed to resolve. Do not build yet: No competitor comparison or cost reporting yet. ### Phase 5 · Cost, comparison and what this cannot tell you Build: Track cost per prompt per round, since sampling multiplies it. Compare against competitors on the same samples. State plainly what the measurement does not establish, including that it does not measure traffic or influence. Trap: Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read. Done when: Cost per sampling round is tracked and projected before running. Competitors are compared on the same prompts and sample sizes rather than on separate ones. The report states what the measurement does not establish, prominently. A budget limit refuses further sampling rather than warning. Do not build yet: Nothing after this. ### Out of scope (and why) - Any rank or position figure, since a generated answer has no ranking and the word implies a stability that does not exist. - Attributing traffic or revenue to a mention, which is not observable from the answer side at all. - Advice on how to be mentioned more, which is speculation dressed as method and changes with every model update. - Coverage of every assistant, which multiplies sampling cost linearly and produces incomparable figures. ### README must contain - That a generated answer has no ranking, so nothing in this build reports a position - That every prompt is sampled repeatedly and a below-minimum sample produces no figure - That every proportion is shown with its sample size and interval - That a period-over-period change with overlapping intervals is reported as indistinguishable - That mention rate is not traffic and not influence, stated where the figures are read ===== ARCHITECTURE.md ===== # Architecture · LLM Pulse ## Stack | Part | Choice | Why | | --- | --- | --- | | Sampling | Every prompt run repeatedly, results stored per run | one run is an anecdote and the variation is the finding | | Extraction | Mentions and cited sources identified per response with the quote kept | a claim about a mention should be checkable | | Statistics | Proportions with intervals, sample size shown beside every figure | a percentage from ten runs is not a percentage | | Cost | Per-run cost tracked, since sampling multiplies it | honest measurement is expensive and the budget must be visible | | Runtime | TypeScript with PostgreSQL | runs, extractions and aggregates are relational | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Sampling | Repeated runs with model versions recorded. | Any provider, provided one run is never treated as a measurement. | | Extraction | Quoted mentions with sentiment. | Any extractor, provided quotes are stored and checkable. | | Statistics | Proportions with intervals. | Any method, provided the interval is as prominent as the estimate. | | Citations | Extraction and resolution checking. | Any implementation, provided unresolvable citations are not counted. | | Reporting | Comparison on identical samples with stated limits. | Any presentation, provided no rank is implied. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Sampled responses and extracted mentions. - `RUNS_PER_PROMPT` · required · The sample size. Below the minimum no figure is reported at all. - `MIN_SAMPLE_FOR_REPORT` · required · Below this a prompt produces no number rather than a provisional one. - `MONTHLY_BUDGET_MINOR` · required · Sampling multiplies cost. The limit refuses rather than warns. - `REPORT_RANK` · required · Present as a constant. A generated answer has no ranking. - `MODEL_API_KEY` · required, secret · Billed per token across every run of every prompt. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · LLM Pulse product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Every prompt run repeatedly, results stored per run, Mentions and cited sources identified per response with the quote kept, Proportions with intervals, sample size shown beside every figure, Per-run cost tracked, since sampling multiplies it, TypeScript with PostgreSQL. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. - Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read. ===== MILESTONES.md ===== # Delivery milestones · LLM Pulse Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Sampling rather than asking once Every prompt is run enough times to say something. ### Steps 1. Run repeatedly and store every response The variation between runs is the finding, not noise to average away. Files: `src/sample.ts` 2. Record the model version per run A model update changes everything and must be visible in the series. Files: `src/runs.ts` 3. Refuse below the minimum sample A provisional figure is read as a figure. Files: `src/aggregate.ts` ### Done when - [ ] Each prompt runs the configured number of times - [ ] Below-minimum samples produce no figure - [ ] Model versions are recorded per run - [ ] Variation between rounds is visible ### Watch out - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. ## M2 · Finding the mention, with the quote Every recorded mention can be checked against the response. ### Steps 1. Keep the quote and position So a claim about a mention can be checked. Files: `src/mentions.ts` 2. Match aliases and misspellings Models get names slightly wrong constantly. Files: `src/entities.ts` 3. Record sentiment separately from presence Being dismissed is worse than being absent. Files: `src/sentiment.ts` ### Done when - [ ] Mentions store a resolvable quote and position - [ ] Aliases and misspellings match - [ ] Dismissals are distinct from recommendations - [ ] Responses mentioning nobody are recorded ### Watch out - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. ## M3 · Numbers with their uncertainty attached Every reported proportion shows its sample size and interval. ### Steps 1. Compute intervals from the sample A proportion without one is a claim you cannot support. Files: `src/stats.ts`, `src/stats.test.ts` 2. Show the interval as prominently as the estimate Otherwise only the point estimate is read. Files: `src/report.ts` 3. Refuse to call an overlap a change It is the specific error this whole build exists to avoid. Files: `src/compare.ts` ### Done when - [ ] Every proportion shows sample size and interval - [ ] Overlapping changes are reported as indistinguishable - [ ] A larger sample visibly narrows the interval - [ ] Interval calculation matches known values ### Watch out - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. ## M4 · Where the answer said it got that Cited sources are recorded and their accuracy is checkable. ### Steps 1. Extract citations with context The surrounding sentence is what the citation was supporting. Files: `src/citations.ts` 2. Verify the page resolves Generated citations are sometimes to pages that do not exist. Files: `src/verify.ts` 3. Aggregate by domain A recurring source is the actionable finding. Files: `src/citations.ts` ### Done when - [ ] Citations are extracted with context - [ ] Unresolvable citations are recorded as such - [ ] Recurring domains are visible - [ ] No citations and failed citations are distinguished ### Watch out - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. ## M5 · Cost, comparison and what this cannot tell you (production only) The report is affordable, comparative and honest about its limits. ### Steps 1. Track and project cost Sampling multiplies it, which is the price of an honest number. Files: `src/cost.ts` 2. Compare on identical samples Different sample sizes make a comparison meaningless. Files: `src/compare.ts` 3. State the limits where they are read Mention rate is not traffic and not influence. Files: `src/report.ts` ### Done when - [ ] Cost is tracked and projected before a round - [ ] Competitor comparison uses identical samples - [ ] The report states its limits prominently - [ ] A budget limit refuses rather than warns ### Watch out - Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read. ===== OPERATIONS.md ===== # Operations · LLM Pulse ## Backup Sampled responses are expensive to reproduce, since regenerating them costs the same money again and the model may have changed. Back them up; the aggregates recompute from them. ## Restore Restore and confirm model versions survived alongside the runs, since comparing samples taken under different model versions without knowing it produces a trend that is entirely an artefact. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Cost per round against budget, and model version changes. A version change invalidates comparison across it and is the single most common cause of an apparent trend. ## Incident checklist A reported trend that turns out to be a model update rather than a change in your visibility is the failure this build is organised against. The recorded model version per run is what makes that diagnosable in minutes. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Below-minimum samples are confirmed to produce no figure - [ ] Every proportion is confirmed shown with sample size and interval - [ ] Overlapping changes are confirmed reported as indistinguishable - [ ] Unresolvable citations are confirmed not counted - [ ] No rank or position figure is confirmed to exist ## Launch constraint Do not market omitted LLM Pulse capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Sampled responses and extracted mentions. DATABASE_URL=postgres://localhost:5432/pulse # Required. The sample size. Below the minimum no figure is reported at all. RUNS_PER_PROMPT=30 # Required. Below this a prompt produces no number rather than a provisional one. MIN_SAMPLE_FOR_REPORT=20 # Required. Sampling multiplies cost. The limit refuses rather than warns. MONTHLY_BUDGET_MINOR=10000 # Required. Present as a constant. A generated answer has no ranking. REPORT_RANK=never # Required · secret. Billed per token across every run of every prompt. MODEL_API_KEY=a provider key
README.md
# LLM Pulse · indie build A measurement tool for brand presence in generated answers, where the central engineering problem is that the thing being measured is stochastic, so everything is a sampled distribution with a stated sample size rather than a position. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Sampling | Every prompt run repeatedly, results stored per run | one run is an anecdote and the variation is the finding | | Extraction | Mentions and cited sources identified per response with the quote kept | a claim about a mention should be checkable | | Statistics | Proportions with intervals, sample size shown beside every figure | a percentage from ten runs is not a percentage | | Cost | Per-run cost tracked, since sampling multiplies it | honest measurement is expensive and the budget must be visible | | Runtime | TypeScript with PostgreSQL | runs, extractions and aggregates are relational | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the sampler and the reporting. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally - Why: Runs, mentions and aggregates with enough volume that sampling adds up. - Get it: Docker or your package manager. - Verify: psql connects and creates a table - [ ] **API access to at least one assistant** · per token, multiplied by your sample size - Why: Sampling means many calls per prompt, and the cost is the constraint that determines how many prompts you can track at all. - Get it: An account with per-token billing, and calculate the cost of your intended sample size before starting. - Verify: A sampling round of the intended size completes and its cost matches the projection - [ ] **Twenty prompts a real customer might ask** · free - Why: Measurement quality depends entirely on whether the prompts resemble what people actually type, and prompts written to include your brand name measure nothing. - Get it: Write prompts describing the problem rather than naming your product, ideally taken from real questions your customers ask. - Verify: Twenty prompts exist and none contains your brand name - [ ] **Your minimum sample size and budget** · free to decide - Why: Below a certain sample the figures are noise, and above a certain cost the measurement is not worth it, so both belong decided before the first round. - Get it: Work out the sample size that gives an interval you would act on, and the cost of that across your prompts. - Verify: Both numbers are written down and the cost has been calculated ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any rank or position figure. - Attribution of traffic or revenue to a mention. - Advice on how to be mentioned more. - Coverage of every assistant. - managed execution across the full model set - long-term historical comparisons and evidence - reputation, source, traffic, and competitor workflows - team permissions, exports, alerts, and integrations - production monitoring and support If one of those is essential to you, that is the reason to keep paying for LLM Pulse, and the README should say so rather than pretend.
BRIEF.md
# Build brief · LLM Pulse The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me tracking how your brand appears in generated answers, without pretending it is a ranking to replace LLM Pulse. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This category borrows the language of search ranking for something that does not behave like one, and the borrowed language is what makes the numbers misleading. A generated answer is not a ranked list. Ask the same question twice and you can get different answers, different sources and a different mention of you, from the same model on the same day. There is no position to hold, no index to appear in, and no equivalent of a rank that persists between two identical queries. So the only honest measurement is a distribution. Run each prompt many times, report how often you were mentioned and with what variation, and show the sample size beside every figure. A single run is an anecdote, and a percentage from ten runs has an uncertainty wide enough that a change of ten points means nothing. The build therefore refuses to report a single-run result as a finding, refuses to call anything a rank, and shows the interval rather than the point estimate. That makes the numbers less exciting and considerably more useful. ### Stack (fixed, do not substitute) - Sampling: Every prompt run repeatedly, results stored per run · one run is an anecdote and the variation is the finding - Extraction: Mentions and cited sources identified per response with the quote kept · a claim about a mention should be checkable - Statistics: Proportions with intervals, sample size shown beside every figure · a percentage from ten runs is not a percentage - Cost: Per-run cost tracked, since sampling multiplies it · honest measurement is expensive and the budget must be visible - Runtime: TypeScript with PostgreSQL · runs, extractions and aggregates are relational ### Data model (create this before Phase 1) - prompts(id, text, category, added_at, retired_at) - runs(id, prompt_id, model, model_version, at, response_text, tokens_in, tokens_out, cost_minor) - mentions(run_id, entity_id, quote, char_start, char_end, sentiment, position_in_response) - citations(run_id, url, domain, quote_context) · where the answer said it got something - entities(id, name, aliases_json, kind) · kind is own_brand or competitor - aggregates(prompt_id, entity_id, model, window_start, sample_size, mention_rate, interval_low, interval_high) - There is deliberately no rank or position column, because a generated answer has no ranking and a stored number would be presented as one. ### Phase 1 · Sampling rather than asking once Build: Run each prompt a configured number of times per model, storing every response. Record the model version with each run. Refuse to report on a prompt with fewer runs than the minimum sample. Trap: Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. Done when: Each prompt runs the configured number of times and every response is stored. A prompt with fewer runs than the minimum produces no reported figure. The model version is recorded per run and a version change is visible in the data. Two sampling rounds on an unchanged prompt show the variation rather than hiding it. Do not build yet: No extraction, aggregation or reporting. ### Phase 2 · Finding the mention, with the quote Build: Detect mentions of your brand and competitors including aliases, storing the quote and position. Record where in the response the mention appeared. Handle a mention that is negative or that recommends a competitor instead. Trap: Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. Done when: Every mention stores the quote and its position, and the quote resolves to the response text. Aliases and common misspellings are matched. A mention that dismisses the brand is recorded distinctly from one that recommends it. A response mentioning nobody is recorded as such rather than being skipped. Do not build yet: No aggregation, citations or reporting yet. ### Phase 3 · Numbers with their uncertainty attached Build: Compute mention rates as proportions with a confidence interval from the sample size. Display the interval as prominently as the estimate. Refuse to report a change between two periods whose intervals overlap as a change. Trap: Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. Done when: Every proportion is displayed with its sample size and interval. A period-over-period change whose intervals overlap is reported as not distinguishable rather than as a change. Increasing the sample narrows the interval visibly. The interval calculation is unit tested against known values. Do not build yet: No citations, competitors or cost reporting yet. ### Phase 4 · Where the answer said it got that Build: Extract cited sources per response with the surrounding context. Verify that a cited page exists and, where possible, that it supports the claim. Record a citation that does not resolve as such. Trap: Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. Done when: Cited sources are extracted with context and their resolution is checked. A citation to a page that does not exist is recorded as unresolved rather than counted. Domains are aggregated so a recurring source is visible. A response with no citations is distinguished from one whose citations failed to resolve. Do not build yet: No competitor comparison or cost reporting yet. ### Phase 5 · Cost, comparison and what this cannot tell you Build: Track cost per prompt per round, since sampling multiplies it. Compare against competitors on the same samples. State plainly what the measurement does not establish, including that it does not measure traffic or influence. Trap: Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read. Done when: Cost per sampling round is tracked and projected before running. Competitors are compared on the same prompts and sample sizes rather than on separate ones. The report states what the measurement does not establish, prominently. A budget limit refuses further sampling rather than warning. Do not build yet: Nothing after this. ### Out of scope (and why) - Any rank or position figure, since a generated answer has no ranking and the word implies a stability that does not exist. - Attributing traffic or revenue to a mention, which is not observable from the answer side at all. - Advice on how to be mentioned more, which is speculation dressed as method and changes with every model update. - Coverage of every assistant, which multiplies sampling cost linearly and produces incomparable figures. ### README must contain - That a generated answer has no ranking, so nothing in this build reports a position - That every prompt is sampled repeatedly and a below-minimum sample produces no figure - That every proportion is shown with its sample size and interval - That a period-over-period change with overlapping intervals is reported as indistinguishable - That mention rate is not traffic and not influence, stated where the figures are read
AGENTS.md
# Agent instructions · LLM Pulse indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Every prompt run repeatedly, results stored per run, Mentions and cited sources identified per response with the quote kept, Proportions with intervals, sample size shown beside every figure, Per-run cost tracked, since sampling multiplies it, TypeScript with PostgreSQL. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them.
BUILD_PLAN.md
# Build plan · LLM Pulse A measurement tool for brand presence in generated answers, where the central engineering problem is that the thing being measured is stochastic, so everything is a sampled distribution with a stated sample size rather than a position. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sampling rather than asking once Every prompt is run enough times to say something. ### Steps 1. Run repeatedly and store every response The variation between runs is the finding, not noise to average away. Files: `src/sample.ts` 2. Record the model version per run A model update changes everything and must be visible in the series. Files: `src/runs.ts` 3. Refuse below the minimum sample A provisional figure is read as a figure. Files: `src/aggregate.ts` ### Done when - [ ] Each prompt runs the configured number of times - [ ] Below-minimum samples produce no figure - [ ] Model versions are recorded per run - [ ] Variation between rounds is visible ### Watch out - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. ## Phase 2 · Finding the mention, with the quote Every recorded mention can be checked against the response. ### Steps 1. Keep the quote and position So a claim about a mention can be checked. Files: `src/mentions.ts` 2. Match aliases and misspellings Models get names slightly wrong constantly. Files: `src/entities.ts` 3. Record sentiment separately from presence Being dismissed is worse than being absent. Files: `src/sentiment.ts` ### Done when - [ ] Mentions store a resolvable quote and position - [ ] Aliases and misspellings match - [ ] Dismissals are distinct from recommendations - [ ] Responses mentioning nobody are recorded ### Watch out - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. ## Phase 3 · Numbers with their uncertainty attached Every reported proportion shows its sample size and interval. ### Steps 1. Compute intervals from the sample A proportion without one is a claim you cannot support. Files: `src/stats.ts`, `src/stats.test.ts` 2. Show the interval as prominently as the estimate Otherwise only the point estimate is read. Files: `src/report.ts` 3. Refuse to call an overlap a change It is the specific error this whole build exists to avoid. Files: `src/compare.ts` ### Done when - [ ] Every proportion shows sample size and interval - [ ] Overlapping changes are reported as indistinguishable - [ ] A larger sample visibly narrows the interval - [ ] Interval calculation matches known values ### Watch out - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. ## Phase 4 · Where the answer said it got that Cited sources are recorded and their accuracy is checkable. ### Steps 1. Extract citations with context The surrounding sentence is what the citation was supporting. Files: `src/citations.ts` 2. Verify the page resolves Generated citations are sometimes to pages that do not exist. Files: `src/verify.ts` 3. Aggregate by domain A recurring source is the actionable finding. Files: `src/citations.ts` ### Done when - [ ] Citations are extracted with context - [ ] Unresolvable citations are recorded as such - [ ] Recurring domains are visible - [ ] No citations and failed citations are distinguished ### Watch out - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. ## Not in this build - Any rank or position figure. - Attribution of traffic or revenue to a mention. - Advice on how to be mentioned more. - Coverage of every assistant. ## After v1, if you want it - Alerting on a mention rate change large enough to clear the interval, which is the only actionable signal here - Prompt clustering so similar questions are measured together with a larger effective sample - Recording the full response set for a period, which makes a qualitative read possible
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Sampled responses and extracted mentions. DATABASE_URL=postgres://localhost:5432/pulse # Required. The sample size. Below the minimum no figure is reported at all. RUNS_PER_PROMPT=30 # Required. Below this a prompt produces no number rather than a provisional one. MIN_SAMPLE_FOR_REPORT=20 # Required. Sampling multiplies cost. The limit refuses rather than warns. MONTHLY_BUDGET_MINOR=10000 # Required. Present as a constant. A generated answer has no ranking. REPORT_RANK=never # Required · secret. Billed per token across every run of every prompt. MODEL_API_KEY=a provider key
PRODUCT.md
# LLM Pulse · product brief ## Problem The core loop is a realistic weekend build: run a fixed prompt set against a model API, detect brand and competitor mentions, collect citations, and chart the results. The gap appears when you need dependable runs across many models, long-term evidence, team access, exports, alerts, and the broader visibility and reputation workflow. ## Product outcome A measurement of how often you appear in generated answers, with the sample size and uncertainty attached, that refuses to describe noise as a trend. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - one compatible model API key - Node.js 22 - SQLite - a scheduled local process - a small API budget ## Explicit non-goals for v1 - Any rank or position figure. - Attribution of traffic or revenue to a mention. - Advice on how to be mentioned more. - Coverage of every assistant. - managed execution across the full model set - long-term historical comparisons and evidence - reputation, source, traffic, and competitor workflows - team permissions, exports, alerts, and integrations - production monitoring and support ## Success criteria - Below-minimum samples are confirmed to produce no figure - Every proportion is confirmed shown with sample size and interval - Overlapping changes are confirmed reported as indistinguishable - Unresolvable citations are confirmed not counted - No rank or position figure is confirmed to exist
BRIEF.md
# Build brief · LLM Pulse The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me tracking how your brand appears in generated answers, without pretending it is a ranking to replace LLM Pulse. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This category borrows the language of search ranking for something that does not behave like one, and the borrowed language is what makes the numbers misleading. A generated answer is not a ranked list. Ask the same question twice and you can get different answers, different sources and a different mention of you, from the same model on the same day. There is no position to hold, no index to appear in, and no equivalent of a rank that persists between two identical queries. So the only honest measurement is a distribution. Run each prompt many times, report how often you were mentioned and with what variation, and show the sample size beside every figure. A single run is an anecdote, and a percentage from ten runs has an uncertainty wide enough that a change of ten points means nothing. The build therefore refuses to report a single-run result as a finding, refuses to call anything a rank, and shows the interval rather than the point estimate. That makes the numbers less exciting and considerably more useful. ### Stack (fixed, do not substitute) - Sampling: Every prompt run repeatedly, results stored per run · one run is an anecdote and the variation is the finding - Extraction: Mentions and cited sources identified per response with the quote kept · a claim about a mention should be checkable - Statistics: Proportions with intervals, sample size shown beside every figure · a percentage from ten runs is not a percentage - Cost: Per-run cost tracked, since sampling multiplies it · honest measurement is expensive and the budget must be visible - Runtime: TypeScript with PostgreSQL · runs, extractions and aggregates are relational ### Data model (create this before Phase 1) - prompts(id, text, category, added_at, retired_at) - runs(id, prompt_id, model, model_version, at, response_text, tokens_in, tokens_out, cost_minor) - mentions(run_id, entity_id, quote, char_start, char_end, sentiment, position_in_response) - citations(run_id, url, domain, quote_context) · where the answer said it got something - entities(id, name, aliases_json, kind) · kind is own_brand or competitor - aggregates(prompt_id, entity_id, model, window_start, sample_size, mention_rate, interval_low, interval_high) - There is deliberately no rank or position column, because a generated answer has no ranking and a stored number would be presented as one. ### Phase 1 · Sampling rather than asking once Build: Run each prompt a configured number of times per model, storing every response. Record the model version with each run. Refuse to report on a prompt with fewer runs than the minimum sample. Trap: Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. Done when: Each prompt runs the configured number of times and every response is stored. A prompt with fewer runs than the minimum produces no reported figure. The model version is recorded per run and a version change is visible in the data. Two sampling rounds on an unchanged prompt show the variation rather than hiding it. Do not build yet: No extraction, aggregation or reporting. ### Phase 2 · Finding the mention, with the quote Build: Detect mentions of your brand and competitors including aliases, storing the quote and position. Record where in the response the mention appeared. Handle a mention that is negative or that recommends a competitor instead. Trap: Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. Done when: Every mention stores the quote and its position, and the quote resolves to the response text. Aliases and common misspellings are matched. A mention that dismisses the brand is recorded distinctly from one that recommends it. A response mentioning nobody is recorded as such rather than being skipped. Do not build yet: No aggregation, citations or reporting yet. ### Phase 3 · Numbers with their uncertainty attached Build: Compute mention rates as proportions with a confidence interval from the sample size. Display the interval as prominently as the estimate. Refuse to report a change between two periods whose intervals overlap as a change. Trap: Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. Done when: Every proportion is displayed with its sample size and interval. A period-over-period change whose intervals overlap is reported as not distinguishable rather than as a change. Increasing the sample narrows the interval visibly. The interval calculation is unit tested against known values. Do not build yet: No citations, competitors or cost reporting yet. ### Phase 4 · Where the answer said it got that Build: Extract cited sources per response with the surrounding context. Verify that a cited page exists and, where possible, that it supports the claim. Record a citation that does not resolve as such. Trap: Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. Done when: Cited sources are extracted with context and their resolution is checked. A citation to a page that does not exist is recorded as unresolved rather than counted. Domains are aggregated so a recurring source is visible. A response with no citations is distinguished from one whose citations failed to resolve. Do not build yet: No competitor comparison or cost reporting yet. ### Phase 5 · Cost, comparison and what this cannot tell you Build: Track cost per prompt per round, since sampling multiplies it. Compare against competitors on the same samples. State plainly what the measurement does not establish, including that it does not measure traffic or influence. Trap: Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read. Done when: Cost per sampling round is tracked and projected before running. Competitors are compared on the same prompts and sample sizes rather than on separate ones. The report states what the measurement does not establish, prominently. A budget limit refuses further sampling rather than warning. Do not build yet: Nothing after this. ### Out of scope (and why) - Any rank or position figure, since a generated answer has no ranking and the word implies a stability that does not exist. - Attributing traffic or revenue to a mention, which is not observable from the answer side at all. - Advice on how to be mentioned more, which is speculation dressed as method and changes with every model update. - Coverage of every assistant, which multiplies sampling cost linearly and produces incomparable figures. ### README must contain - That a generated answer has no ranking, so nothing in this build reports a position - That every prompt is sampled repeatedly and a below-minimum sample produces no figure - That every proportion is shown with its sample size and interval - That a period-over-period change with overlapping intervals is reported as indistinguishable - That mention rate is not traffic and not influence, stated where the figures are read
ARCHITECTURE.md
# Architecture · LLM Pulse ## Stack | Part | Choice | Why | | --- | --- | --- | | Sampling | Every prompt run repeatedly, results stored per run | one run is an anecdote and the variation is the finding | | Extraction | Mentions and cited sources identified per response with the quote kept | a claim about a mention should be checkable | | Statistics | Proportions with intervals, sample size shown beside every figure | a percentage from ten runs is not a percentage | | Cost | Per-run cost tracked, since sampling multiplies it | honest measurement is expensive and the budget must be visible | | Runtime | TypeScript with PostgreSQL | runs, extractions and aggregates are relational | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Sampling | Repeated runs with model versions recorded. | Any provider, provided one run is never treated as a measurement. | | Extraction | Quoted mentions with sentiment. | Any extractor, provided quotes are stored and checkable. | | Statistics | Proportions with intervals. | Any method, provided the interval is as prominent as the estimate. | | Citations | Extraction and resolution checking. | Any implementation, provided unresolvable citations are not counted. | | Reporting | Comparison on identical samples with stated limits. | Any presentation, provided no rank is implied. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Sampled responses and extracted mentions. - `RUNS_PER_PROMPT` · required · The sample size. Below the minimum no figure is reported at all. - `MIN_SAMPLE_FOR_REPORT` · required · Below this a prompt produces no number rather than a provisional one. - `MONTHLY_BUDGET_MINOR` · required · Sampling multiplies cost. The limit refuses rather than warns. - `REPORT_RANK` · required · Present as a constant. A generated answer has no ranking. - `MODEL_API_KEY` · required, secret · Billed per token across every run of every prompt. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · LLM Pulse product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Every prompt run repeatedly, results stored per run, Mentions and cited sources identified per response with the quote kept, Proportions with intervals, sample size shown beside every figure, Per-run cost tracked, since sampling multiplies it, TypeScript with PostgreSQL. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. - Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read.
MILESTONES.md
# Delivery milestones · LLM Pulse Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Sampling rather than asking once Every prompt is run enough times to say something. ### Steps 1. Run repeatedly and store every response The variation between runs is the finding, not noise to average away. Files: `src/sample.ts` 2. Record the model version per run A model update changes everything and must be visible in the series. Files: `src/runs.ts` 3. Refuse below the minimum sample A provisional figure is read as a figure. Files: `src/aggregate.ts` ### Done when - [ ] Each prompt runs the configured number of times - [ ] Below-minimum samples produce no figure - [ ] Model versions are recorded per run - [ ] Variation between rounds is visible ### Watch out - Running a prompt once and recording whether you were mentioned produces a metric that flips between measurements for no reason at all, and a report built on it will show a trend that is entirely noise. Sample repeatedly, and make a below-minimum sample produce no figure rather than a provisional one. ## M2 · Finding the mention, with the quote Every recorded mention can be checked against the response. ### Steps 1. Keep the quote and position So a claim about a mention can be checked. Files: `src/mentions.ts` 2. Match aliases and misspellings Models get names slightly wrong constantly. Files: `src/entities.ts` 3. Record sentiment separately from presence Being dismissed is worse than being absent. Files: `src/sentiment.ts` ### Done when - [ ] Mentions store a resolvable quote and position - [ ] Aliases and misspellings match - [ ] Dismissals are distinct from recommendations - [ ] Responses mentioning nobody are recorded ### Watch out - Counting a mention without keeping the context treats being recommended and being dismissed as the same event, and the second is worse than not being mentioned at all. Keep the quote, record the sentiment, and never report a bare mention count as a positive signal. ## M3 · Numbers with their uncertainty attached Every reported proportion shows its sample size and interval. ### Steps 1. Compute intervals from the sample A proportion without one is a claim you cannot support. Files: `src/stats.ts`, `src/stats.test.ts` 2. Show the interval as prominently as the estimate Otherwise only the point estimate is read. Files: `src/report.ts` 3. Refuse to call an overlap a change It is the specific error this whole build exists to avoid. Files: `src/compare.ts` ### Done when - [ ] Every proportion shows sample size and interval - [ ] Overlapping changes are reported as indistinguishable - [ ] A larger sample visibly narrows the interval - [ ] Interval calculation matches known values ### Watch out - Reporting a move from forty percent to fifty percent as a ten point improvement, when both come from twenty runs, describes noise with the confidence of a measurement, and someone will change their content strategy because of it. Compute intervals and refuse to call an overlapping change a change. ## M4 · Where the answer said it got that Cited sources are recorded and their accuracy is checkable. ### Steps 1. Extract citations with context The surrounding sentence is what the citation was supporting. Files: `src/citations.ts` 2. Verify the page resolves Generated citations are sometimes to pages that do not exist. Files: `src/verify.ts` 3. Aggregate by domain A recurring source is the actionable finding. Files: `src/citations.ts` ### Done when - [ ] Citations are extracted with context - [ ] Unresolvable citations are recorded as such - [ ] Recurring domains are visible - [ ] No citations and failed citations are distinguished ### Watch out - Treating a citation as evidence that a page influenced the answer assumes the citation is accurate, and generated citations are sometimes to pages that do not exist or do not say what was claimed. Check that the page resolves, and record unresolvable citations rather than counting them. ## M5 · Cost, comparison and what this cannot tell you (production only) The report is affordable, comparative and honest about its limits. ### Steps 1. Track and project cost Sampling multiplies it, which is the price of an honest number. Files: `src/cost.ts` 2. Compare on identical samples Different sample sizes make a comparison meaningless. Files: `src/compare.ts` 3. State the limits where they are read Mention rate is not traffic and not influence. Files: `src/report.ts` ### Done when - [ ] Cost is tracked and projected before a round - [ ] Competitor comparison uses identical samples - [ ] The report states its limits prominently - [ ] A budget limit refuses rather than warns ### Watch out - Presenting mention rate as though it corresponds to traffic or to influence implies a causal link the data cannot support, since being mentioned in an answer and someone acting on it are different events and only the first is observable here. Say so where the figures are read.
OPERATIONS.md
# Operations · LLM Pulse ## Backup Sampled responses are expensive to reproduce, since regenerating them costs the same money again and the model may have changed. Back them up; the aggregates recompute from them. ## Restore Restore and confirm model versions survived alongside the runs, since comparing samples taken under different model versions without knowing it produces a trend that is entirely an artefact. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Cost per round against budget, and model version changes. A version change invalidates comparison across it and is the single most common cause of an apparent trend. ## Incident checklist A reported trend that turns out to be a model update rather than a change in your visibility is the failure this build is organised against. The recorded model version per run is what makes that diagnosable in minutes. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Below-minimum samples are confirmed to produce no figure - [ ] Every proportion is confirmed shown with sample size and interval - [ ] Overlapping changes are confirmed reported as indistinguishable - [ ] Unresolvable citations are confirmed not counted - [ ] No rank or position figure is confirmed to exist ## Launch constraint Do not market omitted LLM Pulse capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Sampled responses and extracted mentions. DATABASE_URL=postgres://localhost:5432/pulse # Required. The sample size. Below the minimum no figure is reported at all. RUNS_PER_PROMPT=30 # Required. Below this a prompt produces no number rather than a provisional one. MIN_SAMPLE_FOR_REPORT=20 # Required. Sampling multiplies cost. The limit refuses rather than warns. MONTHLY_BUDGET_MINOR=10000 # Required. Present as a constant. A generated answer has no ranking. REPORT_RANK=never # Required · secret. Billed per token across every run of every prompt. MODEL_API_KEY=a provider key
$ choose a build depth, inspect the files, then open the complete pack in your agent
Teams pay to keep large prompt sets running on schedule, preserve evidence over time, and analyze mentions, citations, sentiment, competitors, and traffic in one dependable workflow without maintaining the execution pipeline themselves.
xmanaged execution across the full model set
xlong-term historical comparisons and evidence
xreputation, source, traffic, and competitor workflows
xteam permissions, exports, alerts, and integrations
xproduction monitoring and support
share on X ↗"I just replaced LLM Pulse ($56.52/mo) with one prompt"
Don't feel like building it? These folks already made it free.
ElmoTracks mentions, citations and competitors across the major engines; sentiment and referral traffic are still on the road map.open source↗no votes, no pay-to-list · just what's real
LLM Pulse pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| starter weekly | $56.52/workspace | $47.09/workspace | 1 project; 50 prompts; 50 AI responses/week/model; 10 competitors. |
| growth weekly | $114.19/workspace | $95.16/workspace | 2 projects; 150 prompts; 150 AI responses/week/model; 15 competitors. |
| scale weekly | $344.87/workspace | $287.39/workspace | 5 projects; 450 prompts; 450 AI responses/week/model; 20 competitors. |
| scale+ weekly | $690.89/workspace | $575.74/workspace | 10 projects; 1,200 prompts; 1,200 AI responses/week/model; 20 competitors. |
| scale++ weekly | $1382.93/workspace | $1152.44/workspace | 15 projects; 2,400 prompts; 2,400 AI responses/week/model; 25 competitors. |
| starter daily | $91.12/workspace | $75.93/workspace | 1 project; 50 prompts; 50 AI responses/day/model; 10 competitors. |
| growth daily | $171.86/workspace | $143.22/workspace | 2 projects; 150 prompts; 150 AI responses/day/model; 15 competitors. |
| scale daily | $517.88/workspace | $431.57/workspace | 5 projects; 450 prompts; 450 AI responses/day/model; 20 competitors. |
| scale+ daily | $1036.91/workspace | $864.09/workspace | 10 projects; 1,200 prompts; 1,200 AI responses/day/model; 20 competitors. |
| scale++ daily | $2190.31/workspace | $1825.26/workspace | 15 projects; 2,400 prompts; 2,400 AI responses/day/model; 25 competitors. |
| enterprise | custom | — | Custom projects, prompt volume, refresh frequency, model coverage, data access, and support. |
free tierno free tier; 14-day card-required trial on weekly Starter, Growth, and Scale only; daily plans and Scale+/Scale++ start immediately
billingmonthly + annual (annual is billed for 10 months, effectively 2 months free); VAT/tax may be added
hidden costsAdditional AI models are sold as paid add-ons; public add-on rates are not disclosed.
verified 2026-08-14 · source ↗
Vibecode LLM Pulse
Kinda. The core of LLM Pulse is buildable in a weekend with the prompt on this page, but there are real gaps: managed execution across the full model set, long-term historical comparisons and evidence. Read the honest list above before committing.
How much does LLM Pulse cost?
LLM Pulse costs about $56.52/month (Starter Weekly, checked 2026-08-14), which is $678.24 per year.
What do I lose by replacing LLM Pulse?
Honestly: managed execution across the full model set; long-term historical comparisons and evidence; reputation, source, traffic, and competitor workflows; team permissions, exports, alerts, and integrations; production monitoring and support. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to LLM Pulse?
Yes: Elmo (Tracks mentions, citations and competitors across the major engines; sentiment and referral traffic are still on the road map.) The prompt is for when you want it exactly your way.