Vibecode Surfer SEO
track this build5 phases, 15 steps, beginner friendly0%KINDA · weekend project
A SERP-scrape-to-content-brief tool is buildable, but reliable SERP data, NLP scoring, editor workflow, AI articles, audits, and integrations create ongoing work.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Surfer SEO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Surfer SEO · indie build Fetch the pages currently ranking for a query through a search data provider, extract their structure and vocabulary, produce a brief covering the subtopics they all address, score a draft against that coverage as a checklist, and record what actually happened to the page afterwards so the score can be judged rather than trusted. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Data | A paid search results API | the only supportable source; scraping is against terms and returns unreliable data | | Runtime | Python 3.12 with SQLite | fetching, extracting and comparing text, at a volume one person generates | | Extraction | trafilatura for article text and a parser for heading structure | the structure matters as much as the words and is easier to compare | | Analysis | Term frequency comparison plus subtopic coverage | explainable arithmetic beats an opaque score you cannot debug or defend | | Outcomes | Recorded position over time per page you published | the only way to learn whether any of this worked | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the fetching and analysis. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **A paid search results API** · priced per call; a few hundred queries a month is typically tens of US dollars and position tracking multiplies it - Why: This is the input and it has no free substitute. Scraping is against terms, unreliable and blocked, which makes every downstream number meaningless. - Get it: Choose a provider, read its per-call pricing carefully, and start on the smallest plan. Estimate your monthly calls from briefs plus position tracking before committing. - Verify: A query returns ranked results with urls - [ ] **Ten published pages of your own with known outcomes** · free - Why: Phase 5 evaluates whether any of this works, and it needs pages whose positions you can actually track. Without them the tool cannot be judged. - Get it: Use pages you have already published. If you have none, this build cannot be evaluated yet and you should know that before spending on search data. - Verify: Ten pages with target queries are recorded - [ ] **A willingness to accept a negative result** · free - Why: The most likely finding is that coverage score and outcome are weakly related. A tool built to confirm a belief is worse than no tool. - Get it: Decide before Phase 5 that you will act on the finding either way, and write that down. - Verify: The commitment is recorded before outcomes are measured - [ ] **A monthly budget for search data** · the cap you choose - Why: Per-call pricing plus weekly position tracking grows faster than expected, and the point of leaving a subscription is spending less. - Get it: Set a figure and a hard cap at the provider before the first bulk run. - Verify: A provider spend cap is configured ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Search result data of your own. - The commercial scoring model. - The editor experience. - Any causal claim about ranking. - SERP data reliability - scoring model - editor UX - AI article workflows - integrations - audits - support If one of those is essential to you, that is the reason to keep paying for Surfer SEO, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Surfer SEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a content brief built from what currently ranks, with an honest view of what the score is worth to replace Surfer SEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two things to settle before building. Search result data has to come from a paid provider: scraping a search engine's result pages breaks its terms and the result is unreliable and blocked. Budget for an API or accept that this build has no data. And be clear-eyed about the score. Matching the term frequencies of pages that currently rank is a proxy, not a cause: those pages may rank because of links, domain history or brand, none of which your word count reproduces. A score is useful as a checklist for things you forgot to cover and actively harmful as a target to maximise, because writing to it produces text nobody wants to read. Build it as the first and label it clearly. ### Stack (fixed, do not substitute) - Data: A paid search results API · the only supportable source; scraping is against terms and returns unreliable data - Runtime: Python 3.12 with SQLite · fetching, extracting and comparing text, at a volume one person generates - Extraction: trafilatura for article text and a parser for heading structure · the structure matters as much as the words and is easier to compare - Analysis: Term frequency comparison plus subtopic coverage · explainable arithmetic beats an opaque score you cannot debug or defend - Outcomes: Recorded position over time per page you published · the only way to learn whether any of this worked ### Data model (create this before Phase 1) - queries(id, text, locale, device, checked_at) - serp_results(query_id, position, url, title, fetched_at) - pages(id, url, fetched_at, word_count, headings_json, text, status) - terms(page_id, term, count) · after normalisation, for comparison rather than for display - subtopics(query_id, name, present_in_count, example_headings_json) · what most ranking pages cover - briefs(id, query_id, created_at, subtopics_json, target_words, notes) - drafts(id, brief_id, text, scored_at, coverage_json, score) - outcomes(page_url, query_id, checked_on, position) · what happened after publishing, which is the only real evidence ### Phase 1 · Search data from a supportable source Build: Connect a search results API with locale and device parameters. Store results with the date fetched, because rankings change and a brief built last month was built against a different page set. Cache aggressively, since these queries are priced per call. Trap: Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. Done when: A query returns ranked results with urls, recorded with a fetch date. A repeated query within the cache window makes no billed call, verified by comparing the call count. Locale and device parameters change the results as expected. Cost per query is recorded. Do not build yet: No page fetching, extraction or analysis. ### Phase 2 · Fetching and extracting the ranking pages Build: Fetch each ranking page politely with a timeout and byte cap, extract the article text and the heading hierarchy, and record failures. Respect robots and identify yourself. Pages you cannot fetch are excluded rather than guessed at. Trap: Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. Done when: Ranking pages are fetched with their heading structure preserved. A page that cannot be fetched is excluded and counted rather than treated as empty. The brief reports how many of the results were successfully analysed. Fetching is rate limited and identifies itself. Do not build yet: No analysis or brief yet. ### Phase 3 · Subtopics, which are more useful than terms Build: Cluster the heading structures across ranking pages to find subtopics most of them address. Report each subtopic with how many pages cover it and example headings. Derive a word count range from the analysed pages rather than a single target. Trap: A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. Done when: A brief lists subtopics with the number of ranking pages covering each. Word count is reported as a range with a median rather than a single target. A subtopic covered by one page of ten is reported as such rather than presented as required. Example headings are shown so a subtopic is understandable. Do not build yet: No scoring or outcome tracking. ### Phase 4 · Scoring a draft, honestly labelled Build: Score a draft on subtopic coverage rather than term density: which subtopics it addresses and which it omits. Show a term comparison as secondary information. Label the score as a checklist and refuse to present it as a prediction of ranking. Trap: A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. Done when: A draft's score is presented as covered and uncovered subtopics rather than a single number to maximise. Term comparison is available and clearly secondary. No interface text implies the score predicts a position. Adding a paragraph covering a missing subtopic changes the checklist as expected. Do not build yet: No outcome tracking yet, which is the honest part. ### Phase 5 · What actually happened Build: Track the position of pages you published for their target queries on a schedule. Compare outcomes against the coverage score at publication. Report the relationship honestly, including when there is none. Trap: Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. Done when: Positions are tracked on a schedule for published pages. A report compares coverage score at publication against position after a period. The report states plainly when the relationship is weak. At least ten published pages are tracked before any conclusion is drawn. Do not build yet: Cost comparison comes last. ### Phase 6 · Cost, and whether this was worth it Build: Total the search data spend per month. Compare against the subscription you replaced and against the outcomes you measured. Write down the decision either way. Trap: Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong. Done when: Monthly search data spend is reported and compared against the subscription price. The comparison includes outcome evidence rather than only cost. A written decision records whether to keep this, with reasoning. Position tracking frequency can be reduced to control spend. Do not build yet: Nothing after this. ### Out of scope (and why) - Reliable search result data of your own. It comes from a paid provider and that cost is the floor of this build. - The scoring model behind the commercial product, which is tuned against far more data than you have and is still a proxy. - The editor experience, which is much of what people pay for and which a checklist does not replace. - Any claim that following a brief causes a page to rank. Correlation with pages that already rank is not a mechanism. ### README must contain - That search data comes from a paid provider and scraping is neither permitted nor reliable - That the score is a coverage checklist rather than a ranking prediction - That word count is reported as a range from analysed pages rather than a target - That pages which could not be fetched are excluded and counted - The measured relationship between coverage score and actual outcomes ===== AGENTS.md ===== # Agent instructions · Surfer SEO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A paid search results API, Python 3.12 with SQLite, trafilatura for article text and a parser for heading structure, Term frequency comparison plus subtopic coverage, Recorded position over time per page you published. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. ===== BUILD_PLAN.md ===== # Build plan · Surfer SEO Fetch the pages currently ranking for a query through a search data provider, extract their structure and vocabulary, produce a brief covering the subtopics they all address, score a draft against that coverage as a checklist, and record what actually happened to the page afterwards so the score can be judged rather than trusted. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Search data from a supportable source You can fetch what currently ranks for a query, repeatedly, without being blocked. ### Steps 1. Connect a search data provider With locale and device. Read the pricing before building, because this is the recurring cost. Files: `src/serp.py` 2. Cache by query, locale and device These calls are priced individually and rankings do not change hourly. Files: `src/cache.py` 3. Record cost per query So the eventual comparison against the subscription is real rather than a feeling. Files: `src/cost.py` ### Done when - [ ] A query returns ranked results with a fetch date - [ ] A cached query makes no billed call - [ ] Locale and device change results - [ ] Cost per query is recorded ### Watch out - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. ## Phase 2 · Fetching and extracting the ranking pages You have the text and structure of the pages you are comparing against. ### Steps 1. Fetch politely with limits Timeout, byte cap, rate limit, honest user agent. You are reading someone's site. Files: `src/fetch.py` 2. Extract text and headings The hierarchy matters as much as the words for a brief. Files: `src/extract.py` 3. Exclude and count failures Never treat a failure as empty. Report the analysed count on the brief. Files: `src/extract.py` ### Done when - [ ] Heading structure is preserved on extraction - [ ] A failed fetch is excluded and counted - [ ] The analysed count appears on the brief - [ ] Fetching is rate limited and identifies itself ### Watch out - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. ## Phase 3 · Subtopics, which are more useful than terms The brief says what the ranking pages all cover, which is the part worth acting on. ### Steps 1. Cluster headings into subtopics Across pages, grouping equivalent headings. Report coverage counts rather than a flat list. Files: `src/subtopics.py` 2. Report ranges, not targets Word count as a range and median from analysed pages only. Files: `src/brief.py` 3. Show the evidence Example headings per subtopic, so the writer can judge relevance rather than obeying. Files: `src/brief.py` ### Done when - [ ] Subtopics list their coverage counts - [ ] Word count is a range with a median - [ ] A rarely covered subtopic is marked as such - [ ] Example headings are shown per subtopic ### Watch out - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. ## Phase 4 · Scoring a draft, honestly labelled A draft gets a coverage checklist rather than a number to maximise. ### Steps 1. Score coverage first Subtopics addressed and omitted. That is the actionable output. Files: `src/score.py` 2. Keep terms secondary Available, labelled, not the headline. Term matching is a weak proxy and should look like one. Files: `src/score.py` 3. Refuse predictive language Nothing in the interface should imply a ranking outcome. Check the strings deliberately. Files: `src/ui.py` ### Done when - [ ] The score is presented as covered and uncovered subtopics - [ ] Term comparison is clearly secondary - [ ] No interface text implies a ranking prediction - [ ] Covering a missing subtopic updates the checklist ### Watch out - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. ## Phase 5 · What actually happened You record where your published pages ended up, so the tool can be judged. ### Steps 1. Track positions over time On a schedule, through the same paid API, with the cost accounted for. Files: `src/tracking.py` 2. Compare score against outcome At a sample size where the comparison means something. Ten pages is a minimum, not a good number. Files: `src/evaluate.py` 3. Report weak relationships plainly It is the most likely finding and the most useful one. Files: `src/evaluate.py` ### Done when - [ ] Positions are tracked on a schedule - [ ] Score and outcome are compared over at least ten pages - [ ] A weak relationship is reported plainly - [ ] Tracking cost is included in the running total ### Watch out - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. ## Not in this build - Search result data of your own. - The commercial scoring model. - The editor experience. - Any causal claim about ranking. ## After v1, if you want it - Tracking competitor pages for the same queries, which explains movement better than your own page alone - A content refresh queue for pages whose position has decayed - Comparing briefs against outcomes by subtopic, which is the analysis that would actually teach you something ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your search data provider. Priced per call, so treat it as a spending instrument. SERP_API_KEY=replace-with-provider-key # Required. How long a query's results are reused. Rankings do not change hourly and calls cost money. SERP_CACHE_HOURS=168 # Required. Per-page timeout when reading ranking pages. FETCH_TIMEOUT_MS=8000 # Required. Identify yourself honestly. You are reading other people's sites. FETCH_USER_AGENT=ContentBrief/1.0 (contact: you@example.com) # Required. How often positions are checked. This is the cost that grows quietly. TRACKING_INTERVAL_DAYS=7 # Required. Stop making billed calls at this figure. Mirror it as a hard cap at the provider. MONTHLY_SPEND_CAP_USD=30
You are building a lean indie version of Surfer SEO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Surfer SEO · indie build Fetch the pages currently ranking for a query through a search data provider, extract their structure and vocabulary, produce a brief covering the subtopics they all address, score a draft against that coverage as a checklist, and record what actually happened to the page afterwards so the score can be judged rather than trusted. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Data | A paid search results API | the only supportable source; scraping is against terms and returns unreliable data | | Runtime | Python 3.12 with SQLite | fetching, extracting and comparing text, at a volume one person generates | | Extraction | trafilatura for article text and a parser for heading structure | the structure matters as much as the words and is easier to compare | | Analysis | Term frequency comparison plus subtopic coverage | explainable arithmetic beats an opaque score you cannot debug or defend | | Outcomes | Recorded position over time per page you published | the only way to learn whether any of this worked | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the fetching and analysis. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **A paid search results API** · priced per call; a few hundred queries a month is typically tens of US dollars and position tracking multiplies it - Why: This is the input and it has no free substitute. Scraping is against terms, unreliable and blocked, which makes every downstream number meaningless. - Get it: Choose a provider, read its per-call pricing carefully, and start on the smallest plan. Estimate your monthly calls from briefs plus position tracking before committing. - Verify: A query returns ranked results with urls - [ ] **Ten published pages of your own with known outcomes** · free - Why: Phase 5 evaluates whether any of this works, and it needs pages whose positions you can actually track. Without them the tool cannot be judged. - Get it: Use pages you have already published. If you have none, this build cannot be evaluated yet and you should know that before spending on search data. - Verify: Ten pages with target queries are recorded - [ ] **A willingness to accept a negative result** · free - Why: The most likely finding is that coverage score and outcome are weakly related. A tool built to confirm a belief is worse than no tool. - Get it: Decide before Phase 5 that you will act on the finding either way, and write that down. - Verify: The commitment is recorded before outcomes are measured - [ ] **A monthly budget for search data** · the cap you choose - Why: Per-call pricing plus weekly position tracking grows faster than expected, and the point of leaving a subscription is spending less. - Get it: Set a figure and a hard cap at the provider before the first bulk run. - Verify: A provider spend cap is configured ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Search result data of your own. - The commercial scoring model. - The editor experience. - Any causal claim about ranking. - SERP data reliability - scoring model - editor UX - AI article workflows - integrations - audits - support If one of those is essential to you, that is the reason to keep paying for Surfer SEO, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Surfer SEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a content brief built from what currently ranks, with an honest view of what the score is worth to replace Surfer SEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two things to settle before building. Search result data has to come from a paid provider: scraping a search engine's result pages breaks its terms and the result is unreliable and blocked. Budget for an API or accept that this build has no data. And be clear-eyed about the score. Matching the term frequencies of pages that currently rank is a proxy, not a cause: those pages may rank because of links, domain history or brand, none of which your word count reproduces. A score is useful as a checklist for things you forgot to cover and actively harmful as a target to maximise, because writing to it produces text nobody wants to read. Build it as the first and label it clearly. ### Stack (fixed, do not substitute) - Data: A paid search results API · the only supportable source; scraping is against terms and returns unreliable data - Runtime: Python 3.12 with SQLite · fetching, extracting and comparing text, at a volume one person generates - Extraction: trafilatura for article text and a parser for heading structure · the structure matters as much as the words and is easier to compare - Analysis: Term frequency comparison plus subtopic coverage · explainable arithmetic beats an opaque score you cannot debug or defend - Outcomes: Recorded position over time per page you published · the only way to learn whether any of this worked ### Data model (create this before Phase 1) - queries(id, text, locale, device, checked_at) - serp_results(query_id, position, url, title, fetched_at) - pages(id, url, fetched_at, word_count, headings_json, text, status) - terms(page_id, term, count) · after normalisation, for comparison rather than for display - subtopics(query_id, name, present_in_count, example_headings_json) · what most ranking pages cover - briefs(id, query_id, created_at, subtopics_json, target_words, notes) - drafts(id, brief_id, text, scored_at, coverage_json, score) - outcomes(page_url, query_id, checked_on, position) · what happened after publishing, which is the only real evidence ### Phase 1 · Search data from a supportable source Build: Connect a search results API with locale and device parameters. Store results with the date fetched, because rankings change and a brief built last month was built against a different page set. Cache aggressively, since these queries are priced per call. Trap: Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. Done when: A query returns ranked results with urls, recorded with a fetch date. A repeated query within the cache window makes no billed call, verified by comparing the call count. Locale and device parameters change the results as expected. Cost per query is recorded. Do not build yet: No page fetching, extraction or analysis. ### Phase 2 · Fetching and extracting the ranking pages Build: Fetch each ranking page politely with a timeout and byte cap, extract the article text and the heading hierarchy, and record failures. Respect robots and identify yourself. Pages you cannot fetch are excluded rather than guessed at. Trap: Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. Done when: Ranking pages are fetched with their heading structure preserved. A page that cannot be fetched is excluded and counted rather than treated as empty. The brief reports how many of the results were successfully analysed. Fetching is rate limited and identifies itself. Do not build yet: No analysis or brief yet. ### Phase 3 · Subtopics, which are more useful than terms Build: Cluster the heading structures across ranking pages to find subtopics most of them address. Report each subtopic with how many pages cover it and example headings. Derive a word count range from the analysed pages rather than a single target. Trap: A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. Done when: A brief lists subtopics with the number of ranking pages covering each. Word count is reported as a range with a median rather than a single target. A subtopic covered by one page of ten is reported as such rather than presented as required. Example headings are shown so a subtopic is understandable. Do not build yet: No scoring or outcome tracking. ### Phase 4 · Scoring a draft, honestly labelled Build: Score a draft on subtopic coverage rather than term density: which subtopics it addresses and which it omits. Show a term comparison as secondary information. Label the score as a checklist and refuse to present it as a prediction of ranking. Trap: A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. Done when: A draft's score is presented as covered and uncovered subtopics rather than a single number to maximise. Term comparison is available and clearly secondary. No interface text implies the score predicts a position. Adding a paragraph covering a missing subtopic changes the checklist as expected. Do not build yet: No outcome tracking yet, which is the honest part. ### Phase 5 · What actually happened Build: Track the position of pages you published for their target queries on a schedule. Compare outcomes against the coverage score at publication. Report the relationship honestly, including when there is none. Trap: Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. Done when: Positions are tracked on a schedule for published pages. A report compares coverage score at publication against position after a period. The report states plainly when the relationship is weak. At least ten published pages are tracked before any conclusion is drawn. Do not build yet: Cost comparison comes last. ### Phase 6 · Cost, and whether this was worth it Build: Total the search data spend per month. Compare against the subscription you replaced and against the outcomes you measured. Write down the decision either way. Trap: Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong. Done when: Monthly search data spend is reported and compared against the subscription price. The comparison includes outcome evidence rather than only cost. A written decision records whether to keep this, with reasoning. Position tracking frequency can be reduced to control spend. Do not build yet: Nothing after this. ### Out of scope (and why) - Reliable search result data of your own. It comes from a paid provider and that cost is the floor of this build. - The scoring model behind the commercial product, which is tuned against far more data than you have and is still a proxy. - The editor experience, which is much of what people pay for and which a checklist does not replace. - Any claim that following a brief causes a page to rank. Correlation with pages that already rank is not a mechanism. ### README must contain - That search data comes from a paid provider and scraping is neither permitted nor reliable - That the score is a coverage checklist rather than a ranking prediction - That word count is reported as a range from analysed pages rather than a target - That pages which could not be fetched are excluded and counted - The measured relationship between coverage score and actual outcomes ===== AGENTS.md ===== # Agent instructions · Surfer SEO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A paid search results API, Python 3.12 with SQLite, trafilatura for article text and a parser for heading structure, Term frequency comparison plus subtopic coverage, Recorded position over time per page you published. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. ===== BUILD_PLAN.md ===== # Build plan · Surfer SEO Fetch the pages currently ranking for a query through a search data provider, extract their structure and vocabulary, produce a brief covering the subtopics they all address, score a draft against that coverage as a checklist, and record what actually happened to the page afterwards so the score can be judged rather than trusted. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Search data from a supportable source You can fetch what currently ranks for a query, repeatedly, without being blocked. ### Steps 1. Connect a search data provider With locale and device. Read the pricing before building, because this is the recurring cost. Files: `src/serp.py` 2. Cache by query, locale and device These calls are priced individually and rankings do not change hourly. Files: `src/cache.py` 3. Record cost per query So the eventual comparison against the subscription is real rather than a feeling. Files: `src/cost.py` ### Done when - [ ] A query returns ranked results with a fetch date - [ ] A cached query makes no billed call - [ ] Locale and device change results - [ ] Cost per query is recorded ### Watch out - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. ## Phase 2 · Fetching and extracting the ranking pages You have the text and structure of the pages you are comparing against. ### Steps 1. Fetch politely with limits Timeout, byte cap, rate limit, honest user agent. You are reading someone's site. Files: `src/fetch.py` 2. Extract text and headings The hierarchy matters as much as the words for a brief. Files: `src/extract.py` 3. Exclude and count failures Never treat a failure as empty. Report the analysed count on the brief. Files: `src/extract.py` ### Done when - [ ] Heading structure is preserved on extraction - [ ] A failed fetch is excluded and counted - [ ] The analysed count appears on the brief - [ ] Fetching is rate limited and identifies itself ### Watch out - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. ## Phase 3 · Subtopics, which are more useful than terms The brief says what the ranking pages all cover, which is the part worth acting on. ### Steps 1. Cluster headings into subtopics Across pages, grouping equivalent headings. Report coverage counts rather than a flat list. Files: `src/subtopics.py` 2. Report ranges, not targets Word count as a range and median from analysed pages only. Files: `src/brief.py` 3. Show the evidence Example headings per subtopic, so the writer can judge relevance rather than obeying. Files: `src/brief.py` ### Done when - [ ] Subtopics list their coverage counts - [ ] Word count is a range with a median - [ ] A rarely covered subtopic is marked as such - [ ] Example headings are shown per subtopic ### Watch out - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. ## Phase 4 · Scoring a draft, honestly labelled A draft gets a coverage checklist rather than a number to maximise. ### Steps 1. Score coverage first Subtopics addressed and omitted. That is the actionable output. Files: `src/score.py` 2. Keep terms secondary Available, labelled, not the headline. Term matching is a weak proxy and should look like one. Files: `src/score.py` 3. Refuse predictive language Nothing in the interface should imply a ranking outcome. Check the strings deliberately. Files: `src/ui.py` ### Done when - [ ] The score is presented as covered and uncovered subtopics - [ ] Term comparison is clearly secondary - [ ] No interface text implies a ranking prediction - [ ] Covering a missing subtopic updates the checklist ### Watch out - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. ## Phase 5 · What actually happened You record where your published pages ended up, so the tool can be judged. ### Steps 1. Track positions over time On a schedule, through the same paid API, with the cost accounted for. Files: `src/tracking.py` 2. Compare score against outcome At a sample size where the comparison means something. Ten pages is a minimum, not a good number. Files: `src/evaluate.py` 3. Report weak relationships plainly It is the most likely finding and the most useful one. Files: `src/evaluate.py` ### Done when - [ ] Positions are tracked on a schedule - [ ] Score and outcome are compared over at least ten pages - [ ] A weak relationship is reported plainly - [ ] Tracking cost is included in the running total ### Watch out - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. ## Not in this build - Search result data of your own. - The commercial scoring model. - The editor experience. - Any causal claim about ranking. ## After v1, if you want it - Tracking competitor pages for the same queries, which explains movement better than your own page alone - A content refresh queue for pages whose position has decayed - Comparing briefs against outcomes by subtopic, which is the analysis that would actually teach you something ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your search data provider. Priced per call, so treat it as a spending instrument. SERP_API_KEY=replace-with-provider-key # Required. How long a query's results are reused. Rankings do not change hourly and calls cost money. SERP_CACHE_HOURS=168 # Required. Per-page timeout when reading ranking pages. FETCH_TIMEOUT_MS=8000 # Required. Identify yourself honestly. You are reading other people's sites. FETCH_USER_AGENT=ContentBrief/1.0 (contact: you@example.com) # Required. How often positions are checked. This is the cost that grows quietly. TRACKING_INTERVAL_DAYS=7 # Required. Stop making billed calls at this figure. Mirror it as a hard cap at the provider. MONTHLY_SPEND_CAP_USD=30
You are building a production product version of Surfer SEO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Surfer SEO · product brief ## Problem A SERP-scrape-to-content-brief tool is buildable, but reliable SERP data, NLP scoring, editor workflow, AI articles, audits, and integrations create ongoing work. ## Product outcome A brief built from pages that currently rank, presented as coverage you can act on rather than a number to chase, with outcomes tracked so you can tell whether any of it helped. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - search/SERP API - web scraper/parser - LLM API - content editor - scoring logic - hosted backend ## Explicit non-goals for v1 - Search result data of your own. - The commercial scoring model. - The editor experience. - Any causal claim about ranking. - SERP data reliability - scoring model - editor UX - AI article workflows - integrations - audits - support ## Success criteria - A cached query is confirmed to make no billed call - Failed page fetches are confirmed excluded and counted - No interface text implies a ranking prediction - Outcome tracking is running over at least ten pages - The monthly spend cap is configured at the provider as well as locally ===== BRIEF.md ===== # Build brief · Surfer SEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a content brief built from what currently ranks, with an honest view of what the score is worth to replace Surfer SEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two things to settle before building. Search result data has to come from a paid provider: scraping a search engine's result pages breaks its terms and the result is unreliable and blocked. Budget for an API or accept that this build has no data. And be clear-eyed about the score. Matching the term frequencies of pages that currently rank is a proxy, not a cause: those pages may rank because of links, domain history or brand, none of which your word count reproduces. A score is useful as a checklist for things you forgot to cover and actively harmful as a target to maximise, because writing to it produces text nobody wants to read. Build it as the first and label it clearly. ### Stack (fixed, do not substitute) - Data: A paid search results API · the only supportable source; scraping is against terms and returns unreliable data - Runtime: Python 3.12 with SQLite · fetching, extracting and comparing text, at a volume one person generates - Extraction: trafilatura for article text and a parser for heading structure · the structure matters as much as the words and is easier to compare - Analysis: Term frequency comparison plus subtopic coverage · explainable arithmetic beats an opaque score you cannot debug or defend - Outcomes: Recorded position over time per page you published · the only way to learn whether any of this worked ### Data model (create this before Phase 1) - queries(id, text, locale, device, checked_at) - serp_results(query_id, position, url, title, fetched_at) - pages(id, url, fetched_at, word_count, headings_json, text, status) - terms(page_id, term, count) · after normalisation, for comparison rather than for display - subtopics(query_id, name, present_in_count, example_headings_json) · what most ranking pages cover - briefs(id, query_id, created_at, subtopics_json, target_words, notes) - drafts(id, brief_id, text, scored_at, coverage_json, score) - outcomes(page_url, query_id, checked_on, position) · what happened after publishing, which is the only real evidence ### Phase 1 · Search data from a supportable source Build: Connect a search results API with locale and device parameters. Store results with the date fetched, because rankings change and a brief built last month was built against a different page set. Cache aggressively, since these queries are priced per call. Trap: Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. Done when: A query returns ranked results with urls, recorded with a fetch date. A repeated query within the cache window makes no billed call, verified by comparing the call count. Locale and device parameters change the results as expected. Cost per query is recorded. Do not build yet: No page fetching, extraction or analysis. ### Phase 2 · Fetching and extracting the ranking pages Build: Fetch each ranking page politely with a timeout and byte cap, extract the article text and the heading hierarchy, and record failures. Respect robots and identify yourself. Pages you cannot fetch are excluded rather than guessed at. Trap: Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. Done when: Ranking pages are fetched with their heading structure preserved. A page that cannot be fetched is excluded and counted rather than treated as empty. The brief reports how many of the results were successfully analysed. Fetching is rate limited and identifies itself. Do not build yet: No analysis or brief yet. ### Phase 3 · Subtopics, which are more useful than terms Build: Cluster the heading structures across ranking pages to find subtopics most of them address. Report each subtopic with how many pages cover it and example headings. Derive a word count range from the analysed pages rather than a single target. Trap: A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. Done when: A brief lists subtopics with the number of ranking pages covering each. Word count is reported as a range with a median rather than a single target. A subtopic covered by one page of ten is reported as such rather than presented as required. Example headings are shown so a subtopic is understandable. Do not build yet: No scoring or outcome tracking. ### Phase 4 · Scoring a draft, honestly labelled Build: Score a draft on subtopic coverage rather than term density: which subtopics it addresses and which it omits. Show a term comparison as secondary information. Label the score as a checklist and refuse to present it as a prediction of ranking. Trap: A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. Done when: A draft's score is presented as covered and uncovered subtopics rather than a single number to maximise. Term comparison is available and clearly secondary. No interface text implies the score predicts a position. Adding a paragraph covering a missing subtopic changes the checklist as expected. Do not build yet: No outcome tracking yet, which is the honest part. ### Phase 5 · What actually happened Build: Track the position of pages you published for their target queries on a schedule. Compare outcomes against the coverage score at publication. Report the relationship honestly, including when there is none. Trap: Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. Done when: Positions are tracked on a schedule for published pages. A report compares coverage score at publication against position after a period. The report states plainly when the relationship is weak. At least ten published pages are tracked before any conclusion is drawn. Do not build yet: Cost comparison comes last. ### Phase 6 · Cost, and whether this was worth it Build: Total the search data spend per month. Compare against the subscription you replaced and against the outcomes you measured. Write down the decision either way. Trap: Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong. Done when: Monthly search data spend is reported and compared against the subscription price. The comparison includes outcome evidence rather than only cost. A written decision records whether to keep this, with reasoning. Position tracking frequency can be reduced to control spend. Do not build yet: Nothing after this. ### Out of scope (and why) - Reliable search result data of your own. It comes from a paid provider and that cost is the floor of this build. - The scoring model behind the commercial product, which is tuned against far more data than you have and is still a proxy. - The editor experience, which is much of what people pay for and which a checklist does not replace. - Any claim that following a brief causes a page to rank. Correlation with pages that already rank is not a mechanism. ### README must contain - That search data comes from a paid provider and scraping is neither permitted nor reliable - That the score is a coverage checklist rather than a ranking prediction - That word count is reported as a range from analysed pages rather than a target - That pages which could not be fetched are excluded and counted - The measured relationship between coverage score and actual outcomes ===== ARCHITECTURE.md ===== # Architecture · Surfer SEO ## Stack | Part | Choice | Why | | --- | --- | --- | | Data | A paid search results API | the only supportable source; scraping is against terms and returns unreliable data | | Runtime | Python 3.12 with SQLite | fetching, extracting and comparing text, at a volume one person generates | | Extraction | trafilatura for article text and a parser for heading structure | the structure matters as much as the words and is easier to compare | | Analysis | Term frequency comparison plus subtopic coverage | explainable arithmetic beats an opaque score you cannot debug or defend | | Outcomes | Recorded position over time per page you published | the only way to learn whether any of this worked | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Search data | What currently ranks. | Providers are interchangeable behind one interface. Scraping is not an option that belongs behind it. | | Extraction | Text and structure of ranking pages. | Any extractor. Excluding rather than zeroing failures is the correctness property. | | Subtopics | What the ranking pages cover. | Clustering can improve; reporting coverage counts rather than a flat requirement list should not. | | Score | A coverage checklist. | Any scoring, provided it never reads as a ranking prediction. | | Outcomes | What actually happened. | Never remove this. Without it the tool cannot be evaluated and will be used on faith. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `SERP_API_KEY` · required, secret · Your search data provider. Priced per call, so treat it as a spending instrument. - `SERP_CACHE_HOURS` · required · How long a query's results are reused. Rankings do not change hourly and calls cost money. - `FETCH_TIMEOUT_MS` · required · Per-page timeout when reading ranking pages. - `FETCH_USER_AGENT` · required · Identify yourself honestly. You are reading other people's sites. - `TRACKING_INTERVAL_DAYS` · required · How often positions are checked. This is the cost that grows quietly. - `MONTHLY_SPEND_CAP_USD` · required · Stop making billed calls at this figure. Mirror it as a hard cap at the provider. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Surfer SEO product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A paid search results API, Python 3.12 with SQLite, trafilatura for article text and a parser for heading structure, Term frequency comparison plus subtopic coverage, Recorded position over time per page you published. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. - Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong. ===== MILESTONES.md ===== # Delivery milestones · Surfer SEO Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Search data from a supportable source You can fetch what currently ranks for a query, repeatedly, without being blocked. ### Steps 1. Connect a search data provider With locale and device. Read the pricing before building, because this is the recurring cost. Files: `src/serp.py` 2. Cache by query, locale and device These calls are priced individually and rankings do not change hourly. Files: `src/cache.py` 3. Record cost per query So the eventual comparison against the subscription is real rather than a feeling. Files: `src/cost.py` ### Done when - [ ] A query returns ranked results with a fetch date - [ ] A cached query makes no billed call - [ ] Locale and device change results - [ ] Cost per query is recorded ### Watch out - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. ## M2 · Fetching and extracting the ranking pages You have the text and structure of the pages you are comparing against. ### Steps 1. Fetch politely with limits Timeout, byte cap, rate limit, honest user agent. You are reading someone's site. Files: `src/fetch.py` 2. Extract text and headings The hierarchy matters as much as the words for a brief. Files: `src/extract.py` 3. Exclude and count failures Never treat a failure as empty. Report the analysed count on the brief. Files: `src/extract.py` ### Done when - [ ] Heading structure is preserved on extraction - [ ] A failed fetch is excluded and counted - [ ] The analysed count appears on the brief - [ ] Fetching is rate limited and identifies itself ### Watch out - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. ## M3 · Subtopics, which are more useful than terms The brief says what the ranking pages all cover, which is the part worth acting on. ### Steps 1. Cluster headings into subtopics Across pages, grouping equivalent headings. Report coverage counts rather than a flat list. Files: `src/subtopics.py` 2. Report ranges, not targets Word count as a range and median from analysed pages only. Files: `src/brief.py` 3. Show the evidence Example headings per subtopic, so the writer can judge relevance rather than obeying. Files: `src/brief.py` ### Done when - [ ] Subtopics list their coverage counts - [ ] Word count is a range with a median - [ ] A rarely covered subtopic is marked as such - [ ] Example headings are shown per subtopic ### Watch out - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. ## M4 · Scoring a draft, honestly labelled A draft gets a coverage checklist rather than a number to maximise. ### Steps 1. Score coverage first Subtopics addressed and omitted. That is the actionable output. Files: `src/score.py` 2. Keep terms secondary Available, labelled, not the headline. Term matching is a weak proxy and should look like one. Files: `src/score.py` 3. Refuse predictive language Nothing in the interface should imply a ranking outcome. Check the strings deliberately. Files: `src/ui.py` ### Done when - [ ] The score is presented as covered and uncovered subtopics - [ ] Term comparison is clearly secondary - [ ] No interface text implies a ranking prediction - [ ] Covering a missing subtopic updates the checklist ### Watch out - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. ## M5 · What actually happened You record where your published pages ended up, so the tool can be judged. ### Steps 1. Track positions over time On a schedule, through the same paid API, with the cost accounted for. Files: `src/tracking.py` 2. Compare score against outcome At a sample size where the comparison means something. Ten pages is a minimum, not a good number. Files: `src/evaluate.py` 3. Report weak relationships plainly It is the most likely finding and the most useful one. Files: `src/evaluate.py` ### Done when - [ ] Positions are tracked on a schedule - [ ] Score and outcome are compared over at least ten pages - [ ] A weak relationship is reported plainly - [ ] Tracking cost is included in the running total ### Watch out - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. ## M6 · Cost, and whether this was worth it (production only) You can say what this costs and whether to keep it. ### Steps 1. Total spend monthly Briefs and tracking separately. Tracking is the one that grows quietly. Files: `src/cost.py` 2. Compare cost and outcomes together Cheaper and useless is not a win. Both halves belong in the comparison. Files: `docs/comparison.md` 3. Write the decision down Including the case where you go back. A documented negative result is worth more than a quiet retreat. Files: `docs/comparison.md` ### Done when - [ ] Monthly spend is reported by category - [ ] The comparison includes outcome evidence - [ ] The decision and reasoning are written down - [ ] Tracking frequency is adjustable to control spend ### Watch out - Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong. ===== OPERATIONS.md ===== # Operations · Surfer SEO ## Backup The database holds briefs, extracted pages and outcome history. The outcome history is the irreplaceable part, because it accumulates slowly and is the only evidence about whether this works. ## Restore Restore and check that outcome history is intact. Briefs are regenerable at the cost of API calls; outcome history is not regenerable at all. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Monthly spend against the cap, cache hit rate, and page fetch failure rate. A falling cache hit rate means you are paying for queries you already have. ## Incident checklist If spend spikes, the cap should have stopped it and the provider cap is the backstop. If briefs suddenly get worse, check the fetch failure rate before the analysis, since a brief built from three pages of ten is a different artefact. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A cached query is confirmed to make no billed call - [ ] Failed page fetches are confirmed excluded and counted - [ ] No interface text implies a ranking prediction - [ ] Outcome tracking is running over at least ten pages - [ ] The monthly spend cap is configured at the provider as well as locally ## Launch constraint Do not market omitted Surfer SEO capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your search data provider. Priced per call, so treat it as a spending instrument. SERP_API_KEY=replace-with-provider-key # Required. How long a query's results are reused. Rankings do not change hourly and calls cost money. SERP_CACHE_HOURS=168 # Required. Per-page timeout when reading ranking pages. FETCH_TIMEOUT_MS=8000 # Required. Identify yourself honestly. You are reading other people's sites. FETCH_USER_AGENT=ContentBrief/1.0 (contact: you@example.com) # Required. How often positions are checked. This is the cost that grows quietly. TRACKING_INTERVAL_DAYS=7 # Required. Stop making billed calls at this figure. Mirror it as a hard cap at the provider. MONTHLY_SPEND_CAP_USD=30
README.md
# Surfer SEO · indie build Fetch the pages currently ranking for a query through a search data provider, extract their structure and vocabulary, produce a brief covering the subtopics they all address, score a draft against that coverage as a checklist, and record what actually happened to the page afterwards so the score can be judged rather than trusted. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Data | A paid search results API | the only supportable source; scraping is against terms and returns unreliable data | | Runtime | Python 3.12 with SQLite | fetching, extracting and comparing text, at a volume one person generates | | Extraction | trafilatura for article text and a parser for heading structure | the structure matters as much as the words and is easier to compare | | Analysis | Term frequency comparison plus subtopic coverage | explainable arithmetic beats an opaque score you cannot debug or defend | | Outcomes | Recorded position over time per page you published | the only way to learn whether any of this worked | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the fetching and analysis. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **A paid search results API** · priced per call; a few hundred queries a month is typically tens of US dollars and position tracking multiplies it - Why: This is the input and it has no free substitute. Scraping is against terms, unreliable and blocked, which makes every downstream number meaningless. - Get it: Choose a provider, read its per-call pricing carefully, and start on the smallest plan. Estimate your monthly calls from briefs plus position tracking before committing. - Verify: A query returns ranked results with urls - [ ] **Ten published pages of your own with known outcomes** · free - Why: Phase 5 evaluates whether any of this works, and it needs pages whose positions you can actually track. Without them the tool cannot be judged. - Get it: Use pages you have already published. If you have none, this build cannot be evaluated yet and you should know that before spending on search data. - Verify: Ten pages with target queries are recorded - [ ] **A willingness to accept a negative result** · free - Why: The most likely finding is that coverage score and outcome are weakly related. A tool built to confirm a belief is worse than no tool. - Get it: Decide before Phase 5 that you will act on the finding either way, and write that down. - Verify: The commitment is recorded before outcomes are measured - [ ] **A monthly budget for search data** · the cap you choose - Why: Per-call pricing plus weekly position tracking grows faster than expected, and the point of leaving a subscription is spending less. - Get it: Set a figure and a hard cap at the provider before the first bulk run. - Verify: A provider spend cap is configured ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Search result data of your own. - The commercial scoring model. - The editor experience. - Any causal claim about ranking. - SERP data reliability - scoring model - editor UX - AI article workflows - integrations - audits - support If one of those is essential to you, that is the reason to keep paying for Surfer SEO, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Surfer SEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a content brief built from what currently ranks, with an honest view of what the score is worth to replace Surfer SEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two things to settle before building. Search result data has to come from a paid provider: scraping a search engine's result pages breaks its terms and the result is unreliable and blocked. Budget for an API or accept that this build has no data. And be clear-eyed about the score. Matching the term frequencies of pages that currently rank is a proxy, not a cause: those pages may rank because of links, domain history or brand, none of which your word count reproduces. A score is useful as a checklist for things you forgot to cover and actively harmful as a target to maximise, because writing to it produces text nobody wants to read. Build it as the first and label it clearly. ### Stack (fixed, do not substitute) - Data: A paid search results API · the only supportable source; scraping is against terms and returns unreliable data - Runtime: Python 3.12 with SQLite · fetching, extracting and comparing text, at a volume one person generates - Extraction: trafilatura for article text and a parser for heading structure · the structure matters as much as the words and is easier to compare - Analysis: Term frequency comparison plus subtopic coverage · explainable arithmetic beats an opaque score you cannot debug or defend - Outcomes: Recorded position over time per page you published · the only way to learn whether any of this worked ### Data model (create this before Phase 1) - queries(id, text, locale, device, checked_at) - serp_results(query_id, position, url, title, fetched_at) - pages(id, url, fetched_at, word_count, headings_json, text, status) - terms(page_id, term, count) · after normalisation, for comparison rather than for display - subtopics(query_id, name, present_in_count, example_headings_json) · what most ranking pages cover - briefs(id, query_id, created_at, subtopics_json, target_words, notes) - drafts(id, brief_id, text, scored_at, coverage_json, score) - outcomes(page_url, query_id, checked_on, position) · what happened after publishing, which is the only real evidence ### Phase 1 · Search data from a supportable source Build: Connect a search results API with locale and device parameters. Store results with the date fetched, because rankings change and a brief built last month was built against a different page set. Cache aggressively, since these queries are priced per call. Trap: Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. Done when: A query returns ranked results with urls, recorded with a fetch date. A repeated query within the cache window makes no billed call, verified by comparing the call count. Locale and device parameters change the results as expected. Cost per query is recorded. Do not build yet: No page fetching, extraction or analysis. ### Phase 2 · Fetching and extracting the ranking pages Build: Fetch each ranking page politely with a timeout and byte cap, extract the article text and the heading hierarchy, and record failures. Respect robots and identify yourself. Pages you cannot fetch are excluded rather than guessed at. Trap: Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. Done when: Ranking pages are fetched with their heading structure preserved. A page that cannot be fetched is excluded and counted rather than treated as empty. The brief reports how many of the results were successfully analysed. Fetching is rate limited and identifies itself. Do not build yet: No analysis or brief yet. ### Phase 3 · Subtopics, which are more useful than terms Build: Cluster the heading structures across ranking pages to find subtopics most of them address. Report each subtopic with how many pages cover it and example headings. Derive a word count range from the analysed pages rather than a single target. Trap: A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. Done when: A brief lists subtopics with the number of ranking pages covering each. Word count is reported as a range with a median rather than a single target. A subtopic covered by one page of ten is reported as such rather than presented as required. Example headings are shown so a subtopic is understandable. Do not build yet: No scoring or outcome tracking. ### Phase 4 · Scoring a draft, honestly labelled Build: Score a draft on subtopic coverage rather than term density: which subtopics it addresses and which it omits. Show a term comparison as secondary information. Label the score as a checklist and refuse to present it as a prediction of ranking. Trap: A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. Done when: A draft's score is presented as covered and uncovered subtopics rather than a single number to maximise. Term comparison is available and clearly secondary. No interface text implies the score predicts a position. Adding a paragraph covering a missing subtopic changes the checklist as expected. Do not build yet: No outcome tracking yet, which is the honest part. ### Phase 5 · What actually happened Build: Track the position of pages you published for their target queries on a schedule. Compare outcomes against the coverage score at publication. Report the relationship honestly, including when there is none. Trap: Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. Done when: Positions are tracked on a schedule for published pages. A report compares coverage score at publication against position after a period. The report states plainly when the relationship is weak. At least ten published pages are tracked before any conclusion is drawn. Do not build yet: Cost comparison comes last. ### Phase 6 · Cost, and whether this was worth it Build: Total the search data spend per month. Compare against the subscription you replaced and against the outcomes you measured. Write down the decision either way. Trap: Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong. Done when: Monthly search data spend is reported and compared against the subscription price. The comparison includes outcome evidence rather than only cost. A written decision records whether to keep this, with reasoning. Position tracking frequency can be reduced to control spend. Do not build yet: Nothing after this. ### Out of scope (and why) - Reliable search result data of your own. It comes from a paid provider and that cost is the floor of this build. - The scoring model behind the commercial product, which is tuned against far more data than you have and is still a proxy. - The editor experience, which is much of what people pay for and which a checklist does not replace. - Any claim that following a brief causes a page to rank. Correlation with pages that already rank is not a mechanism. ### README must contain - That search data comes from a paid provider and scraping is neither permitted nor reliable - That the score is a coverage checklist rather than a ranking prediction - That word count is reported as a range from analysed pages rather than a target - That pages which could not be fetched are excluded and counted - The measured relationship between coverage score and actual outcomes
AGENTS.md
# Agent instructions · Surfer SEO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A paid search results API, Python 3.12 with SQLite, trafilatura for article text and a parser for heading structure, Term frequency comparison plus subtopic coverage, Recorded position over time per page you published. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding.
BUILD_PLAN.md
# Build plan · Surfer SEO Fetch the pages currently ranking for a query through a search data provider, extract their structure and vocabulary, produce a brief covering the subtopics they all address, score a draft against that coverage as a checklist, and record what actually happened to the page afterwards so the score can be judged rather than trusted. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Search data from a supportable source You can fetch what currently ranks for a query, repeatedly, without being blocked. ### Steps 1. Connect a search data provider With locale and device. Read the pricing before building, because this is the recurring cost. Files: `src/serp.py` 2. Cache by query, locale and device These calls are priced individually and rankings do not change hourly. Files: `src/cache.py` 3. Record cost per query So the eventual comparison against the subscription is real rather than a feeling. Files: `src/cost.py` ### Done when - [ ] A query returns ranked results with a fetch date - [ ] A cached query makes no billed call - [ ] Locale and device change results - [ ] Cost per query is recorded ### Watch out - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. ## Phase 2 · Fetching and extracting the ranking pages You have the text and structure of the pages you are comparing against. ### Steps 1. Fetch politely with limits Timeout, byte cap, rate limit, honest user agent. You are reading someone's site. Files: `src/fetch.py` 2. Extract text and headings The hierarchy matters as much as the words for a brief. Files: `src/extract.py` 3. Exclude and count failures Never treat a failure as empty. Report the analysed count on the brief. Files: `src/extract.py` ### Done when - [ ] Heading structure is preserved on extraction - [ ] A failed fetch is excluded and counted - [ ] The analysed count appears on the brief - [ ] Fetching is rate limited and identifies itself ### Watch out - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. ## Phase 3 · Subtopics, which are more useful than terms The brief says what the ranking pages all cover, which is the part worth acting on. ### Steps 1. Cluster headings into subtopics Across pages, grouping equivalent headings. Report coverage counts rather than a flat list. Files: `src/subtopics.py` 2. Report ranges, not targets Word count as a range and median from analysed pages only. Files: `src/brief.py` 3. Show the evidence Example headings per subtopic, so the writer can judge relevance rather than obeying. Files: `src/brief.py` ### Done when - [ ] Subtopics list their coverage counts - [ ] Word count is a range with a median - [ ] A rarely covered subtopic is marked as such - [ ] Example headings are shown per subtopic ### Watch out - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. ## Phase 4 · Scoring a draft, honestly labelled A draft gets a coverage checklist rather than a number to maximise. ### Steps 1. Score coverage first Subtopics addressed and omitted. That is the actionable output. Files: `src/score.py` 2. Keep terms secondary Available, labelled, not the headline. Term matching is a weak proxy and should look like one. Files: `src/score.py` 3. Refuse predictive language Nothing in the interface should imply a ranking outcome. Check the strings deliberately. Files: `src/ui.py` ### Done when - [ ] The score is presented as covered and uncovered subtopics - [ ] Term comparison is clearly secondary - [ ] No interface text implies a ranking prediction - [ ] Covering a missing subtopic updates the checklist ### Watch out - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. ## Phase 5 · What actually happened You record where your published pages ended up, so the tool can be judged. ### Steps 1. Track positions over time On a schedule, through the same paid API, with the cost accounted for. Files: `src/tracking.py` 2. Compare score against outcome At a sample size where the comparison means something. Ten pages is a minimum, not a good number. Files: `src/evaluate.py` 3. Report weak relationships plainly It is the most likely finding and the most useful one. Files: `src/evaluate.py` ### Done when - [ ] Positions are tracked on a schedule - [ ] Score and outcome are compared over at least ten pages - [ ] A weak relationship is reported plainly - [ ] Tracking cost is included in the running total ### Watch out - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. ## Not in this build - Search result data of your own. - The commercial scoring model. - The editor experience. - Any causal claim about ranking. ## After v1, if you want it - Tracking competitor pages for the same queries, which explains movement better than your own page alone - A content refresh queue for pages whose position has decayed - Comparing briefs against outcomes by subtopic, which is the analysis that would actually teach you something
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your search data provider. Priced per call, so treat it as a spending instrument. SERP_API_KEY=replace-with-provider-key # Required. How long a query's results are reused. Rankings do not change hourly and calls cost money. SERP_CACHE_HOURS=168 # Required. Per-page timeout when reading ranking pages. FETCH_TIMEOUT_MS=8000 # Required. Identify yourself honestly. You are reading other people's sites. FETCH_USER_AGENT=ContentBrief/1.0 (contact: you@example.com) # Required. How often positions are checked. This is the cost that grows quietly. TRACKING_INTERVAL_DAYS=7 # Required. Stop making billed calls at this figure. Mirror it as a hard cap at the provider. MONTHLY_SPEND_CAP_USD=30
PRODUCT.md
# Surfer SEO · product brief ## Problem A SERP-scrape-to-content-brief tool is buildable, but reliable SERP data, NLP scoring, editor workflow, AI articles, audits, and integrations create ongoing work. ## Product outcome A brief built from pages that currently rank, presented as coverage you can act on rather than a number to chase, with outcomes tracked so you can tell whether any of it helped. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - search/SERP API - web scraper/parser - LLM API - content editor - scoring logic - hosted backend ## Explicit non-goals for v1 - Search result data of your own. - The commercial scoring model. - The editor experience. - Any causal claim about ranking. - SERP data reliability - scoring model - editor UX - AI article workflows - integrations - audits - support ## Success criteria - A cached query is confirmed to make no billed call - Failed page fetches are confirmed excluded and counted - No interface text implies a ranking prediction - Outcome tracking is running over at least ten pages - The monthly spend cap is configured at the provider as well as locally
BRIEF.md
# Build brief · Surfer SEO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a content brief built from what currently ranks, with an honest view of what the score is worth to replace Surfer SEO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Two things to settle before building. Search result data has to come from a paid provider: scraping a search engine's result pages breaks its terms and the result is unreliable and blocked. Budget for an API or accept that this build has no data. And be clear-eyed about the score. Matching the term frequencies of pages that currently rank is a proxy, not a cause: those pages may rank because of links, domain history or brand, none of which your word count reproduces. A score is useful as a checklist for things you forgot to cover and actively harmful as a target to maximise, because writing to it produces text nobody wants to read. Build it as the first and label it clearly. ### Stack (fixed, do not substitute) - Data: A paid search results API · the only supportable source; scraping is against terms and returns unreliable data - Runtime: Python 3.12 with SQLite · fetching, extracting and comparing text, at a volume one person generates - Extraction: trafilatura for article text and a parser for heading structure · the structure matters as much as the words and is easier to compare - Analysis: Term frequency comparison plus subtopic coverage · explainable arithmetic beats an opaque score you cannot debug or defend - Outcomes: Recorded position over time per page you published · the only way to learn whether any of this worked ### Data model (create this before Phase 1) - queries(id, text, locale, device, checked_at) - serp_results(query_id, position, url, title, fetched_at) - pages(id, url, fetched_at, word_count, headings_json, text, status) - terms(page_id, term, count) · after normalisation, for comparison rather than for display - subtopics(query_id, name, present_in_count, example_headings_json) · what most ranking pages cover - briefs(id, query_id, created_at, subtopics_json, target_words, notes) - drafts(id, brief_id, text, scored_at, coverage_json, score) - outcomes(page_url, query_id, checked_on, position) · what happened after publishing, which is the only real evidence ### Phase 1 · Search data from a supportable source Build: Connect a search results API with locale and device parameters. Store results with the date fetched, because rankings change and a brief built last month was built against a different page set. Cache aggressively, since these queries are priced per call. Trap: Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. Done when: A query returns ranked results with urls, recorded with a fetch date. A repeated query within the cache window makes no billed call, verified by comparing the call count. Locale and device parameters change the results as expected. Cost per query is recorded. Do not build yet: No page fetching, extraction or analysis. ### Phase 2 · Fetching and extracting the ranking pages Build: Fetch each ranking page politely with a timeout and byte cap, extract the article text and the heading hierarchy, and record failures. Respect robots and identify yourself. Pages you cannot fetch are excluded rather than guessed at. Trap: Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. Done when: Ranking pages are fetched with their heading structure preserved. A page that cannot be fetched is excluded and counted rather than treated as empty. The brief reports how many of the results were successfully analysed. Fetching is rate limited and identifies itself. Do not build yet: No analysis or brief yet. ### Phase 3 · Subtopics, which are more useful than terms Build: Cluster the heading structures across ranking pages to find subtopics most of them address. Report each subtopic with how many pages cover it and example headings. Derive a word count range from the analysed pages rather than a single target. Trap: A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. Done when: A brief lists subtopics with the number of ranking pages covering each. Word count is reported as a range with a median rather than a single target. A subtopic covered by one page of ten is reported as such rather than presented as required. Example headings are shown so a subtopic is understandable. Do not build yet: No scoring or outcome tracking. ### Phase 4 · Scoring a draft, honestly labelled Build: Score a draft on subtopic coverage rather than term density: which subtopics it addresses and which it omits. Show a term comparison as secondary information. Label the score as a checklist and refuse to present it as a prediction of ranking. Trap: A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. Done when: A draft's score is presented as covered and uncovered subtopics rather than a single number to maximise. Term comparison is available and clearly secondary. No interface text implies the score predicts a position. Adding a paragraph covering a missing subtopic changes the checklist as expected. Do not build yet: No outcome tracking yet, which is the honest part. ### Phase 5 · What actually happened Build: Track the position of pages you published for their target queries on a schedule. Compare outcomes against the coverage score at publication. Report the relationship honestly, including when there is none. Trap: Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. Done when: Positions are tracked on a schedule for published pages. A report compares coverage score at publication against position after a period. The report states plainly when the relationship is weak. At least ten published pages are tracked before any conclusion is drawn. Do not build yet: Cost comparison comes last. ### Phase 6 · Cost, and whether this was worth it Build: Total the search data spend per month. Compare against the subscription you replaced and against the outcomes you measured. Write down the decision either way. Trap: Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong. Done when: Monthly search data spend is reported and compared against the subscription price. The comparison includes outcome evidence rather than only cost. A written decision records whether to keep this, with reasoning. Position tracking frequency can be reduced to control spend. Do not build yet: Nothing after this. ### Out of scope (and why) - Reliable search result data of your own. It comes from a paid provider and that cost is the floor of this build. - The scoring model behind the commercial product, which is tuned against far more data than you have and is still a proxy. - The editor experience, which is much of what people pay for and which a checklist does not replace. - Any claim that following a brief causes a page to rank. Correlation with pages that already rank is not a mechanism. ### README must contain - That search data comes from a paid provider and scraping is neither permitted nor reliable - That the score is a coverage checklist rather than a ranking prediction - That word count is reported as a range from analysed pages rather than a target - That pages which could not be fetched are excluded and counted - The measured relationship between coverage score and actual outcomes
ARCHITECTURE.md
# Architecture · Surfer SEO ## Stack | Part | Choice | Why | | --- | --- | --- | | Data | A paid search results API | the only supportable source; scraping is against terms and returns unreliable data | | Runtime | Python 3.12 with SQLite | fetching, extracting and comparing text, at a volume one person generates | | Extraction | trafilatura for article text and a parser for heading structure | the structure matters as much as the words and is easier to compare | | Analysis | Term frequency comparison plus subtopic coverage | explainable arithmetic beats an opaque score you cannot debug or defend | | Outcomes | Recorded position over time per page you published | the only way to learn whether any of this worked | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Search data | What currently ranks. | Providers are interchangeable behind one interface. Scraping is not an option that belongs behind it. | | Extraction | Text and structure of ranking pages. | Any extractor. Excluding rather than zeroing failures is the correctness property. | | Subtopics | What the ranking pages cover. | Clustering can improve; reporting coverage counts rather than a flat requirement list should not. | | Score | A coverage checklist. | Any scoring, provided it never reads as a ranking prediction. | | Outcomes | What actually happened. | Never remove this. Without it the tool cannot be evaluated and will be used on faith. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `SERP_API_KEY` · required, secret · Your search data provider. Priced per call, so treat it as a spending instrument. - `SERP_CACHE_HOURS` · required · How long a query's results are reused. Rankings do not change hourly and calls cost money. - `FETCH_TIMEOUT_MS` · required · Per-page timeout when reading ranking pages. - `FETCH_USER_AGENT` · required · Identify yourself honestly. You are reading other people's sites. - `TRACKING_INTERVAL_DAYS` · required · How often positions are checked. This is the cost that grows quietly. - `MONTHLY_SPEND_CAP_USD` · required · Stop making billed calls at this figure. Mirror it as a hard cap at the provider. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Surfer SEO product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A paid search results API, Python 3.12 with SQLite, trafilatura for article text and a parser for heading structure, Term frequency comparison plus subtopic coverage, Recorded position over time per page you published. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. - Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong.
MILESTONES.md
# Delivery milestones · Surfer SEO Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Search data from a supportable source You can fetch what currently ranks for a query, repeatedly, without being blocked. ### Steps 1. Connect a search data provider With locale and device. Read the pricing before building, because this is the recurring cost. Files: `src/serp.py` 2. Cache by query, locale and device These calls are priced individually and rankings do not change hourly. Files: `src/cache.py` 3. Record cost per query So the eventual comparison against the subscription is real rather than a feeling. Files: `src/cost.py` ### Done when - [ ] A query returns ranked results with a fetch date - [ ] A cached query makes no billed call - [ ] Locale and device change results - [ ] Cost per query is recorded ### Watch out - Scraping search result pages directly is against the terms of every major engine, returns different results for automated clients, and gets blocked. It also produces data you cannot rely on, which makes every downstream analysis meaningless. Pay for the API or accept that the build has no input. ## M2 · Fetching and extracting the ranking pages You have the text and structure of the pages you are comparing against. ### Steps 1. Fetch politely with limits Timeout, byte cap, rate limit, honest user agent. You are reading someone's site. Files: `src/fetch.py` 2. Extract text and headings The hierarchy matters as much as the words for a brief. Files: `src/extract.py` 3. Exclude and count failures Never treat a failure as empty. Report the analysed count on the brief. Files: `src/extract.py` ### Done when - [ ] Heading structure is preserved on extraction - [ ] A failed fetch is excluded and counted - [ ] The analysed count appears on the brief - [ ] Fetching is rate limited and identifies itself ### Watch out - Treating a failed fetch as an empty page skews every average downward and makes the target word count nonsense. Exclude failures from the analysis and report how many of the ranking pages you actually managed to read, because a brief built from three of ten pages is a different thing. ## M3 · Subtopics, which are more useful than terms The brief says what the ranking pages all cover, which is the part worth acting on. ### Steps 1. Cluster headings into subtopics Across pages, grouping equivalent headings. Report coverage counts rather than a flat list. Files: `src/subtopics.py` 2. Report ranges, not targets Word count as a range and median from analysed pages only. Files: `src/brief.py` 3. Show the evidence Example headings per subtopic, so the writer can judge relevance rather than obeying. Files: `src/brief.py` ### Done when - [ ] Subtopics list their coverage counts - [ ] Word count is a range with a median - [ ] A rarely covered subtopic is marked as such - [ ] Example headings are shown per subtopic ### Watch out - A single target word count implies a precision that does not exist, and writing to it produces padding. Report the range and the median across the pages you actually analysed, and let the writer decide, since covering the subject well is the thing that correlates with the number rather than the reverse. ## M4 · Scoring a draft, honestly labelled A draft gets a coverage checklist rather than a number to maximise. ### Steps 1. Score coverage first Subtopics addressed and omitted. That is the actionable output. Files: `src/score.py` 2. Keep terms secondary Available, labelled, not the headline. Term matching is a weak proxy and should look like one. Files: `src/score.py` 3. Refuse predictive language Nothing in the interface should imply a ranking outcome. Check the strings deliberately. Files: `src/ui.py` ### Done when - [ ] The score is presented as covered and uncovered subtopics - [ ] Term comparison is clearly secondary - [ ] No interface text implies a ranking prediction - [ ] Covering a missing subtopic updates the checklist ### Watch out - A single score invites optimisation, and optimising term frequency produces text that reads as though it was written for a machine, which is the outcome search engines have spent two decades getting better at detecting. Lead with coverage, keep terms secondary, and never phrase the score as a likelihood of ranking. ## M5 · What actually happened You record where your published pages ended up, so the tool can be judged. ### Steps 1. Track positions over time On a schedule, through the same paid API, with the cost accounted for. Files: `src/tracking.py` 2. Compare score against outcome At a sample size where the comparison means something. Ten pages is a minimum, not a good number. Files: `src/evaluate.py` 3. Report weak relationships plainly It is the most likely finding and the most useful one. Files: `src/evaluate.py` ### Done when - [ ] Positions are tracked on a schedule - [ ] Score and outcome are compared over at least ten pages - [ ] A weak relationship is reported plainly - [ ] Tracking cost is included in the running total ### Watch out - Without outcome tracking this tool cannot be evaluated and you will keep using it because it feels productive. Record positions over time, and be prepared for the answer that coverage score and outcome are barely related, because that is a genuinely possible and useful finding. ## M6 · Cost, and whether this was worth it (production only) You can say what this costs and whether to keep it. ### Steps 1. Total spend monthly Briefs and tracking separately. Tracking is the one that grows quietly. Files: `src/cost.py` 2. Compare cost and outcomes together Cheaper and useless is not a win. Both halves belong in the comparison. Files: `docs/comparison.md` 3. Write the decision down Including the case where you go back. A documented negative result is worth more than a quiet retreat. Files: `docs/comparison.md` ### Done when - [ ] Monthly spend is reported by category - [ ] The comparison includes outcome evidence - [ ] The decision and reasoning are written down - [ ] Tracking frequency is adjustable to control spend ### Watch out - Search data pricing is per call and briefs plus position tracking add up faster than people expect, particularly once you track many queries weekly. Total it monthly against the subscription, because the build being free is the assumption that usually turns out wrong.
OPERATIONS.md
# Operations · Surfer SEO ## Backup The database holds briefs, extracted pages and outcome history. The outcome history is the irreplaceable part, because it accumulates slowly and is the only evidence about whether this works. ## Restore Restore and check that outcome history is intact. Briefs are regenerable at the cost of API calls; outcome history is not regenerable at all. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Monthly spend against the cap, cache hit rate, and page fetch failure rate. A falling cache hit rate means you are paying for queries you already have. ## Incident checklist If spend spikes, the cap should have stopped it and the provider cap is the backstop. If briefs suddenly get worse, check the fetch failure rate before the analysis, since a brief built from three pages of ten is a different artefact. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A cached query is confirmed to make no billed call - [ ] Failed page fetches are confirmed excluded and counted - [ ] No interface text implies a ranking prediction - [ ] Outcome tracking is running over at least ten pages - [ ] The monthly spend cap is configured at the provider as well as locally ## Launch constraint Do not market omitted Surfer SEO capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your search data provider. Priced per call, so treat it as a spending instrument. SERP_API_KEY=replace-with-provider-key # Required. How long a query's results are reused. Rankings do not change hourly and calls cost money. SERP_CACHE_HOURS=168 # Required. Per-page timeout when reading ranking pages. FETCH_TIMEOUT_MS=8000 # Required. Identify yourself honestly. You are reading other people's sites. FETCH_USER_AGENT=ContentBrief/1.0 (contact: you@example.com) # Required. How often positions are checked. This is the cost that grows quietly. TRACKING_INTERVAL_DAYS=7 # Required. Stop making billed calls at this figure. Mirror it as a hard cap at the provider. MONTHLY_SPEND_CAP_USD=30
$ choose a build depth, inspect the files, then open the complete pack in your agent
They pay because content optimization needs repeatable data and an editor that writers will use.
xSERP data reliability
xscoring model
xeditor UX
xAI article workflows
xintegrations
xaudits
xsupport
Nothing worth pointing at. That's why the prompt exists.
share on X ↗"I just replaced Surfer SEO ($99/mo) with one prompt"
Surfer SEO pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| discovery | $59/workspace | $49/workspace | 120 documents/year; 10 tracked pages; 1 brand workspace; 1 seat. |
| standard | $119/workspace | $99/workspace | 360 documents/year; 25 AI Search prompts/week; 50 tracked pages; 1 brand workspace; 3 seats. |
| pro | $219/workspace | $182/workspace | 360 documents/year; 50 AI Search prompts/day; 5 brand workspaces; 200 tracked pages; 5 seats. |
| peace of mind | $359/workspace | $299/workspace | Unlimited documents under fair use; 100 AI Search prompts/day; unlimited brand workspaces; 500 tracked pages; 10 seats; API access. |
| ai search analytics | $185/workspace | $158/workspace | 100 prompts/day; 5 brand workspaces. |
| enterprise | $999/workspace | — | Starts at $999/month with custom documents, prompts, workspaces, tracked pages, seats, API, security, and support. |
free tierno standing free tier; a trial/start-free path is offered, but public duration and numeric caps were not verified
billingmonthly + annual
hidden costsThe 'unlimited' document allowance is subject to fair use. AI Search Analytics capacity is sold separately or through dynamic prompt/workspace configurations.
verified 2026-08-14 · source ↗
Vibecode Surfer SEO
Kinda. The core of Surfer SEO is buildable in a weekend with the prompt on this page, but there are real gaps: SERP data reliability, scoring model. Read the honest list above before committing.
How much does Surfer SEO cost?
Surfer SEO costs about $99/month (Essential/Standard, checked 2026-07-30), which is $1188 per year.
What do I lose by replacing Surfer SEO?
Honestly: SERP data reliability; scoring model; editor UX; AI article workflows; integrations; audits; support. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Surfer SEO?
No mature open-source alternative worth pointing at, which is exactly why the one-shot prompt on this page exists.