Vibecode Perplexity
track this build5 phases, 15 steps, beginner friendly0%KINDA · weekend project
A retrieval-plus-LLM answer engine is buildable, but Perplexity's search stack, source ranking, UX, mobile/browser surfaces, and model access make full parity hard.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Perplexity. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Perplexity · indie build A self-hosted answer engine: a SearXNG instance you control providing results, a fetcher that extracts readable text under strict caps, passage ranking, an answer step whose citations are validated against the passages that were actually retrieved, and threads with follow-up questions that carry context. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Search | SearXNG, self-hosted, with the JSON output format enabled | a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours | | Runtime | Python 3.12 with FastAPI | the work is orchestrating slow network calls, which is what async is for | | Extraction | trafilatura | purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper | | Ranking | BM25 over sentence-window passages, with an optional cross-encoder reranker | start with the method that needs no model and no GPU, and measure before adding one | | Answering | Any chat model through its API, or a local model through Ollama | the answer step is the only part that needs a large model, and it is a swappable adapter | | Storage | SQLite with full text search | threads, cached pages and passages are all rows, and one file is the whole backup | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Docker with Compose v2** · free for personal use; check Docker Desktop's licence if this is for a company - Why: Runs SearXNG. Installing SearXNG without containers is possible and is a worse afternoon. - Get it: Install Docker Desktop, or Docker Engine plus the compose plugin on Linux. - Verify: docker compose version prints v2 or higher - [ ] **Python 3.12 or newer** · free - Why: Runs the pipeline. - Get it: python.org or your package manager, then create a virtual environment in the project directory. - Verify: python3 --version prints 3.12 or higher - [ ] **A model API key, or Ollama for a local model** · pay per token; a short answer over five passages is typically well under a cent, and long threads add up faster than you expect - Why: The answer step needs a model. Everything before it works without one, which is useful while you are building. - Get it: Create an account with a provider and generate a key in their console, and set a spend limit at the same time. Or install Ollama and pull a model if you would rather pay nothing and accept weaker answers. - Verify: A single test completion returns text and appears in the provider usage dashboard - [ ] **Your own SearXNG instance, not a public one** · free, but you carry the upstream engines' rate limits yourself - Why: Public instances block automated traffic and rate-limit, so building against one produces a system that fails as soon as it works. - Get it: Run the official container locally, then enable json in the search formats of settings.yml and restart. - Verify: curl with format=json returns JSON rather than a 403 - [ ] **Ten queries with known correct sources** · free - Why: Ranking changes feel better and are often worse. Without a fixed set you are tuning on vibes. - Get it: Write them yourself from questions you actually asked this week, and record the url that genuinely answers each. An hour of work that pays for itself in Phase 3. ## Quick start ```sh docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json' ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A web index of your own, which is the moat and stays the moat. - Search quality tuned by a team with query logs at scale. - Model routing and the credit allowance a subscription includes. - Polished mobile and browser applications. - search quality - source ranking - model routing - mobile/browser apps - publisher integrations - speed If one of those is essential to you, that is the reason to keep paying for Perplexity, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Perplexity The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal answer engine that searches the web, reads the pages, and answers with citations you can check to replace Perplexity. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not build a web index. Crawling and ranking the web is the moat and it is not reproducible at any budget you have. You are building the layer above search: retrieve, read, rank passages, answer, and cite honestly. If you would rather deploy than build, the open-source project that does this was called Perplexica and was renamed Vane in March 2026. Deploying it is a legitimate afternoon and this prompt is the alternative for people who want the pipeline to be theirs. ### Stack (fixed, do not substitute) - Search: SearXNG, self-hosted, with the JSON output format enabled · a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours - Runtime: Python 3.12 with FastAPI · the work is orchestrating slow network calls, which is what async is for - Extraction: trafilatura · purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper - Ranking: BM25 over sentence-window passages, with an optional cross-encoder reranker · start with the method that needs no model and no GPU, and measure before adding one - Answering: Any chat model through its API, or a local model through Ollama · the answer step is the only part that needs a large model, and it is a swappable adapter - Storage: SQLite with full text search · threads, cached pages and passages are all rows, and one file is the whole backup ### Data model (create this before Phase 1) - threads(id, title, created_at) - queries(id, thread_id, text, rewritten_text, asked_at) - results(id, query_id, url, title, snippet, engine, rank) - documents(id, url, fetched_at, status, content_hash, text, byte_count) · the cache, keyed by url - passages(id, document_id, ordinal, text, score) · what the answer is allowed to cite - answers(id, query_id, text, model, citations_json, validated) - Nothing may be cited that is not a row in passages for that query. That invariant is the whole point of Phase 4. ### Phase 1 · Search you control Build: Run SearXNG in Docker and enable the JSON output format in settings.yml, which is off by default. Add a client that queries it, normalises results, and caches identical queries for a short window so development does not hammer upstream engines. Trap: SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. Trap: Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. Done when: A query returns at least ten results as JSON with url, title and engine. The same query within the cache window is served from cache, verified by an unchanged upstream request count. A query returning nothing produces an empty result set and a clear message, not a stack trace. Do not build yet: No fetching, no ranking, no model calls. Search results only. ### Phase 2 · Fetch and extract, with limits that hold Build: Fetch the top results concurrently with a per-request timeout, a total time budget and a byte cap. Extract readable text with trafilatura. Respect robots.txt. Cache by url with a content hash so a re-fetch of unchanged content costs nothing. Trap: A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. Done when: Fetching eight pages where two never respond completes inside the total time budget and returns the six that worked. A deliberately large file is aborted at the cap rather than fully downloaded, verified by the bytes actually read. A page disallowed by robots.txt is skipped with a recorded reason. Do not build yet: No ranking and no answering yet. ### Phase 3 · Passages and ranking Build: Chunk extracted text into overlapping passages of a few sentences. Score them against the query with BM25 and keep the top handful across all documents, not the top few per document. Record the score so you can inspect why a passage was chosen. Trap: Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. Done when: For a factual query with a known answer, the passage containing that answer is in the top five by score. Passage boundaries never split a sentence. Dropping the best document from the input changes the answer's citations, proving retrieval feeds the answer rather than the model's memory. Do not build yet: No model call yet. ### Phase 4 · Answers whose citations are checked Build: Send the ranked passages, numbered, and require the model to cite by number. Then validate: every cited number must exist in the passages you sent. Reject and retry once on a violation, and if it fails again, return the answer with the invalid citations stripped and a visible note rather than pretending it was clean. Trap: Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. Done when: An answer citing passage seven when only five were supplied fails validation, verified by feeding a crafted response through the validator directly. Every citation in a normal answer resolves to a url that was fetched in that query. A query whose passages contain no answer produces an explicit statement that the sources do not answer it, rather than a fluent guess. Do not build yet: No threads or follow-ups yet. ### Phase 5 · Threads and follow-ups Build: Persist threads. For a follow-up, rewrite the question into a standalone query using the thread so far, and show the rewritten query so the user can see what was actually searched. Reuse cached documents from earlier turns where they are still relevant. Trap: Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. Done when: Asking what about the second one after a list question searches a rewritten standalone query, and that rewritten text is visible in the interface. A follow-up that is already standalone is not rewritten unnecessarily. Reopening a thread shows the same answers and citations without re-running any search. Do not build yet: No hardening or limits yet. ### Phase 6 · Limits, cost and running it for others Build: Add authentication, per-user rate limits and a hard monthly spend cap that stops serving rather than overspending. Log per-query cost. Put it behind TLS with a real certificate, and back up the SQLite file off the machine. Trap: An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view. Done when: An unauthenticated request never reaches a model call, verified by the spend log staying flat under a burst of anonymous requests. Passing the monthly cap returns a clear refusal instead of continuing to spend. A backup of the database has been restored on a second machine and the date recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - A web index. Crawling and ranking the open web is the actual asset, and metasearch borrows it rather than replacing it. - Search result quality tuning at the level a search company does it. Your ranking is BM25 over whatever SearXNG returned, and it will be worse in ways you can measure. - Model routing across many providers by task, which is a real part of what the subscription buys. - Mobile and browser applications, and the polish that makes the hosted product pleasant on a phone. ### README must contain - That SearXNG needs the json format enabled in settings.yml, because nothing works until it is - That every citation is validated against retrieved passages, and what happens when validation fails - The byte cap, the timeout and the total budget, and where to change them - Which model is used for answering, which for rewriting, and what a typical query costs - That the open-source project doing this whole job is Vane, formerly Perplexica, if you would rather deploy than build ===== AGENTS.md ===== # Agent instructions · Perplexity indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: SearXNG, self-hosted, with the JSON output format enabled, Python 3.12 with FastAPI, trafilatura, BM25 over sentence-window passages, with an optional cross-encoder reranker, Any chat model through its API, or a local model through Ollama, SQLite with full text search. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. ===== BUILD_PLAN.md ===== # Build plan · Perplexity A self-hosted answer engine: a SearXNG instance you control providing results, a fetcher that extracts readable text under strict caps, passage ranking, an answer step whose citations are validated against the passages that were actually retrieved, and threads with follow-up questions that carry context. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Search you control A query returns structured results from your own SearXNG, not from scraping someone's results page. ### Steps 1. Run SearXNG and enable JSON Bring it up with Docker, then edit settings.yml so the search formats include json alongside html, and restart. This single line is the most common thing people miss. Files: `searxng/settings.yml` ```sh docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json' ``` 2. Write the search client Query with a timeout, normalise into your results shape, and keep the engine name so you can tell later which upstream is producing rubbish. Files: `search.py` 3. Cache identical queries briefly Key on the normalised query text with a short expiry. This is for your own iteration speed as much as for politeness. Files: `cache.py` ### Done when - [ ] A query returns ten or more JSON results with urls - [ ] A repeated query inside the cache window makes no upstream request - [ ] A nonsense query returns an empty list and a clear message - [ ] Stopping SearXNG produces a handled connection error, not an unhandled exception ### Watch out - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. ## Phase 2 · Fetch and extract, with limits that hold Pages become clean text without a slow or hostile page being able to hang the request. ### Steps 1. Fetch concurrently with a total budget Bounded concurrency, a per-request timeout, and an overall deadline for the batch. Partial results are the expected outcome, not a failure. Files: `fetch.py` 2. Stream and cap the body Read in chunks and stop at the cap. Record the byte count so the check in this phase is measurable rather than asserted. Files: `fetch.py` 3. Extract and cache by content hash trafilatura for the text. Hash the extracted text; if it matches the cached hash, keep the existing passages and skip re-chunking. Files: `extract.py` ```sh pip install trafilatura httpx ``` ### Done when - [ ] Eight fetches with two dead hosts finish inside the total budget - [ ] A large file is aborted at the byte cap, proved by the recorded byte count - [ ] A robots.txt disallow is skipped with a stored reason - [ ] Re-fetching unchanged content does not re-chunk it ### Watch out - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. ## Phase 3 · Passages and ranking The text handed to the model is the part of the page that answers the question, not the whole page. ### Steps 1. Chunk with sentence awareness and overlap Split on sentence boundaries, group into windows of a few sentences with an overlap so an answer spanning a boundary is not lost. Files: `chunk.py` 2. Score with BM25 across all documents One corpus for the whole query, not one per document. Store the score on the passage row. Files: `rank.py` ```sh pip install rank-bm25 ``` 3. Build a small evaluation set Ten queries with a known correct source. Measure whether the right passage lands in the top five, before and after any change to chunking or scoring. Files: `eval/queries.json` ### Done when - [ ] The passage containing a known answer is in the top five for that query - [ ] No passage begins or ends mid-sentence - [ ] Ranking is global, verified by a case where one document contributes several passages and another contributes none - [ ] The evaluation set runs as a script and reports a score you can compare over time ### Watch out - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. ## Phase 4 · Answers whose citations are checked Every citation points at a passage that was actually retrieved, and an answer that invents one is rejected. ### Steps 1. Number the passages and demand numeric citations Pass passages as a numbered list and instruct the model to cite only those numbers. Ask for the answer and the citation list as structured output so parsing is not a regular expression. Files: `answer.py` 2. Validate every citation against the supplied set A pure function from answer plus passage ids to a verdict. Unit test it with a crafted out-of-range citation; do not wait for a model to produce one. Files: `validate.py`, `tests/test_validate.py` 3. Handle the no-answer case explicitly If the top passages do not support an answer, say so. This is the behaviour that makes the tool trustworthy, and it needs to be an instruction and a test, not a hope. Files: `answer.py` ### Done when - [ ] A crafted answer citing a passage that was not supplied is rejected by the validator - [ ] Every citation in a real answer resolves to a fetched url - [ ] A query with no supporting passages produces an explicit refusal to answer - [ ] The validator is unit tested independently of any model call ### Watch out - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. ## Phase 5 · Threads and follow-ups A follow-up question understands the conversation without re-searching everything from scratch. ### Steps 1. Persist threads, queries, answers and citations Store enough that a reopened thread renders from the database with no network access at all. Files: `store.py` 2. Rewrite follow-ups to standalone queries A small model call that takes the thread and the new question and returns one search query. Show it in the interface. Files: `rewrite.py` 3. Reuse the document cache across turns If a follow-up retrieves a url already fetched in this thread and it is still fresh, use the cached text. Files: `cache.py` ### Done when - [ ] A pronoun-laden follow-up searches a sensible standalone query - [ ] The rewritten query is visible to the user - [ ] Reopening a thread renders with no network calls - [ ] A repeated url within a thread is not re-fetched while fresh ### Watch out - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. ## Not in this build - A web index of your own, which is the moat and stays the moat. - Search quality tuned by a team with query logs at scale. - Model routing and the credit allowance a subscription includes. - Polished mobile and browser applications. ## After v1, if you want it - A cross-encoder reranker, added only after the evaluation set says BM25 is the bottleneck - Restricting a thread to a set of domains you trust for a given topic - Scheduled queries that email you when the answer to a standing question changes ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Your own instance. Not a public one. SEARXNG_URL=http://localhost:8080 # Required. Per-request timeout. Slow pages are dropped rather than allowed to hold up the answer. FETCH_TIMEOUT_MS=4000 # Required. The whole retrieval step's deadline. Partial results are expected and correct. FETCH_TOTAL_BUDGET_MS=12000 # Required. Streaming byte cap per page. Enforced during download, not after. FETCH_MAX_BYTES=2000000 # Required. The model used for the answer step. The rewrite step can use something smaller. ANSWER_MODEL=claude-sonnet-5 # Required · secret. Your provider console. Set a hard spend limit there as well as in this application. MODEL_API_KEY=sk-... # Required. The application stops answering at this figure rather than continuing to spend. MONTHLY_CAP_USD=25 # Required. SQLite file holding threads, cache and passages. This is the whole backup. DB_PATH=./data/answers.db
You are building a lean indie version of Perplexity. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Perplexity · indie build A self-hosted answer engine: a SearXNG instance you control providing results, a fetcher that extracts readable text under strict caps, passage ranking, an answer step whose citations are validated against the passages that were actually retrieved, and threads with follow-up questions that carry context. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Search | SearXNG, self-hosted, with the JSON output format enabled | a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours | | Runtime | Python 3.12 with FastAPI | the work is orchestrating slow network calls, which is what async is for | | Extraction | trafilatura | purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper | | Ranking | BM25 over sentence-window passages, with an optional cross-encoder reranker | start with the method that needs no model and no GPU, and measure before adding one | | Answering | Any chat model through its API, or a local model through Ollama | the answer step is the only part that needs a large model, and it is a swappable adapter | | Storage | SQLite with full text search | threads, cached pages and passages are all rows, and one file is the whole backup | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Docker with Compose v2** · free for personal use; check Docker Desktop's licence if this is for a company - Why: Runs SearXNG. Installing SearXNG without containers is possible and is a worse afternoon. - Get it: Install Docker Desktop, or Docker Engine plus the compose plugin on Linux. - Verify: docker compose version prints v2 or higher - [ ] **Python 3.12 or newer** · free - Why: Runs the pipeline. - Get it: python.org or your package manager, then create a virtual environment in the project directory. - Verify: python3 --version prints 3.12 or higher - [ ] **A model API key, or Ollama for a local model** · pay per token; a short answer over five passages is typically well under a cent, and long threads add up faster than you expect - Why: The answer step needs a model. Everything before it works without one, which is useful while you are building. - Get it: Create an account with a provider and generate a key in their console, and set a spend limit at the same time. Or install Ollama and pull a model if you would rather pay nothing and accept weaker answers. - Verify: A single test completion returns text and appears in the provider usage dashboard - [ ] **Your own SearXNG instance, not a public one** · free, but you carry the upstream engines' rate limits yourself - Why: Public instances block automated traffic and rate-limit, so building against one produces a system that fails as soon as it works. - Get it: Run the official container locally, then enable json in the search formats of settings.yml and restart. - Verify: curl with format=json returns JSON rather than a 403 - [ ] **Ten queries with known correct sources** · free - Why: Ranking changes feel better and are often worse. Without a fixed set you are tuning on vibes. - Get it: Write them yourself from questions you actually asked this week, and record the url that genuinely answers each. An hour of work that pays for itself in Phase 3. ## Quick start ```sh docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json' ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A web index of your own, which is the moat and stays the moat. - Search quality tuned by a team with query logs at scale. - Model routing and the credit allowance a subscription includes. - Polished mobile and browser applications. - search quality - source ranking - model routing - mobile/browser apps - publisher integrations - speed If one of those is essential to you, that is the reason to keep paying for Perplexity, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Perplexity The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal answer engine that searches the web, reads the pages, and answers with citations you can check to replace Perplexity. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not build a web index. Crawling and ranking the web is the moat and it is not reproducible at any budget you have. You are building the layer above search: retrieve, read, rank passages, answer, and cite honestly. If you would rather deploy than build, the open-source project that does this was called Perplexica and was renamed Vane in March 2026. Deploying it is a legitimate afternoon and this prompt is the alternative for people who want the pipeline to be theirs. ### Stack (fixed, do not substitute) - Search: SearXNG, self-hosted, with the JSON output format enabled · a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours - Runtime: Python 3.12 with FastAPI · the work is orchestrating slow network calls, which is what async is for - Extraction: trafilatura · purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper - Ranking: BM25 over sentence-window passages, with an optional cross-encoder reranker · start with the method that needs no model and no GPU, and measure before adding one - Answering: Any chat model through its API, or a local model through Ollama · the answer step is the only part that needs a large model, and it is a swappable adapter - Storage: SQLite with full text search · threads, cached pages and passages are all rows, and one file is the whole backup ### Data model (create this before Phase 1) - threads(id, title, created_at) - queries(id, thread_id, text, rewritten_text, asked_at) - results(id, query_id, url, title, snippet, engine, rank) - documents(id, url, fetched_at, status, content_hash, text, byte_count) · the cache, keyed by url - passages(id, document_id, ordinal, text, score) · what the answer is allowed to cite - answers(id, query_id, text, model, citations_json, validated) - Nothing may be cited that is not a row in passages for that query. That invariant is the whole point of Phase 4. ### Phase 1 · Search you control Build: Run SearXNG in Docker and enable the JSON output format in settings.yml, which is off by default. Add a client that queries it, normalises results, and caches identical queries for a short window so development does not hammer upstream engines. Trap: SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. Trap: Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. Done when: A query returns at least ten results as JSON with url, title and engine. The same query within the cache window is served from cache, verified by an unchanged upstream request count. A query returning nothing produces an empty result set and a clear message, not a stack trace. Do not build yet: No fetching, no ranking, no model calls. Search results only. ### Phase 2 · Fetch and extract, with limits that hold Build: Fetch the top results concurrently with a per-request timeout, a total time budget and a byte cap. Extract readable text with trafilatura. Respect robots.txt. Cache by url with a content hash so a re-fetch of unchanged content costs nothing. Trap: A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. Done when: Fetching eight pages where two never respond completes inside the total time budget and returns the six that worked. A deliberately large file is aborted at the cap rather than fully downloaded, verified by the bytes actually read. A page disallowed by robots.txt is skipped with a recorded reason. Do not build yet: No ranking and no answering yet. ### Phase 3 · Passages and ranking Build: Chunk extracted text into overlapping passages of a few sentences. Score them against the query with BM25 and keep the top handful across all documents, not the top few per document. Record the score so you can inspect why a passage was chosen. Trap: Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. Done when: For a factual query with a known answer, the passage containing that answer is in the top five by score. Passage boundaries never split a sentence. Dropping the best document from the input changes the answer's citations, proving retrieval feeds the answer rather than the model's memory. Do not build yet: No model call yet. ### Phase 4 · Answers whose citations are checked Build: Send the ranked passages, numbered, and require the model to cite by number. Then validate: every cited number must exist in the passages you sent. Reject and retry once on a violation, and if it fails again, return the answer with the invalid citations stripped and a visible note rather than pretending it was clean. Trap: Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. Done when: An answer citing passage seven when only five were supplied fails validation, verified by feeding a crafted response through the validator directly. Every citation in a normal answer resolves to a url that was fetched in that query. A query whose passages contain no answer produces an explicit statement that the sources do not answer it, rather than a fluent guess. Do not build yet: No threads or follow-ups yet. ### Phase 5 · Threads and follow-ups Build: Persist threads. For a follow-up, rewrite the question into a standalone query using the thread so far, and show the rewritten query so the user can see what was actually searched. Reuse cached documents from earlier turns where they are still relevant. Trap: Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. Done when: Asking what about the second one after a list question searches a rewritten standalone query, and that rewritten text is visible in the interface. A follow-up that is already standalone is not rewritten unnecessarily. Reopening a thread shows the same answers and citations without re-running any search. Do not build yet: No hardening or limits yet. ### Phase 6 · Limits, cost and running it for others Build: Add authentication, per-user rate limits and a hard monthly spend cap that stops serving rather than overspending. Log per-query cost. Put it behind TLS with a real certificate, and back up the SQLite file off the machine. Trap: An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view. Done when: An unauthenticated request never reaches a model call, verified by the spend log staying flat under a burst of anonymous requests. Passing the monthly cap returns a clear refusal instead of continuing to spend. A backup of the database has been restored on a second machine and the date recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - A web index. Crawling and ranking the open web is the actual asset, and metasearch borrows it rather than replacing it. - Search result quality tuning at the level a search company does it. Your ranking is BM25 over whatever SearXNG returned, and it will be worse in ways you can measure. - Model routing across many providers by task, which is a real part of what the subscription buys. - Mobile and browser applications, and the polish that makes the hosted product pleasant on a phone. ### README must contain - That SearXNG needs the json format enabled in settings.yml, because nothing works until it is - That every citation is validated against retrieved passages, and what happens when validation fails - The byte cap, the timeout and the total budget, and where to change them - Which model is used for answering, which for rewriting, and what a typical query costs - That the open-source project doing this whole job is Vane, formerly Perplexica, if you would rather deploy than build ===== AGENTS.md ===== # Agent instructions · Perplexity indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: SearXNG, self-hosted, with the JSON output format enabled, Python 3.12 with FastAPI, trafilatura, BM25 over sentence-window passages, with an optional cross-encoder reranker, Any chat model through its API, or a local model through Ollama, SQLite with full text search. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. ===== BUILD_PLAN.md ===== # Build plan · Perplexity A self-hosted answer engine: a SearXNG instance you control providing results, a fetcher that extracts readable text under strict caps, passage ranking, an answer step whose citations are validated against the passages that were actually retrieved, and threads with follow-up questions that carry context. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Search you control A query returns structured results from your own SearXNG, not from scraping someone's results page. ### Steps 1. Run SearXNG and enable JSON Bring it up with Docker, then edit settings.yml so the search formats include json alongside html, and restart. This single line is the most common thing people miss. Files: `searxng/settings.yml` ```sh docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json' ``` 2. Write the search client Query with a timeout, normalise into your results shape, and keep the engine name so you can tell later which upstream is producing rubbish. Files: `search.py` 3. Cache identical queries briefly Key on the normalised query text with a short expiry. This is for your own iteration speed as much as for politeness. Files: `cache.py` ### Done when - [ ] A query returns ten or more JSON results with urls - [ ] A repeated query inside the cache window makes no upstream request - [ ] A nonsense query returns an empty list and a clear message - [ ] Stopping SearXNG produces a handled connection error, not an unhandled exception ### Watch out - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. ## Phase 2 · Fetch and extract, with limits that hold Pages become clean text without a slow or hostile page being able to hang the request. ### Steps 1. Fetch concurrently with a total budget Bounded concurrency, a per-request timeout, and an overall deadline for the batch. Partial results are the expected outcome, not a failure. Files: `fetch.py` 2. Stream and cap the body Read in chunks and stop at the cap. Record the byte count so the check in this phase is measurable rather than asserted. Files: `fetch.py` 3. Extract and cache by content hash trafilatura for the text. Hash the extracted text; if it matches the cached hash, keep the existing passages and skip re-chunking. Files: `extract.py` ```sh pip install trafilatura httpx ``` ### Done when - [ ] Eight fetches with two dead hosts finish inside the total budget - [ ] A large file is aborted at the byte cap, proved by the recorded byte count - [ ] A robots.txt disallow is skipped with a stored reason - [ ] Re-fetching unchanged content does not re-chunk it ### Watch out - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. ## Phase 3 · Passages and ranking The text handed to the model is the part of the page that answers the question, not the whole page. ### Steps 1. Chunk with sentence awareness and overlap Split on sentence boundaries, group into windows of a few sentences with an overlap so an answer spanning a boundary is not lost. Files: `chunk.py` 2. Score with BM25 across all documents One corpus for the whole query, not one per document. Store the score on the passage row. Files: `rank.py` ```sh pip install rank-bm25 ``` 3. Build a small evaluation set Ten queries with a known correct source. Measure whether the right passage lands in the top five, before and after any change to chunking or scoring. Files: `eval/queries.json` ### Done when - [ ] The passage containing a known answer is in the top five for that query - [ ] No passage begins or ends mid-sentence - [ ] Ranking is global, verified by a case where one document contributes several passages and another contributes none - [ ] The evaluation set runs as a script and reports a score you can compare over time ### Watch out - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. ## Phase 4 · Answers whose citations are checked Every citation points at a passage that was actually retrieved, and an answer that invents one is rejected. ### Steps 1. Number the passages and demand numeric citations Pass passages as a numbered list and instruct the model to cite only those numbers. Ask for the answer and the citation list as structured output so parsing is not a regular expression. Files: `answer.py` 2. Validate every citation against the supplied set A pure function from answer plus passage ids to a verdict. Unit test it with a crafted out-of-range citation; do not wait for a model to produce one. Files: `validate.py`, `tests/test_validate.py` 3. Handle the no-answer case explicitly If the top passages do not support an answer, say so. This is the behaviour that makes the tool trustworthy, and it needs to be an instruction and a test, not a hope. Files: `answer.py` ### Done when - [ ] A crafted answer citing a passage that was not supplied is rejected by the validator - [ ] Every citation in a real answer resolves to a fetched url - [ ] A query with no supporting passages produces an explicit refusal to answer - [ ] The validator is unit tested independently of any model call ### Watch out - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. ## Phase 5 · Threads and follow-ups A follow-up question understands the conversation without re-searching everything from scratch. ### Steps 1. Persist threads, queries, answers and citations Store enough that a reopened thread renders from the database with no network access at all. Files: `store.py` 2. Rewrite follow-ups to standalone queries A small model call that takes the thread and the new question and returns one search query. Show it in the interface. Files: `rewrite.py` 3. Reuse the document cache across turns If a follow-up retrieves a url already fetched in this thread and it is still fresh, use the cached text. Files: `cache.py` ### Done when - [ ] A pronoun-laden follow-up searches a sensible standalone query - [ ] The rewritten query is visible to the user - [ ] Reopening a thread renders with no network calls - [ ] A repeated url within a thread is not re-fetched while fresh ### Watch out - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. ## Not in this build - A web index of your own, which is the moat and stays the moat. - Search quality tuned by a team with query logs at scale. - Model routing and the credit allowance a subscription includes. - Polished mobile and browser applications. ## After v1, if you want it - A cross-encoder reranker, added only after the evaluation set says BM25 is the bottleneck - Restricting a thread to a set of domains you trust for a given topic - Scheduled queries that email you when the answer to a standing question changes ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Your own instance. Not a public one. SEARXNG_URL=http://localhost:8080 # Required. Per-request timeout. Slow pages are dropped rather than allowed to hold up the answer. FETCH_TIMEOUT_MS=4000 # Required. The whole retrieval step's deadline. Partial results are expected and correct. FETCH_TOTAL_BUDGET_MS=12000 # Required. Streaming byte cap per page. Enforced during download, not after. FETCH_MAX_BYTES=2000000 # Required. The model used for the answer step. The rewrite step can use something smaller. ANSWER_MODEL=claude-sonnet-5 # Required · secret. Your provider console. Set a hard spend limit there as well as in this application. MODEL_API_KEY=sk-... # Required. The application stops answering at this figure rather than continuing to spend. MONTHLY_CAP_USD=25 # Required. SQLite file holding threads, cache and passages. This is the whole backup. DB_PATH=./data/answers.db
You are building a production product version of Perplexity. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Perplexity · product brief ## Problem A retrieval-plus-LLM answer engine is buildable, but Perplexity's search stack, source ranking, UX, mobile/browser surfaces, and model access make full parity hard. ## Product outcome A private answer engine whose citations are guaranteed to point at pages it actually read, that refuses to answer when the sources do not support one, and whose monthly cost has a ceiling it will not cross. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - search API - web fetcher/parser - LLM API - citation renderer - hosted backend ## Explicit non-goals for v1 - A web index of your own, which is the moat and stays the moat. - Search quality tuned by a team with query logs at scale. - Model routing and the credit allowance a subscription includes. - Polished mobile and browser applications. - search quality - source ranking - model routing - mobile/browser apps - publisher integrations - speed ## Success criteria - The citation validator rejects a crafted out-of-range citation in a unit test - The evaluation set score is recorded and has not regressed - An anonymous request burst produces zero model spend - The monthly cap has been tested by lowering it and confirming refusal - A restore has been performed on a second machine and dated ===== BRIEF.md ===== # Build brief · Perplexity The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal answer engine that searches the web, reads the pages, and answers with citations you can check to replace Perplexity. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not build a web index. Crawling and ranking the web is the moat and it is not reproducible at any budget you have. You are building the layer above search: retrieve, read, rank passages, answer, and cite honestly. If you would rather deploy than build, the open-source project that does this was called Perplexica and was renamed Vane in March 2026. Deploying it is a legitimate afternoon and this prompt is the alternative for people who want the pipeline to be theirs. ### Stack (fixed, do not substitute) - Search: SearXNG, self-hosted, with the JSON output format enabled · a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours - Runtime: Python 3.12 with FastAPI · the work is orchestrating slow network calls, which is what async is for - Extraction: trafilatura · purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper - Ranking: BM25 over sentence-window passages, with an optional cross-encoder reranker · start with the method that needs no model and no GPU, and measure before adding one - Answering: Any chat model through its API, or a local model through Ollama · the answer step is the only part that needs a large model, and it is a swappable adapter - Storage: SQLite with full text search · threads, cached pages and passages are all rows, and one file is the whole backup ### Data model (create this before Phase 1) - threads(id, title, created_at) - queries(id, thread_id, text, rewritten_text, asked_at) - results(id, query_id, url, title, snippet, engine, rank) - documents(id, url, fetched_at, status, content_hash, text, byte_count) · the cache, keyed by url - passages(id, document_id, ordinal, text, score) · what the answer is allowed to cite - answers(id, query_id, text, model, citations_json, validated) - Nothing may be cited that is not a row in passages for that query. That invariant is the whole point of Phase 4. ### Phase 1 · Search you control Build: Run SearXNG in Docker and enable the JSON output format in settings.yml, which is off by default. Add a client that queries it, normalises results, and caches identical queries for a short window so development does not hammer upstream engines. Trap: SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. Trap: Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. Done when: A query returns at least ten results as JSON with url, title and engine. The same query within the cache window is served from cache, verified by an unchanged upstream request count. A query returning nothing produces an empty result set and a clear message, not a stack trace. Do not build yet: No fetching, no ranking, no model calls. Search results only. ### Phase 2 · Fetch and extract, with limits that hold Build: Fetch the top results concurrently with a per-request timeout, a total time budget and a byte cap. Extract readable text with trafilatura. Respect robots.txt. Cache by url with a content hash so a re-fetch of unchanged content costs nothing. Trap: A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. Done when: Fetching eight pages where two never respond completes inside the total time budget and returns the six that worked. A deliberately large file is aborted at the cap rather than fully downloaded, verified by the bytes actually read. A page disallowed by robots.txt is skipped with a recorded reason. Do not build yet: No ranking and no answering yet. ### Phase 3 · Passages and ranking Build: Chunk extracted text into overlapping passages of a few sentences. Score them against the query with BM25 and keep the top handful across all documents, not the top few per document. Record the score so you can inspect why a passage was chosen. Trap: Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. Done when: For a factual query with a known answer, the passage containing that answer is in the top five by score. Passage boundaries never split a sentence. Dropping the best document from the input changes the answer's citations, proving retrieval feeds the answer rather than the model's memory. Do not build yet: No model call yet. ### Phase 4 · Answers whose citations are checked Build: Send the ranked passages, numbered, and require the model to cite by number. Then validate: every cited number must exist in the passages you sent. Reject and retry once on a violation, and if it fails again, return the answer with the invalid citations stripped and a visible note rather than pretending it was clean. Trap: Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. Done when: An answer citing passage seven when only five were supplied fails validation, verified by feeding a crafted response through the validator directly. Every citation in a normal answer resolves to a url that was fetched in that query. A query whose passages contain no answer produces an explicit statement that the sources do not answer it, rather than a fluent guess. Do not build yet: No threads or follow-ups yet. ### Phase 5 · Threads and follow-ups Build: Persist threads. For a follow-up, rewrite the question into a standalone query using the thread so far, and show the rewritten query so the user can see what was actually searched. Reuse cached documents from earlier turns where they are still relevant. Trap: Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. Done when: Asking what about the second one after a list question searches a rewritten standalone query, and that rewritten text is visible in the interface. A follow-up that is already standalone is not rewritten unnecessarily. Reopening a thread shows the same answers and citations without re-running any search. Do not build yet: No hardening or limits yet. ### Phase 6 · Limits, cost and running it for others Build: Add authentication, per-user rate limits and a hard monthly spend cap that stops serving rather than overspending. Log per-query cost. Put it behind TLS with a real certificate, and back up the SQLite file off the machine. Trap: An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view. Done when: An unauthenticated request never reaches a model call, verified by the spend log staying flat under a burst of anonymous requests. Passing the monthly cap returns a clear refusal instead of continuing to spend. A backup of the database has been restored on a second machine and the date recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - A web index. Crawling and ranking the open web is the actual asset, and metasearch borrows it rather than replacing it. - Search result quality tuning at the level a search company does it. Your ranking is BM25 over whatever SearXNG returned, and it will be worse in ways you can measure. - Model routing across many providers by task, which is a real part of what the subscription buys. - Mobile and browser applications, and the polish that makes the hosted product pleasant on a phone. ### README must contain - That SearXNG needs the json format enabled in settings.yml, because nothing works until it is - That every citation is validated against retrieved passages, and what happens when validation fails - The byte cap, the timeout and the total budget, and where to change them - Which model is used for answering, which for rewriting, and what a typical query costs - That the open-source project doing this whole job is Vane, formerly Perplexica, if you would rather deploy than build ===== ARCHITECTURE.md ===== # Architecture · Perplexity ## Stack | Part | Choice | Why | | --- | --- | --- | | Search | SearXNG, self-hosted, with the JSON output format enabled | a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours | | Runtime | Python 3.12 with FastAPI | the work is orchestrating slow network calls, which is what async is for | | Extraction | trafilatura | purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper | | Ranking | BM25 over sentence-window passages, with an optional cross-encoder reranker | start with the method that needs no model and no GPU, and measure before adding one | | Answering | Any chat model through its API, or a local model through Ollama | the answer step is the only part that needs a large model, and it is a swappable adapter | | Storage | SQLite with full text search | threads, cached pages and passages are all rows, and one file is the whole backup | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Search | Turning a question into candidate urls. | SearXNG is one implementation of a results feed. A paid search API is a drop-in behind the same client interface, and costs money per query in exchange for reliability. | | Fetch | Retrieving pages under strict time and byte limits. | Add a headless browser for pages that need JavaScript, but keep it behind the same interface and the same budget, because it is far slower. | | Rank | Chunking and scoring passages. | A cross-encoder reranker slots in after BM25. Add it only when the evaluation set says BM25 is the limiting factor. | | Answer | The model call and citation formatting. | Any chat model with structured output. The validator is what makes swapping safe, because a weaker model's invented citation is caught rather than shipped. | | Validate | Rejecting citations that were never retrieved. | Never swap this out. It is the difference between an answer engine and a confident guess. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `SEARXNG_URL` · required · Your own instance. Not a public one. - `FETCH_TIMEOUT_MS` · required · Per-request timeout. Slow pages are dropped rather than allowed to hold up the answer. - `FETCH_TOTAL_BUDGET_MS` · required · The whole retrieval step's deadline. Partial results are expected and correct. - `FETCH_MAX_BYTES` · required · Streaming byte cap per page. Enforced during download, not after. - `ANSWER_MODEL` · required · The model used for the answer step. The rewrite step can use something smaller. - `MODEL_API_KEY` · required, secret · Your provider console. Set a hard spend limit there as well as in this application. - `MONTHLY_CAP_USD` · required · The application stops answering at this figure rather than continuing to spend. - `DB_PATH` · required · SQLite file holding threads, cache and passages. This is the whole backup. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Perplexity product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: SearXNG, self-hosted, with the JSON output format enabled, Python 3.12 with FastAPI, trafilatura, BM25 over sentence-window passages, with an optional cross-encoder reranker, Any chat model through its API, or a local model through Ollama, SQLite with full text search. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. - An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view. ===== MILESTONES.md ===== # Delivery milestones · Perplexity Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Search you control A query returns structured results from your own SearXNG, not from scraping someone's results page. ### Steps 1. Run SearXNG and enable JSON Bring it up with Docker, then edit settings.yml so the search formats include json alongside html, and restart. This single line is the most common thing people miss. Files: `searxng/settings.yml` ```sh docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json' ``` 2. Write the search client Query with a timeout, normalise into your results shape, and keep the engine name so you can tell later which upstream is producing rubbish. Files: `search.py` 3. Cache identical queries briefly Key on the normalised query text with a short expiry. This is for your own iteration speed as much as for politeness. Files: `cache.py` ### Done when - [ ] A query returns ten or more JSON results with urls - [ ] A repeated query inside the cache window makes no upstream request - [ ] A nonsense query returns an empty list and a clear message - [ ] Stopping SearXNG produces a handled connection error, not an unhandled exception ### Watch out - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. ## M2 · Fetch and extract, with limits that hold Pages become clean text without a slow or hostile page being able to hang the request. ### Steps 1. Fetch concurrently with a total budget Bounded concurrency, a per-request timeout, and an overall deadline for the batch. Partial results are the expected outcome, not a failure. Files: `fetch.py` 2. Stream and cap the body Read in chunks and stop at the cap. Record the byte count so the check in this phase is measurable rather than asserted. Files: `fetch.py` 3. Extract and cache by content hash trafilatura for the text. Hash the extracted text; if it matches the cached hash, keep the existing passages and skip re-chunking. Files: `extract.py` ```sh pip install trafilatura httpx ``` ### Done when - [ ] Eight fetches with two dead hosts finish inside the total budget - [ ] A large file is aborted at the byte cap, proved by the recorded byte count - [ ] A robots.txt disallow is skipped with a stored reason - [ ] Re-fetching unchanged content does not re-chunk it ### Watch out - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. ## M3 · Passages and ranking The text handed to the model is the part of the page that answers the question, not the whole page. ### Steps 1. Chunk with sentence awareness and overlap Split on sentence boundaries, group into windows of a few sentences with an overlap so an answer spanning a boundary is not lost. Files: `chunk.py` 2. Score with BM25 across all documents One corpus for the whole query, not one per document. Store the score on the passage row. Files: `rank.py` ```sh pip install rank-bm25 ``` 3. Build a small evaluation set Ten queries with a known correct source. Measure whether the right passage lands in the top five, before and after any change to chunking or scoring. Files: `eval/queries.json` ### Done when - [ ] The passage containing a known answer is in the top five for that query - [ ] No passage begins or ends mid-sentence - [ ] Ranking is global, verified by a case where one document contributes several passages and another contributes none - [ ] The evaluation set runs as a script and reports a score you can compare over time ### Watch out - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. ## M4 · Answers whose citations are checked Every citation points at a passage that was actually retrieved, and an answer that invents one is rejected. ### Steps 1. Number the passages and demand numeric citations Pass passages as a numbered list and instruct the model to cite only those numbers. Ask for the answer and the citation list as structured output so parsing is not a regular expression. Files: `answer.py` 2. Validate every citation against the supplied set A pure function from answer plus passage ids to a verdict. Unit test it with a crafted out-of-range citation; do not wait for a model to produce one. Files: `validate.py`, `tests/test_validate.py` 3. Handle the no-answer case explicitly If the top passages do not support an answer, say so. This is the behaviour that makes the tool trustworthy, and it needs to be an instruction and a test, not a hope. Files: `answer.py` ### Done when - [ ] A crafted answer citing a passage that was not supplied is rejected by the validator - [ ] Every citation in a real answer resolves to a fetched url - [ ] A query with no supporting passages produces an explicit refusal to answer - [ ] The validator is unit tested independently of any model call ### Watch out - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. ## M5 · Threads and follow-ups A follow-up question understands the conversation without re-searching everything from scratch. ### Steps 1. Persist threads, queries, answers and citations Store enough that a reopened thread renders from the database with no network access at all. Files: `store.py` 2. Rewrite follow-ups to standalone queries A small model call that takes the thread and the new question and returns one search query. Show it in the interface. Files: `rewrite.py` 3. Reuse the document cache across turns If a follow-up retrieves a url already fetched in this thread and it is still fresh, use the cached text. Files: `cache.py` ### Done when - [ ] A pronoun-laden follow-up searches a sensible standalone query - [ ] The rewritten query is visible to the user - [ ] Reopening a thread renders with no network calls - [ ] A repeated url within a thread is not re-fetched while fresh ### Watch out - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. ## M6 · Limits, cost and running it for others (production only) Exposing it beyond your own machine does not hand anyone your model budget. ### Steps 1. Authenticate before any paid call Session or token authentication checked in the handler that triggers retrieval, not merely on the page that renders the form. Files: `auth.py` 2. Meter and cap spending Record tokens and cost per query. Compare against a monthly cap and stop serving answers when it is reached, with a message saying why. Files: `meter.py` 3. Deploy with TLS and back up the database Caddy in front for certificates. A scheduled copy of the SQLite file off the machine, and one restore performed on purpose. Files: `Caddyfile` ### Done when - [ ] A burst of anonymous requests produces no model spend - [ ] Passing the monthly cap returns a refusal rather than continuing - [ ] The site serves over HTTPS with a valid certificate - [ ] A database restore has been performed on a second machine and dated ### Watch out - An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view. ===== OPERATIONS.md ===== # Operations · Perplexity ## Backup The SQLite file holds threads, answers and the page cache. Copy it off the machine on a schedule. The cache is regenerable, so a smaller backup that excludes document text is a reasonable trade if size becomes a problem. ## Restore Restore the file, start the service, and open an old thread. It should render with no network access at all, which is also the test that persistence is complete. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Track spend per day against the cap, the proportion of queries that fail citation validation, and fetch failure rates per host. A rising validation failure rate usually means retrieval got worse, not that the model did. ## Incident checklist If spend spikes, the cap should already have stopped it; if it did not, that is the bug to fix first. If a search backend starts returning nothing, the empty-result path from Phase 1 keeps the application answering honestly rather than crashing. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The citation validator rejects a crafted out-of-range citation in a unit test - [ ] The evaluation set score is recorded and has not regressed - [ ] An anonymous request burst produces zero model spend - [ ] The monthly cap has been tested by lowering it and confirming refusal - [ ] A restore has been performed on a second machine and dated ## Launch constraint Do not market omitted Perplexity capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Your own instance. Not a public one. SEARXNG_URL=http://localhost:8080 # Required. Per-request timeout. Slow pages are dropped rather than allowed to hold up the answer. FETCH_TIMEOUT_MS=4000 # Required. The whole retrieval step's deadline. Partial results are expected and correct. FETCH_TOTAL_BUDGET_MS=12000 # Required. Streaming byte cap per page. Enforced during download, not after. FETCH_MAX_BYTES=2000000 # Required. The model used for the answer step. The rewrite step can use something smaller. ANSWER_MODEL=claude-sonnet-5 # Required · secret. Your provider console. Set a hard spend limit there as well as in this application. MODEL_API_KEY=sk-... # Required. The application stops answering at this figure rather than continuing to spend. MONTHLY_CAP_USD=25 # Required. SQLite file holding threads, cache and passages. This is the whole backup. DB_PATH=./data/answers.db
README.md
# Perplexity · indie build A self-hosted answer engine: a SearXNG instance you control providing results, a fetcher that extracts readable text under strict caps, passage ranking, an answer step whose citations are validated against the passages that were actually retrieved, and threads with follow-up questions that carry context. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Search | SearXNG, self-hosted, with the JSON output format enabled | a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours | | Runtime | Python 3.12 with FastAPI | the work is orchestrating slow network calls, which is what async is for | | Extraction | trafilatura | purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper | | Ranking | BM25 over sentence-window passages, with an optional cross-encoder reranker | start with the method that needs no model and no GPU, and measure before adding one | | Answering | Any chat model through its API, or a local model through Ollama | the answer step is the only part that needs a large model, and it is a swappable adapter | | Storage | SQLite with full text search | threads, cached pages and passages are all rows, and one file is the whole backup | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Docker with Compose v2** · free for personal use; check Docker Desktop's licence if this is for a company - Why: Runs SearXNG. Installing SearXNG without containers is possible and is a worse afternoon. - Get it: Install Docker Desktop, or Docker Engine plus the compose plugin on Linux. - Verify: docker compose version prints v2 or higher - [ ] **Python 3.12 or newer** · free - Why: Runs the pipeline. - Get it: python.org or your package manager, then create a virtual environment in the project directory. - Verify: python3 --version prints 3.12 or higher - [ ] **A model API key, or Ollama for a local model** · pay per token; a short answer over five passages is typically well under a cent, and long threads add up faster than you expect - Why: The answer step needs a model. Everything before it works without one, which is useful while you are building. - Get it: Create an account with a provider and generate a key in their console, and set a spend limit at the same time. Or install Ollama and pull a model if you would rather pay nothing and accept weaker answers. - Verify: A single test completion returns text and appears in the provider usage dashboard - [ ] **Your own SearXNG instance, not a public one** · free, but you carry the upstream engines' rate limits yourself - Why: Public instances block automated traffic and rate-limit, so building against one produces a system that fails as soon as it works. - Get it: Run the official container locally, then enable json in the search formats of settings.yml and restart. - Verify: curl with format=json returns JSON rather than a 403 - [ ] **Ten queries with known correct sources** · free - Why: Ranking changes feel better and are often worse. Without a fixed set you are tuning on vibes. - Get it: Write them yourself from questions you actually asked this week, and record the url that genuinely answers each. An hour of work that pays for itself in Phase 3. ## Quick start ```sh docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json' ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A web index of your own, which is the moat and stays the moat. - Search quality tuned by a team with query logs at scale. - Model routing and the credit allowance a subscription includes. - Polished mobile and browser applications. - search quality - source ranking - model routing - mobile/browser apps - publisher integrations - speed If one of those is essential to you, that is the reason to keep paying for Perplexity, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Perplexity The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal answer engine that searches the web, reads the pages, and answers with citations you can check to replace Perplexity. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not build a web index. Crawling and ranking the web is the moat and it is not reproducible at any budget you have. You are building the layer above search: retrieve, read, rank passages, answer, and cite honestly. If you would rather deploy than build, the open-source project that does this was called Perplexica and was renamed Vane in March 2026. Deploying it is a legitimate afternoon and this prompt is the alternative for people who want the pipeline to be theirs. ### Stack (fixed, do not substitute) - Search: SearXNG, self-hosted, with the JSON output format enabled · a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours - Runtime: Python 3.12 with FastAPI · the work is orchestrating slow network calls, which is what async is for - Extraction: trafilatura · purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper - Ranking: BM25 over sentence-window passages, with an optional cross-encoder reranker · start with the method that needs no model and no GPU, and measure before adding one - Answering: Any chat model through its API, or a local model through Ollama · the answer step is the only part that needs a large model, and it is a swappable adapter - Storage: SQLite with full text search · threads, cached pages and passages are all rows, and one file is the whole backup ### Data model (create this before Phase 1) - threads(id, title, created_at) - queries(id, thread_id, text, rewritten_text, asked_at) - results(id, query_id, url, title, snippet, engine, rank) - documents(id, url, fetched_at, status, content_hash, text, byte_count) · the cache, keyed by url - passages(id, document_id, ordinal, text, score) · what the answer is allowed to cite - answers(id, query_id, text, model, citations_json, validated) - Nothing may be cited that is not a row in passages for that query. That invariant is the whole point of Phase 4. ### Phase 1 · Search you control Build: Run SearXNG in Docker and enable the JSON output format in settings.yml, which is off by default. Add a client that queries it, normalises results, and caches identical queries for a short window so development does not hammer upstream engines. Trap: SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. Trap: Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. Done when: A query returns at least ten results as JSON with url, title and engine. The same query within the cache window is served from cache, verified by an unchanged upstream request count. A query returning nothing produces an empty result set and a clear message, not a stack trace. Do not build yet: No fetching, no ranking, no model calls. Search results only. ### Phase 2 · Fetch and extract, with limits that hold Build: Fetch the top results concurrently with a per-request timeout, a total time budget and a byte cap. Extract readable text with trafilatura. Respect robots.txt. Cache by url with a content hash so a re-fetch of unchanged content costs nothing. Trap: A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. Done when: Fetching eight pages where two never respond completes inside the total time budget and returns the six that worked. A deliberately large file is aborted at the cap rather than fully downloaded, verified by the bytes actually read. A page disallowed by robots.txt is skipped with a recorded reason. Do not build yet: No ranking and no answering yet. ### Phase 3 · Passages and ranking Build: Chunk extracted text into overlapping passages of a few sentences. Score them against the query with BM25 and keep the top handful across all documents, not the top few per document. Record the score so you can inspect why a passage was chosen. Trap: Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. Done when: For a factual query with a known answer, the passage containing that answer is in the top five by score. Passage boundaries never split a sentence. Dropping the best document from the input changes the answer's citations, proving retrieval feeds the answer rather than the model's memory. Do not build yet: No model call yet. ### Phase 4 · Answers whose citations are checked Build: Send the ranked passages, numbered, and require the model to cite by number. Then validate: every cited number must exist in the passages you sent. Reject and retry once on a violation, and if it fails again, return the answer with the invalid citations stripped and a visible note rather than pretending it was clean. Trap: Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. Done when: An answer citing passage seven when only five were supplied fails validation, verified by feeding a crafted response through the validator directly. Every citation in a normal answer resolves to a url that was fetched in that query. A query whose passages contain no answer produces an explicit statement that the sources do not answer it, rather than a fluent guess. Do not build yet: No threads or follow-ups yet. ### Phase 5 · Threads and follow-ups Build: Persist threads. For a follow-up, rewrite the question into a standalone query using the thread so far, and show the rewritten query so the user can see what was actually searched. Reuse cached documents from earlier turns where they are still relevant. Trap: Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. Done when: Asking what about the second one after a list question searches a rewritten standalone query, and that rewritten text is visible in the interface. A follow-up that is already standalone is not rewritten unnecessarily. Reopening a thread shows the same answers and citations without re-running any search. Do not build yet: No hardening or limits yet. ### Phase 6 · Limits, cost and running it for others Build: Add authentication, per-user rate limits and a hard monthly spend cap that stops serving rather than overspending. Log per-query cost. Put it behind TLS with a real certificate, and back up the SQLite file off the machine. Trap: An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view. Done when: An unauthenticated request never reaches a model call, verified by the spend log staying flat under a burst of anonymous requests. Passing the monthly cap returns a clear refusal instead of continuing to spend. A backup of the database has been restored on a second machine and the date recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - A web index. Crawling and ranking the open web is the actual asset, and metasearch borrows it rather than replacing it. - Search result quality tuning at the level a search company does it. Your ranking is BM25 over whatever SearXNG returned, and it will be worse in ways you can measure. - Model routing across many providers by task, which is a real part of what the subscription buys. - Mobile and browser applications, and the polish that makes the hosted product pleasant on a phone. ### README must contain - That SearXNG needs the json format enabled in settings.yml, because nothing works until it is - That every citation is validated against retrieved passages, and what happens when validation fails - The byte cap, the timeout and the total budget, and where to change them - Which model is used for answering, which for rewriting, and what a typical query costs - That the open-source project doing this whole job is Vane, formerly Perplexica, if you would rather deploy than build
AGENTS.md
# Agent instructions · Perplexity indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: SearXNG, self-hosted, with the JSON output format enabled, Python 3.12 with FastAPI, trafilatura, BM25 over sentence-window passages, with an optional cross-encoder reranker, Any chat model through its API, or a local model through Ollama, SQLite with full text search. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious.
BUILD_PLAN.md
# Build plan · Perplexity A self-hosted answer engine: a SearXNG instance you control providing results, a fetcher that extracts readable text under strict caps, passage ranking, an answer step whose citations are validated against the passages that were actually retrieved, and threads with follow-up questions that carry context. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Search you control A query returns structured results from your own SearXNG, not from scraping someone's results page. ### Steps 1. Run SearXNG and enable JSON Bring it up with Docker, then edit settings.yml so the search formats include json alongside html, and restart. This single line is the most common thing people miss. Files: `searxng/settings.yml` ```sh docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json' ``` 2. Write the search client Query with a timeout, normalise into your results shape, and keep the engine name so you can tell later which upstream is producing rubbish. Files: `search.py` 3. Cache identical queries briefly Key on the normalised query text with a short expiry. This is for your own iteration speed as much as for politeness. Files: `cache.py` ### Done when - [ ] A query returns ten or more JSON results with urls - [ ] A repeated query inside the cache window makes no upstream request - [ ] A nonsense query returns an empty list and a clear message - [ ] Stopping SearXNG produces a handled connection error, not an unhandled exception ### Watch out - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. ## Phase 2 · Fetch and extract, with limits that hold Pages become clean text without a slow or hostile page being able to hang the request. ### Steps 1. Fetch concurrently with a total budget Bounded concurrency, a per-request timeout, and an overall deadline for the batch. Partial results are the expected outcome, not a failure. Files: `fetch.py` 2. Stream and cap the body Read in chunks and stop at the cap. Record the byte count so the check in this phase is measurable rather than asserted. Files: `fetch.py` 3. Extract and cache by content hash trafilatura for the text. Hash the extracted text; if it matches the cached hash, keep the existing passages and skip re-chunking. Files: `extract.py` ```sh pip install trafilatura httpx ``` ### Done when - [ ] Eight fetches with two dead hosts finish inside the total budget - [ ] A large file is aborted at the byte cap, proved by the recorded byte count - [ ] A robots.txt disallow is skipped with a stored reason - [ ] Re-fetching unchanged content does not re-chunk it ### Watch out - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. ## Phase 3 · Passages and ranking The text handed to the model is the part of the page that answers the question, not the whole page. ### Steps 1. Chunk with sentence awareness and overlap Split on sentence boundaries, group into windows of a few sentences with an overlap so an answer spanning a boundary is not lost. Files: `chunk.py` 2. Score with BM25 across all documents One corpus for the whole query, not one per document. Store the score on the passage row. Files: `rank.py` ```sh pip install rank-bm25 ``` 3. Build a small evaluation set Ten queries with a known correct source. Measure whether the right passage lands in the top five, before and after any change to chunking or scoring. Files: `eval/queries.json` ### Done when - [ ] The passage containing a known answer is in the top five for that query - [ ] No passage begins or ends mid-sentence - [ ] Ranking is global, verified by a case where one document contributes several passages and another contributes none - [ ] The evaluation set runs as a script and reports a score you can compare over time ### Watch out - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. ## Phase 4 · Answers whose citations are checked Every citation points at a passage that was actually retrieved, and an answer that invents one is rejected. ### Steps 1. Number the passages and demand numeric citations Pass passages as a numbered list and instruct the model to cite only those numbers. Ask for the answer and the citation list as structured output so parsing is not a regular expression. Files: `answer.py` 2. Validate every citation against the supplied set A pure function from answer plus passage ids to a verdict. Unit test it with a crafted out-of-range citation; do not wait for a model to produce one. Files: `validate.py`, `tests/test_validate.py` 3. Handle the no-answer case explicitly If the top passages do not support an answer, say so. This is the behaviour that makes the tool trustworthy, and it needs to be an instruction and a test, not a hope. Files: `answer.py` ### Done when - [ ] A crafted answer citing a passage that was not supplied is rejected by the validator - [ ] Every citation in a real answer resolves to a fetched url - [ ] A query with no supporting passages produces an explicit refusal to answer - [ ] The validator is unit tested independently of any model call ### Watch out - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. ## Phase 5 · Threads and follow-ups A follow-up question understands the conversation without re-searching everything from scratch. ### Steps 1. Persist threads, queries, answers and citations Store enough that a reopened thread renders from the database with no network access at all. Files: `store.py` 2. Rewrite follow-ups to standalone queries A small model call that takes the thread and the new question and returns one search query. Show it in the interface. Files: `rewrite.py` 3. Reuse the document cache across turns If a follow-up retrieves a url already fetched in this thread and it is still fresh, use the cached text. Files: `cache.py` ### Done when - [ ] A pronoun-laden follow-up searches a sensible standalone query - [ ] The rewritten query is visible to the user - [ ] Reopening a thread renders with no network calls - [ ] A repeated url within a thread is not re-fetched while fresh ### Watch out - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. ## Not in this build - A web index of your own, which is the moat and stays the moat. - Search quality tuned by a team with query logs at scale. - Model routing and the credit allowance a subscription includes. - Polished mobile and browser applications. ## After v1, if you want it - A cross-encoder reranker, added only after the evaluation set says BM25 is the bottleneck - Restricting a thread to a set of domains you trust for a given topic - Scheduled queries that email you when the answer to a standing question changes
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Your own instance. Not a public one. SEARXNG_URL=http://localhost:8080 # Required. Per-request timeout. Slow pages are dropped rather than allowed to hold up the answer. FETCH_TIMEOUT_MS=4000 # Required. The whole retrieval step's deadline. Partial results are expected and correct. FETCH_TOTAL_BUDGET_MS=12000 # Required. Streaming byte cap per page. Enforced during download, not after. FETCH_MAX_BYTES=2000000 # Required. The model used for the answer step. The rewrite step can use something smaller. ANSWER_MODEL=claude-sonnet-5 # Required · secret. Your provider console. Set a hard spend limit there as well as in this application. MODEL_API_KEY=sk-... # Required. The application stops answering at this figure rather than continuing to spend. MONTHLY_CAP_USD=25 # Required. SQLite file holding threads, cache and passages. This is the whole backup. DB_PATH=./data/answers.db
PRODUCT.md
# Perplexity · product brief ## Problem A retrieval-plus-LLM answer engine is buildable, but Perplexity's search stack, source ranking, UX, mobile/browser surfaces, and model access make full parity hard. ## Product outcome A private answer engine whose citations are guaranteed to point at pages it actually read, that refuses to answer when the sources do not support one, and whose monthly cost has a ceiling it will not cross. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - search API - web fetcher/parser - LLM API - citation renderer - hosted backend ## Explicit non-goals for v1 - A web index of your own, which is the moat and stays the moat. - Search quality tuned by a team with query logs at scale. - Model routing and the credit allowance a subscription includes. - Polished mobile and browser applications. - search quality - source ranking - model routing - mobile/browser apps - publisher integrations - speed ## Success criteria - The citation validator rejects a crafted out-of-range citation in a unit test - The evaluation set score is recorded and has not regressed - An anonymous request burst produces zero model spend - The monthly cap has been tested by lowering it and confirming refusal - A restore has been performed on a second machine and dated
BRIEF.md
# Build brief · Perplexity The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal answer engine that searches the web, reads the pages, and answers with citations you can check to replace Perplexity. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not build a web index. Crawling and ranking the web is the moat and it is not reproducible at any budget you have. You are building the layer above search: retrieve, read, rank passages, answer, and cite honestly. If you would rather deploy than build, the open-source project that does this was called Perplexica and was renamed Vane in March 2026. Deploying it is a legitimate afternoon and this prompt is the alternative for people who want the pipeline to be theirs. ### Stack (fixed, do not substitute) - Search: SearXNG, self-hosted, with the JSON output format enabled · a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours - Runtime: Python 3.12 with FastAPI · the work is orchestrating slow network calls, which is what async is for - Extraction: trafilatura · purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper - Ranking: BM25 over sentence-window passages, with an optional cross-encoder reranker · start with the method that needs no model and no GPU, and measure before adding one - Answering: Any chat model through its API, or a local model through Ollama · the answer step is the only part that needs a large model, and it is a swappable adapter - Storage: SQLite with full text search · threads, cached pages and passages are all rows, and one file is the whole backup ### Data model (create this before Phase 1) - threads(id, title, created_at) - queries(id, thread_id, text, rewritten_text, asked_at) - results(id, query_id, url, title, snippet, engine, rank) - documents(id, url, fetched_at, status, content_hash, text, byte_count) · the cache, keyed by url - passages(id, document_id, ordinal, text, score) · what the answer is allowed to cite - answers(id, query_id, text, model, citations_json, validated) - Nothing may be cited that is not a row in passages for that query. That invariant is the whole point of Phase 4. ### Phase 1 · Search you control Build: Run SearXNG in Docker and enable the JSON output format in settings.yml, which is off by default. Add a client that queries it, normalises results, and caches identical queries for a short window so development does not hammer upstream engines. Trap: SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. Trap: Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. Done when: A query returns at least ten results as JSON with url, title and engine. The same query within the cache window is served from cache, verified by an unchanged upstream request count. A query returning nothing produces an empty result set and a clear message, not a stack trace. Do not build yet: No fetching, no ranking, no model calls. Search results only. ### Phase 2 · Fetch and extract, with limits that hold Build: Fetch the top results concurrently with a per-request timeout, a total time budget and a byte cap. Extract readable text with trafilatura. Respect robots.txt. Cache by url with a content hash so a re-fetch of unchanged content costs nothing. Trap: A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. Done when: Fetching eight pages where two never respond completes inside the total time budget and returns the six that worked. A deliberately large file is aborted at the cap rather than fully downloaded, verified by the bytes actually read. A page disallowed by robots.txt is skipped with a recorded reason. Do not build yet: No ranking and no answering yet. ### Phase 3 · Passages and ranking Build: Chunk extracted text into overlapping passages of a few sentences. Score them against the query with BM25 and keep the top handful across all documents, not the top few per document. Record the score so you can inspect why a passage was chosen. Trap: Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. Done when: For a factual query with a known answer, the passage containing that answer is in the top five by score. Passage boundaries never split a sentence. Dropping the best document from the input changes the answer's citations, proving retrieval feeds the answer rather than the model's memory. Do not build yet: No model call yet. ### Phase 4 · Answers whose citations are checked Build: Send the ranked passages, numbered, and require the model to cite by number. Then validate: every cited number must exist in the passages you sent. Reject and retry once on a violation, and if it fails again, return the answer with the invalid citations stripped and a visible note rather than pretending it was clean. Trap: Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. Done when: An answer citing passage seven when only five were supplied fails validation, verified by feeding a crafted response through the validator directly. Every citation in a normal answer resolves to a url that was fetched in that query. A query whose passages contain no answer produces an explicit statement that the sources do not answer it, rather than a fluent guess. Do not build yet: No threads or follow-ups yet. ### Phase 5 · Threads and follow-ups Build: Persist threads. For a follow-up, rewrite the question into a standalone query using the thread so far, and show the rewritten query so the user can see what was actually searched. Reuse cached documents from earlier turns where they are still relevant. Trap: Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. Done when: Asking what about the second one after a list question searches a rewritten standalone query, and that rewritten text is visible in the interface. A follow-up that is already standalone is not rewritten unnecessarily. Reopening a thread shows the same answers and citations without re-running any search. Do not build yet: No hardening or limits yet. ### Phase 6 · Limits, cost and running it for others Build: Add authentication, per-user rate limits and a hard monthly spend cap that stops serving rather than overspending. Log per-query cost. Put it behind TLS with a real certificate, and back up the SQLite file off the machine. Trap: An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view. Done when: An unauthenticated request never reaches a model call, verified by the spend log staying flat under a burst of anonymous requests. Passing the monthly cap returns a clear refusal instead of continuing to spend. A backup of the database has been restored on a second machine and the date recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - A web index. Crawling and ranking the open web is the actual asset, and metasearch borrows it rather than replacing it. - Search result quality tuning at the level a search company does it. Your ranking is BM25 over whatever SearXNG returned, and it will be worse in ways you can measure. - Model routing across many providers by task, which is a real part of what the subscription buys. - Mobile and browser applications, and the polish that makes the hosted product pleasant on a phone. ### README must contain - That SearXNG needs the json format enabled in settings.yml, because nothing works until it is - That every citation is validated against retrieved passages, and what happens when validation fails - The byte cap, the timeout and the total budget, and where to change them - Which model is used for answering, which for rewriting, and what a typical query costs - That the open-source project doing this whole job is Vane, formerly Perplexica, if you would rather deploy than build
ARCHITECTURE.md
# Architecture · Perplexity ## Stack | Part | Choice | Why | | --- | --- | --- | | Search | SearXNG, self-hosted, with the JSON output format enabled | a metasearch layer gives you results without a crawler, and self-hosting means no per-query bill and no rate limit that is not yours | | Runtime | Python 3.12 with FastAPI | the work is orchestrating slow network calls, which is what async is for | | Extraction | trafilatura | purpose-built for pulling article text out of a page, and far better than a hand-rolled tag stripper | | Ranking | BM25 over sentence-window passages, with an optional cross-encoder reranker | start with the method that needs no model and no GPU, and measure before adding one | | Answering | Any chat model through its API, or a local model through Ollama | the answer step is the only part that needs a large model, and it is a swappable adapter | | Storage | SQLite with full text search | threads, cached pages and passages are all rows, and one file is the whole backup | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Search | Turning a question into candidate urls. | SearXNG is one implementation of a results feed. A paid search API is a drop-in behind the same client interface, and costs money per query in exchange for reliability. | | Fetch | Retrieving pages under strict time and byte limits. | Add a headless browser for pages that need JavaScript, but keep it behind the same interface and the same budget, because it is far slower. | | Rank | Chunking and scoring passages. | A cross-encoder reranker slots in after BM25. Add it only when the evaluation set says BM25 is the limiting factor. | | Answer | The model call and citation formatting. | Any chat model with structured output. The validator is what makes swapping safe, because a weaker model's invented citation is caught rather than shipped. | | Validate | Rejecting citations that were never retrieved. | Never swap this out. It is the difference between an answer engine and a confident guess. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `SEARXNG_URL` · required · Your own instance. Not a public one. - `FETCH_TIMEOUT_MS` · required · Per-request timeout. Slow pages are dropped rather than allowed to hold up the answer. - `FETCH_TOTAL_BUDGET_MS` · required · The whole retrieval step's deadline. Partial results are expected and correct. - `FETCH_MAX_BYTES` · required · Streaming byte cap per page. Enforced during download, not after. - `ANSWER_MODEL` · required · The model used for the answer step. The rewrite step can use something smaller. - `MODEL_API_KEY` · required, secret · Your provider console. Set a hard spend limit there as well as in this application. - `MONTHLY_CAP_USD` · required · The application stops answering at this figure rather than continuing to spend. - `DB_PATH` · required · SQLite file holding threads, cache and passages. This is the whole backup. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Perplexity product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: SearXNG, self-hosted, with the JSON output format enabled, Python 3.12 with FastAPI, trafilatura, BM25 over sentence-window passages, with an optional cross-encoder reranker, Any chat model through its API, or a local model through Ollama, SQLite with full text search. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. - An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view.
MILESTONES.md
# Delivery milestones · Perplexity Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Search you control A query returns structured results from your own SearXNG, not from scraping someone's results page. ### Steps 1. Run SearXNG and enable JSON Bring it up with Docker, then edit settings.yml so the search formats include json alongside html, and restart. This single line is the most common thing people miss. Files: `searxng/settings.yml` ```sh docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json' ``` 2. Write the search client Query with a timeout, normalise into your results shape, and keep the engine name so you can tell later which upstream is producing rubbish. Files: `search.py` 3. Cache identical queries briefly Key on the normalised query text with a short expiry. This is for your own iteration speed as much as for politeness. Files: `cache.py` ### Done when - [ ] A query returns ten or more JSON results with urls - [ ] A repeated query inside the cache window makes no upstream request - [ ] A nonsense query returns an empty list and a clear message - [ ] Stopping SearXNG produces a handled connection error, not an unhandled exception ### Watch out - SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself. - Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem. ## M2 · Fetch and extract, with limits that hold Pages become clean text without a slow or hostile page being able to hang the request. ### Steps 1. Fetch concurrently with a total budget Bounded concurrency, a per-request timeout, and an overall deadline for the batch. Partial results are the expected outcome, not a failure. Files: `fetch.py` 2. Stream and cap the body Read in chunks and stop at the cap. Record the byte count so the check in this phase is measurable rather than asserted. Files: `fetch.py` 3. Extract and cache by content hash trafilatura for the text. Hash the extracted text; if it matches the cached hash, keep the existing passages and skip re-chunking. Files: `extract.py` ```sh pip install trafilatura httpx ``` ### Done when - [ ] Eight fetches with two dead hosts finish inside the total budget - [ ] A large file is aborted at the byte cap, proved by the recorded byte count - [ ] A robots.txt disallow is skipped with a stored reason - [ ] Re-fetching unchanged content does not re-chunk it ### Watch out - A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget. ## M3 · Passages and ranking The text handed to the model is the part of the page that answers the question, not the whole page. ### Steps 1. Chunk with sentence awareness and overlap Split on sentence boundaries, group into windows of a few sentences with an overlap so an answer spanning a boundary is not lost. Files: `chunk.py` 2. Score with BM25 across all documents One corpus for the whole query, not one per document. Store the score on the passage row. Files: `rank.py` ```sh pip install rank-bm25 ``` 3. Build a small evaluation set Ten queries with a known correct source. Measure whether the right passage lands in the top five, before and after any change to chunking or scoring. Files: `eval/queries.json` ### Done when - [ ] The passage containing a known answer is in the top five for that query - [ ] No passage begins or ends mid-sentence - [ ] Ranking is global, verified by a case where one document contributes several passages and another contributes none - [ ] The evaluation set runs as a script and reports a score you can compare over time ### Watch out - Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none. ## M4 · Answers whose citations are checked Every citation points at a passage that was actually retrieved, and an answer that invents one is rejected. ### Steps 1. Number the passages and demand numeric citations Pass passages as a numbered list and instruct the model to cite only those numbers. Ask for the answer and the citation list as structured output so parsing is not a regular expression. Files: `answer.py` 2. Validate every citation against the supplied set A pure function from answer plus passage ids to a verdict. Unit test it with a crafted out-of-range citation; do not wait for a model to produce one. Files: `validate.py`, `tests/test_validate.py` 3. Handle the no-answer case explicitly If the top passages do not support an answer, say so. This is the behaviour that makes the tool trustworthy, and it needs to be an instruction and a test, not a hope. Files: `answer.py` ### Done when - [ ] A crafted answer citing a passage that was not supplied is rejected by the validator - [ ] Every citation in a real answer resolves to a fetched url - [ ] A query with no supporting passages produces an explicit refusal to answer - [ ] The validator is unit tested independently of any model call ### Watch out - Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box. ## M5 · Threads and follow-ups A follow-up question understands the conversation without re-searching everything from scratch. ### Steps 1. Persist threads, queries, answers and citations Store enough that a reopened thread renders from the database with no network access at all. Files: `store.py` 2. Rewrite follow-ups to standalone queries A small model call that takes the thread and the new question and returns one search query. Show it in the interface. Files: `rewrite.py` 3. Reuse the document cache across turns If a follow-up retrieves a url already fetched in this thread and it is still fresh, use the cached text. Files: `cache.py` ### Done when - [ ] A pronoun-laden follow-up searches a sensible standalone query - [ ] The rewritten query is visible to the user - [ ] Reopening a thread renders with no network calls - [ ] A repeated url within a thread is not re-fetched while fresh ### Watch out - Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious. ## M6 · Limits, cost and running it for others (production only) Exposing it beyond your own machine does not hand anyone your model budget. ### Steps 1. Authenticate before any paid call Session or token authentication checked in the handler that triggers retrieval, not merely on the page that renders the form. Files: `auth.py` 2. Meter and cap spending Record tokens and cost per query. Compare against a monthly cap and stop serving answers when it is reached, with a message saying why. Files: `meter.py` 3. Deploy with TLS and back up the database Caddy in front for certificates. A scheduled copy of the SQLite file off the machine, and one restore performed on purpose. Files: `Caddyfile` ### Done when - [ ] A burst of anonymous requests produces no model spend - [ ] Passing the monthly cap returns a refusal rather than continuing - [ ] The site serves over HTTPS with a valid certificate - [ ] A database restore has been performed on a second machine and dated ### Watch out - An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view.
OPERATIONS.md
# Operations · Perplexity ## Backup The SQLite file holds threads, answers and the page cache. Copy it off the machine on a schedule. The cache is regenerable, so a smaller backup that excludes document text is a reasonable trade if size becomes a problem. ## Restore Restore the file, start the service, and open an old thread. It should render with no network access at all, which is also the test that persistence is complete. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Track spend per day against the cap, the proportion of queries that fail citation validation, and fetch failure rates per host. A rising validation failure rate usually means retrieval got worse, not that the model did. ## Incident checklist If spend spikes, the cap should already have stopped it; if it did not, that is the bug to fix first. If a search backend starts returning nothing, the empty-result path from Phase 1 keeps the application answering honestly rather than crashing. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The citation validator rejects a crafted out-of-range citation in a unit test - [ ] The evaluation set score is recorded and has not regressed - [ ] An anonymous request burst produces zero model spend - [ ] The monthly cap has been tested by lowering it and confirming refusal - [ ] A restore has been performed on a second machine and dated ## Launch constraint Do not market omitted Perplexity capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Your own instance. Not a public one. SEARXNG_URL=http://localhost:8080 # Required. Per-request timeout. Slow pages are dropped rather than allowed to hold up the answer. FETCH_TIMEOUT_MS=4000 # Required. The whole retrieval step's deadline. Partial results are expected and correct. FETCH_TOTAL_BUDGET_MS=12000 # Required. Streaming byte cap per page. Enforced during download, not after. FETCH_MAX_BYTES=2000000 # Required. The model used for the answer step. The rewrite step can use something smaller. ANSWER_MODEL=claude-sonnet-5 # Required · secret. Your provider console. Set a hard spend limit there as well as in this application. MODEL_API_KEY=sk-... # Required. The application stops answering at this figure rather than continuing to spend. MONTHLY_CAP_USD=25 # Required. SQLite file holding threads, cache and passages. This is the whole backup. DB_PATH=./data/answers.db
$ choose a build depth, inspect the files, then open the complete pack in your agent
They pay because answers arrive fast with sources and fewer query-building chores.
xsearch quality
xsource ranking
xmodel routing
xmobile/browser apps
xpublisher integrations
xspeed
share on X ↗"I just replaced Perplexity ($20/mo) with one prompt"
Don't feel like building it? These folks already made it free.
KhojA personal AI that searches the web and your files; broader than Perplexity, and noticeably heavier to run.open source↗
VanePerplexity with the meter removed: cited answers and deep research, provided you can run one Docker stack.open source↗
GPT ResearcherA research agent rather than a search box; it returns cited reports and a bill from whichever model you chose.open source↗all 5 free alternatives to Perplexity →· no votes, no pay-to-list · just what's real
Perplexity pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| standard | $0 | $0 | 5 Pro Searches/day; 3 file uploads/day; 40 MB/file; practically unlimited basic searches. |
| pro | $20 | $16.67 | Higher Pro Search and file limits; 40 MB/file. |
| education pro | $10/user | — | Discounted Pro access for verified students and educators. |
| max | $200 | $166.67 | 10,000 Computer credits/month. |
| enterprise pro | $40/user | $33.33/user | 500 Computer credits/month; 500 files/project; 15,000 persistent files/user; 100 session uploads/week; 50 MB/file. |
| enterprise max | $325/user | $270.83/user | 15,000 Computer credits/month; 5,000 files/project; 50,000 persistent files/user; 1,000 session uploads/week. |
free tier5 Pro Searches/day; 3 file uploads/day; 40 MB/file
billingmonthly + annual for Pro, Max, and enterprise tiers; Education Pro is monthly; enterprise seats can mix Pro and Max
hidden costsComputer credits expire at the end of each month and do not roll over. Refills and auto-refill are available; 100 credits are priced as $1. Upgrades can be prorated.
verified 2026-08-12 · source ↗
Vibecode Perplexity
Kinda. The core of Perplexity is buildable in a weekend with the prompt on this page, but there are real gaps: search quality, source ranking. Read the honest list above before committing.
How much does Perplexity cost?
Perplexity costs about $20/month (Pro, checked 2026-07-30), which is $240 per year.
What do I lose by replacing Perplexity?
Honestly: search quality; source ranking; model routing; mobile/browser apps; publisher integrations; speed. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Perplexity?
Yes: Khoj (A personal AI that searches the web and your files; broader than Perplexity, and noticeably heavier to run.) Vane (Perplexity with the meter removed: cited answers and deep research, provided you can run one Docker stack.) GPT Researcher (A research agent rather than a search box; it returns cited reports and a bill from whichever model you chose.) All 5 curated free alternatives are at vibecodeit.com/perplexity/alternatives. The prompt is for when you want it exactly your way.