Build Perplexity

KINDAreplaces $20/mosaves $240/yrback to the verdict

0%0 of 20 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A self-hosted answer engine: a SearXNG instance you control providing results, a fetcher that extracts readable text under strict caps, passage ranking, an answer step whose citations are validated against the passages that were actually retrieved, and threads with follow-up questions that carry context.

estimated effort multi-daythe files for this build are in the project pack

SearchSearXNG, self-hosted, with the JSON output format enabledRuntimePython 3.12 with FastAPIExtractiontrafilaturaRankingBM25 over sentence-window passages, with an optional cross-encoder rerankerAnsweringAny chat model through its API, or a local model through OllamaStorageSQLite with full text search

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree for personal use; check Docker Desktop's licence if this is for a company
Why Runs SearXNG. Installing SearXNG without containers is possible and is a worse afternoon.
Get it Install Docker Desktop, or Docker Engine plus the compose plugin on Linux. open ↗
Verify docker compose version prints v2 or higher
installfree
Why Runs the pipeline.
Get it python.org or your package manager, then create a virtual environment in the project directory.
Verify python3 --version prints 3.12 or higher
API keypay per token; a short answer over five passages is typically well under a cent, and long threads add up faster than you expect
Why The answer step needs a model. Everything before it works without one, which is useful while you are building.
Get it Create an account with a provider and generate a key in their console, and set a spend limit at the same time. Or install Ollama and pull a model if you would rather pay nothing and accept weaker answers.
Verify A single test completion returns text and appears in the provider usage dashboard
decidefree, but you carry the upstream engines' rate limits yourself
Why Public instances block automated traffic and rate-limit, so building against one produces a system that fails as soon as it works.
Get it Run the official container locally, then enable json in the search formats of settings.yml and restart. open ↗
Verify curl with format=json returns JSON rather than a 403
have readyfree
Why Ranking changes feel better and are often worse. Without a fixed set you are tuning on vibes.
Get it Write them yourself from questions you actually asked this week, and record the url that genuinely answers each. An hour of work that pays for itself in Phase 3.

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- threads(id, title, created_at)
- queries(id, thread_id, text, rewritten_text, asked_at)
- results(id, query_id, url, title, snippet, engine, rank)
- documents(id, url, fetched_at, status, content_hash, text, byte_count) · the cache, keyed by url
- passages(id, document_id, ordinal, text, score) · what the answer is allowed to cite
- answers(id, query_id, text, model, citations_json, validated)
- Nothing may be cited that is not a row in passages for that query. That invariant is the whole point of Phase 4.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`SEARXNG_URL`	required	`http://localhost:8080`	Your own instance. Not a public one.
`FETCH_TIMEOUT_MS`	required	`4000`	Per-request timeout. Slow pages are dropped rather than allowed to hold up the answer.
`FETCH_TOTAL_BUDGET_MS`	required	`12000`	The whole retrieval step's deadline. Partial results are expected and correct.
`FETCH_MAX_BYTES`	required	`2000000`	Streaming byte cap per page. Enforced during download, not after.
`ANSWER_MODEL`	required	`claude-sonnet-5`	The model used for the answer step. The rewrite step can use something smaller.
`MODEL_API_KEY`secret	required	`sk-...`	Your provider console. Set a hard spend limit there as well as in this application.
`MONTHLY_CAP_USD`	required	`25`	The application stops answering at this figure rather than continuing to spend.
`DB_PATH`	required	`./data/answers.db`	SQLite file holding threads, cache and passages. This is the whole backup.

The build, in order

1
Search you control
A query returns structured results from your own SearXNG, not from scraping someone's results page.
1. Bring it up with Docker, then edit settings.yml so the search formats include json alongside html, and restart. This single line is the most common thing people miss.
  Files searxng/settings.yml
  terminal
  docker run -d -p 8080:8080 -v ./searxng:/etc/searxng searxng/searxng curl 'http://localhost:8080/search?q=test&format=json'
2. Query with a timeout, normalise into your results shape, and keep the engine name so you can tell later which upstream is producing rubbish.
  Files search.py
3. Key on the normalised query text with a short expiry. This is for your own iteration speed as much as for politeness.
  Files cache.py
done when · tick each as it passes
watch out
- SearXNG serves HTML only until you add json to the formats list in settings.yml. The API returns a 403 until you do, and the error does not explain itself.
- Do not point this at a public SearXNG instance. Public instances rate-limit and block automated traffic, and using one as your backend makes your reliability someone else's volunteer problem.
next phase
2
Fetch and extract, with limits that hold
Pages become clean text without a slow or hostile page being able to hang the request.
1. Bounded concurrency, a per-request timeout, and an overall deadline for the batch. Partial results are the expected outcome, not a failure.
  Files fetch.py
2. Read in chunks and stop at the cap. Record the byte count so the check in this phase is measurable rather than asserted.
  Files fetch.py
3. trafilatura for the text. Hash the extracted text; if it matches the cached hash, keep the existing passages and skip re-chunking.
  Files extract.py
  terminal
  pip install trafilatura httpx
done when · tick each as it passes
watch out
- A byte cap enforced after the download completes is not a cap. Stream the response and abort when the limit is passed, or one large file will happily consume your memory and your time budget.
next phase
3
Passages and ranking
The text handed to the model is the part of the page that answers the question, not the whole page.
1. Split on sentence boundaries, group into windows of a few sentences with an overlap so an answer spanning a boundary is not lost.
  Files chunk.py
2. One corpus for the whole query, not one per document. Store the score on the passage row.
  Files rank.py
  terminal
  pip install rank-bm25
3. Ten queries with a known correct source. Measure whether the right passage lands in the top five, before and after any change to chunking or scoring.
  Files eval/queries.json
done when · tick each as it passes
watch out
- Taking the best passages per document rather than across all documents guarantees you include weak passages from weak sources just because they were the best that page had. Rank globally and let a strong page contribute three passages while a weak one contributes none.
next phase
4
Answers whose citations are checked
Every citation points at a passage that was actually retrieved, and an answer that invents one is rejected.
1. Pass passages as a numbered list and instruct the model to cite only those numbers. Ask for the answer and the citation list as structured output so parsing is not a regular expression.
  Files answer.py
2. A pure function from answer plus passage ids to a verdict. Unit test it with a crafted out-of-range citation; do not wait for a model to produce one.
  Files validate.pytests/test_validate.py
3. If the top passages do not support an answer, say so. This is the behaviour that makes the tool trustworthy, and it needs to be an instruction and a test, not a hope.
  Files answer.py
done when · tick each as it passes
watch out
- Models cite confidently and sometimes cite a source they were never given, especially when the passages do not contain the answer. Validation is not defensive programming here, it is the feature that separates this from a chatbot with a search box.
next phase
5
Threads and follow-ups
A follow-up question understands the conversation without re-searching everything from scratch.
1. Store enough that a reopened thread renders from the database with no network access at all.
  Files store.py
2. A small model call that takes the thread and the new question and returns one search query. Show it in the interface.
  Files rewrite.py
3. If a follow-up retrieves a url already fetched in this thread and it is still fresh, use the cached text.
  Files cache.py
done when · tick each as it passes
watch out
- Passing raw conversation history to the search engine produces terrible results, because search engines want a query and not a dialogue. Rewrite to a standalone question first, and display the rewrite so a wrong one is visible rather than mysterious.
6
Limits, cost and running it for othersproduct builder
Exposing it beyond your own machine does not hand anyone your model budget.
1. Session or token authentication checked in the handler that triggers retrieval, not merely on the page that renders the form.
  Files auth.py
2. Record tokens and cost per query. Compare against a monthly cap and stop serving answers when it is reached, with a message saying why.
  Files meter.py
3. Caddy in front for certificates. A scheduled copy of the SQLite file off the machine, and one restore performed on purpose.
  Files Caddyfile
done when · tick each as it passes
watch out
- An answer engine with an open endpoint is a free API for anyone who finds it, billed to your key. Rate limiting by IP alone is not enough; require authentication before the first model call, not before the first page view.

what this build does not replace

A web index of your own, which is the moat and stays the moat.
Search quality tuned by a team with query logs at scale.
Model routing and the credit allowance a subscription includes.
Polished mobile and browser applications.
search quality
source ranking
model routing
mobile/browser apps
publisher integrations
speed

after v1, if you want it

A cross-encoder reranker, added only after the evaluation set says BM25 is the bottleneck
Restricting a thread to a set of domains you trust for a given topic
Scheduled queries that email you when the answer to a standing question changes

Need the files? The project pack on the verdict page hands your agent the whole brief · more ai search.