Build Afforai

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A local document research library whose hard problems are getting text out of a PDF with positions you can point at, fetching correct metadata from the scholarly APIs politely, and never producing a citation that cannot be resolved back to a real span on a real page.

estimated effort multi-daythe files for this build are in the project pack

RuntimeOne local web application with a background worker, no account systemExtractionA PDF text extractor that returns page numbers and character positions, with OCR as an explicit fallbackMetadataCrossref and other open scholarly APIs, identified politely, with the user's value winningRetrievalKeyword search and vector search together over structure-aware chunksVerificationEvery generated claim carries a quote that must match the source text exactlyStorageDocuments and the index on local disk, with an explicit record of what was sent where

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the local application, the worker and the index.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
installfree locally
Why Scanned PDFs are common in older literature and the honest alternative to OCR is refusing the document, which you should still do when OCR is absent.
Get it Tesseract from your package manager, or a hosted OCR you have a key for.
Verify A scanned page produces text with page numbers preserved
API keypence per question
Why Answers and comparisons need one, and it is also the boundary at which document text leaves the machine.
Get it Create a key with its own spending limit.
Verify A short question over one document returns sentences with quotes
have readyfree
Why Two-column papers, scanned reports and a document containing a results table are the three cases every part of this build is judged on.
Get it Open-access papers plus one deliberately scanned document.
Verify The corpus contains a two-column paper, a scanned file and a table
decidefree
Why Whether an unverified claim is ever shown, how long cached extractions are kept, and what may be sent to a hosted model are decisions the code enforces.
Get it Write down all three before the first answer is generated.
Verify Each appears in the configuration and in the README

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- documents(id, title, authors_json, year, doi, source_kind, source_ref, sha256, pages, extraction_state, added_at)
- pages(document_id, page_number, text, char_count, extraction_method) · extraction_method is native or ocr, and an empty page with neither is a failure rather than a blank
- spans(id, document_id, page_number, start_char, end_char, text) · the unit a citation resolves to
- chunks(id, document_id, first_span_id, last_span_id, text, heading_path, token_count, embedding_ref)
- answers(id, question, created_at, model, prompt_tokens, completion_tokens, cost_cents)
- claims(id, answer_id, sentence, document_id, page_number, span_id, quote, verified) · an unverified claim is stored for the record and never shown
- egress(id, at, destination, purpose, bytes, document_ids_json) · what left this machine, when, and why

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`APP_PORT`	required	`3210`	Localhost only. This library holds documents that are often not yours to publish.
`LIBRARY_DIR`	required	`./data/library`	Original files. Deleting a document deletes from here too.
`DATABASE_PATH`	required	`./data/library.db`	Documents, pages, spans, chunks, answers and the egress log.
`CROSSREF_CONTACT`	required	`you@example.com`	Sent with metadata lookups so they reach the polite pool.
`MODEL_API_KEY`secret	optional	`sk-...redacted`	Only used when local-only mode is off. Every call is logged.
`LOCAL_ONLY`	required	`false`	When true, search and citation work and no request leaves the machine.
`MAX_UPLOAD_MB`	required	`200`	A refusal with a reason beats an extraction that runs for an hour.

The build, in order

1
Text out of a PDF, with the page and the position
Every character you index can be pointed at on a page of the original.
1. Everything downstream depends on being able to point at the source.
  Files src/extract.tstests/extract.test.ts
2. An unreadable document must never look like a document that says nothing.
  Files src/extract.ts
3. Read one page of output beside the original before believing any of it.
  Files src/layout.tstests/layout.test.ts
done when · tick each as it passes
watch out
- A two-column paper extracted in raw layout order interleaves the left and right columns line by line, so every chunk is a sentence from one column followed by an unrelated sentence from the other, and both the search results and the quoted citations are nonsense that reads almost like English. Detect columns, extract per column, and check the result by reading one page of output beside the original.
next phase
2
Metadata that is correct, fetched politely
Titles, authors and years come from the record, and the APIs are used the way they ask.
1. Check the current figures rather than copying the ones here.
  Files src/crossref.ts
2. The same paper arrives three times with three filenames.
  Files src/dedupe.tstests/dedupe.test.ts
3. Fetched, typed or unknown, and the user's value wins.
  Files src/metadata.ts
done when · tick each as it passes
watch out
- Guessing metadata from the filename gives you a library where half the entries are called 'main' by an author called 'Elsevier', and once those are in your notes and exports they are extremely hard to find again. Fetch by identifier, keep the source of every field, and mark a document as having unverified metadata rather than inventing a plausible title.
next phase
3
Retrieval you can inspect before you trust it
The passages behind an answer are visible first, and they are whole thoughts.
1. A chunk that cannot resolve to pages cannot be cited.
  Files src/chunk.tstests/chunk.test.ts
2. Exact terms and paraphrases are different queries.
  Files src/retrieve.ts
3. Reading the sources is the fastest way to catch a bad retrieval.
  Files src/views/passages.tsx
done when · tick each as it passes
watch out
- Chunking at a fixed token count cuts tables and numbered findings in half, so a retrieved passage contains the first two rows of a results table and the answer confidently reports a figure whose units were in the row that got cut. Chunk on structure, keep tables whole or mark them as tables, and show the passage to the person before the answer.
next phase
4
Answers pinned to real spans, and comparisons that can say nothing
Every displayed sentence has a quote that exists, and a document that does not address the question says so.
1. Then check it by exact match rather than by similarity.
  Files src/answer.tstests/verify.test.ts
2. A visible drop count tells you when retrieval is failing.
  Files src/answer.ts
3. The alternative is a table that invents agreement.
  Files src/compare.tstests/compare.test.ts
done when · tick each as it passes
watch out
- A comparison table with no way to say 'not stated' pushes the model to fill every cell, so a paper that never mentions the outcome acquires an opinion about it, and the table is exactly the artefact people screenshot into a slide. Make 'not stated' a first-class cell value, verified the same way as a quote, and count it in the summary.
5
Export, deletion, and knowing what left the machineproduct builder
The library walks out in standard formats, deletion is complete, and you can see what was sent to whom.
1. Then import it into one before calling it finished.
  Files src/export.tstests/export.test.ts
2. Then search for a phrase unique to that document and expect nothing.
  Files src/delete.tstests/delete.test.ts
3. It is the answer to the only compliance question that matters here.
  Files src/egress.ts
done when · tick each as it passes
watch out
- Sending chunk text to a hosted model without recording it leaves a researcher unable to answer the only question their ethics committee or their publisher will ask, which is what left the machine and where it went. Log every outbound call with the document identifiers involved, and make local-only mode a switch rather than an aspiration.

what this build does not replace

Automated publisher logins and paywall access.
Licensed metadata and citation graphs.
Shared team libraries with permissions.
Automatic full-text acquisition from arbitrary sites.
licensed scholarly metadata
publisher-specific import reliability
citation graph scale
team libraries and institutional access

after v1, if you want it

A reading view with highlights synced back to spans, which the span table already supports
Local embeddings and a local answer model, once the verification layer has been exercised on real questions
Reference list extraction, which is a second parsing problem and deserves its own phase

Need the files? The project pack on the verdict page hands your agent the whole brief · more read it later.