Build Lateral

0%0 of 18 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A local research library whose hard problems are getting citation metadata from a registry rather than from a web page, keeping highlights anchored when a file is replaced, and refusing to answer or to export anything it cannot show you the source of.

estimated effort weekendthe files for this build are in the project pack

StorageFiles in a folder you own with a SQLite index beside themMetadataResolved from an identifier through the registry, with contact details on every requestTextExtraction per page, with the page number keptSearchSQLite full text search over page text, highlights and notesImportLocal files and addresses you can already reach, never an access bypassAIOptional, answering only with a quote and a page number from your own corpus

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the importer, the extractor and the interface.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
installfree
Why Page numbers are the difference between a searchable pile and a citable library, so an extractor that returns one blob of text is the wrong one.
Get it Any maintained extraction library for your runtime that exposes per-page text.
Verify A known PDF yields the correct text for a named page number
decidefree
Why Identifying your client is what gets you into the pool reserved for polite clients and lets the registry warn you instead of blocking you.
Get it Pick an address you actually read and record it in configuration. open ↗
Verify A request carries it and the response includes rate limit headers
have readyfree
Why The scanned paper is the case that silently disappears from search, and a corpus of clean born-digital PDFs will never show you that bug.
Get it Your own reading, including at least one image-only scan and one paper with no identifier.
Verify The corpus contains at least one scan and one source with no identifier
installfree
Why The only way to know a bibliography export is correct is to import it somewhere you did not write.
Get it Any mature open-source reference manager. open ↗
Verify A BibTeX export from this build imports with authors and years intact
API keyfree locally, pennies per question hosted
Why It is optional and the suite must pass without it, so treat it as an addition rather than a dependency.
Get it A local runtime with a downloaded model, or any API key you are content sending paper text to.
Verify One answer returns with quotes and page numbers, and removing it leaves the app working

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- sources(id, kind, title, doi, authors_json, container, year, url, added_at, metadata_state, metadata_source) · state is confirmed, from_registry, from_file or unknown
- files(source_id, path, sha256, mime, pages, extracted_at, extractor_version, ocr_used) · the extractor version matters, because changing it moves the text
- pages(source_id, page_no, text, char_count) · page level, so every result and every citation carries a number
- highlights(id, source_id, page_no, quote, prefix, suffix, comment, colour, created_at) · anchored by the text and its neighbours, never by a character offset
- collections(id, name) and collection_items(collection_id, source_id, added_at)
- import_log(at, input, outcome, reason) · every refusal with its reason, because most failed imports fail for a dull reason worth reading
- answers(question, answer, cited_page_ids_json, model, at) · every answer keeps the pages it quoted
- Nothing whose metadata_state is unknown is ever written into an exported bibliography.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`LIBRARY_DIR`	required	`~/Documents/library`	Where the files live. They stay readable without this application.
`INDEX_DB`	required	`~/.local/share/library/index.db`	Pages, highlights and metadata. Rebuildable from the folder.
`REGISTRY_CONTACT`	required	`you@example.com`	Sent with every metadata request. It is how a registry can warn you rather than block you.
`MIN_CHARS_PER_PAGE`	required	`50`	Below this a page counts as yielding no text and the file is marked as needing recognition.
`OCR_ENABLED`	required	`false`	Optional. Off means scanned files are marked rather than processed.
`AI_ENDPOINT`	optional	`http://localhost:11434`	Optional. Absent means search only, which is a complete mode.
`AI_API_KEY`secret	optional	`sk-...redacted`	Only if the endpoint needs one. Paper text is sent there, so choose deliberately.
`RETRIEVAL_THRESHOLD`	required	`0.35`	Below this the answering panel refuses rather than guesses.

The build, in order

1
Importing what you are allowed to import
A source arrives with metadata from a registry rather than from a page's markup.
1. The registry record is the citation; the page is marketing.
  Files src/import/resolve.tstests/metadata.test.ts
2. It is how the registry can warn you rather than block you.
  Files src/import/registry-client.ts
3. Most import failures are boring and fixable once you can see them.
  Files src/import/log.ts
done when · tick each as it passes
watch out
- Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident.
next phase
2
Text with page numbers, and the scans that have none
Every indexed page reports how much text it actually yielded.
1. The count is what tells an empty page from a blank one.
  Files src/extract/pages.ts
2. A visible state, not a warning nobody reads.
  Files src/extract/quality.tstests/scanned.test.ts
3. Because changing it silently moves every character offset.
  Files src/extract/version.ts
done when · tick each as it passes
watch out
- A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log.
next phase
3
Highlights that survive a better copy of the paper
Replacing a file or an extractor does not move every highlight.
1. Text moves less than offsets do.
  Files src/highlights/anchor.tstests/reanchor.test.ts
2. And treat a failure as something to show, not to hide.
  Files src/highlights/reanchor.ts
3. The person who made the highlight can fix it in seconds.
  Files src/ui/LostHighlights.tsx
done when · tick each as it passes
watch out
- Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed.
next phase
4
Search, and an answer that has to quote
Search reaches every page and note, and an answer without a quote is refused.
1. You will search for something you wrote as often as something you read.
  Files src/search/index.ts
2. The refusal is the feature and it needs a number.
  Files src/ai/answer.tstests/refusal.test.ts
3. A citation you cannot read is just a claim with a number.
  Files src/ui/Answer.tsx
done when · tick each as it passes
watch out
- A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold.
5
Leaving with your bibliography intactproduct builder
Export produces citations you can trust and files you can read without this application.
1. A silent omission is as bad as a wrong entry.
  Files src/export/bibliography.tstests/export.test.ts
2. The same paper arrives twice under two file names.
  Files src/import/dedupe.ts
3. That is the portability claim, so it is a test.
  Files tests/fresh-install.test.ts
done when · tick each as it passes
watch out
- Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning.

what this build does not replace

Any access bypass or automated institutional login.
Citation graph coverage at scale.
Team libraries and shared annotation.
Systematic review screening workflows.
publisher-specific import reliability
citation graph scale
team libraries and institutional access
licensed scholarly metadata

after v1, if you want it

A browser extension that saves an address you can already read, which reuses the same importer
Per-collection exports with a chosen citation style, once the bibliography export has been proved against a reference manager
Local embeddings over the page text, added only after the refusal threshold has been tuned on real questions

Need the files? The project pack on the verdict page hands your agent the whole brief · more read it later.