Build Genei

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A personal research library whose hard problems are pulling readable text out of documents designed for print, getting bibliographic metadata from a source rather than a guess, and making every generated sentence traceable to a passage.

estimated effort weekendthe files for this build are in the project pack

RuntimeTypeScript on Node 22 with SQLite, run locallyOriginalsKept byte for byte on disk, hashed, and never the thing that gets rewrittenExtractionLayout-aware text with character offsets preserved per pageMetadataFetched from a bibliographic service and stored per field with its sourceSummariesExtractive first, generated second, and always span-anchored

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the importer, the index and the interface locally.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
have readyfree
Why Every extraction bug in this category is a layout bug, and a folder of clean single-column articles will pass tests that a scanned appendix or a two-column paper fails.
Get it Your own PDFs and EPUBs, including at least one two-column paper, one scan and one long book.
Verify All twenty import and the two-column one extracts in reading order
have readyfree
Why The bibliographic API asks callers to identify themselves and offers better service to those who do, and it costs nothing to comply.
Get it An address you monitor, sent as a mailto parameter or in the User-Agent as the service documents. open ↗
Verify A lookup succeeds and your address appears in the request
installfree
Why The citation export is only credible if a real reference manager reads it, and format bugs are invisible until you try.
Get it Any mainstream open-source reference manager. open ↗
Verify Exported BibTeX and CSL-JSON both import with fields intact
API keypennies per document, capped by your configured ceiling
Why The generated summary path is optional by design, and the extractive path has to work without it, so this is genuinely not required to finish the build.
Get it Any provider whose API returns structured output, so spans can be part of the contract.
Verify A summarisation returns sentences with offsets that resolve

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- documents(id, path, sha256, kind, imported_at, title, page_count, text, extractor, extractor_version)
- pages(document_id, page_number, start_offset, end_offset) · so a character offset resolves to a page number a reader can turn to
- works(document_id, doi, isbn, title, container, year, publisher, url, open_access_url) · the bibliographic identity, once found
- work_fields(document_id, field, value, source, fetched_at) · every field records where it came from, including manual
- notes(id, document_id, start_offset, end_offset, body, created_at) · your own writing, anchored like everything else
- claims(id, document_id, summary_id, sentence, start_offset, end_offset, method) · method is extracted or generated, and a generated claim without offsets is never stored
- A summary is a list of claims, each pointing at a character range in documents.text, which the interface renders as a highlight on the stored document.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`LIBRARY_DIR`	required	`./data/library`	Original files, byte for byte as imported.
`DATABASE_PATH`	required	`./data/library.db`	Documents, metadata, notes and claims with their offsets.
`CROSSREF_MAILTO`	required	`you@example.com`	Sent with metadata lookups as the service's documented courtesy convention.
`METADATA_MAX_RETRIES`	required	`4`	Bounded backoff on rate limiting. Never an unlimited retry loop.
`SUMMARY_MODEL`	required	`none`	Set to none for extractive summaries only, which is the default and works offline.
`SUMMARY_API_KEY`secret	optional	`sk-...redacted`	Only when a generated summary model is configured.
`MONTHLY_SPEND_CEILING`	optional	`10.00`	Model spend per month. Reaching it refuses further paid calls.

The build, in order

1
Keeping the document and knowing where you are in it
Text comes out with offsets that map back to a page and a highlight.
1. The failure that gets blamed on the model later.
  Files src/extract/pdf.tstests/two-column.test.ts
2. An offset that maps to a page number is a checkable quote.
  Files src/extract/pages.tssrc/db/schema.sql
3. It is the evidence behind everything the app later says.
  Files src/store/originals.ts
done when · tick each as it passes
watch out
- Naive PDF extraction reads a two-column paper across the columns, so sentences interleave and every quote you later pull is half from one column and half from the other, which looks like the model hallucinating when it is actually your importer. Extract with layout awareness, check a two-column paper by eye during this phase, and store the page offsets so any quote can be checked against the printed page.
next phase
2
Finding out what the thing actually is
Bibliographic fields come from a service or from you, and each one says which.
1. Provenance per field is what makes a citation defensible.
  Files src/metadata/crossref.tssrc/db/schema.sql
2. The documented courtesy, and it gets you the faster pool.
  Files src/metadata/client.tstests/backoff.test.ts
3. A test over the write paths, not a promise in a comment.
  Files tests/no-generated-metadata.test.ts
done when · tick each as it passes
watch out
- Filling in missing authors or a year by asking a language model produces a citation that looks right, passes review, and is wrong, and the person who finds out is a reviewer rather than you. Leave the field empty, mark it as needing attention, and make the citation export refuse to emit a reference with an unsourced required field.
next phase
3
Finding the passage you remember reading
Search covers the whole corpus, returns the passage in context, and filters on metadata.
1. The passage is the thing being searched for.
  Files src/search/passages.ts
2. Author and year are how researchers actually narrow.
  Files src/search/filters.ts
3. A local corpus should stay local.
  Files tests/offline-search.test.ts
done when · tick each as it passes
watch out
- Returning a list of matching documents rather than matching passages means every search ends with you opening a ninety page PDF and using the reader's own find function, which is the workflow the tool was supposed to replace. Return ranked passages with their offsets, and make clicking one open the document scrolled to the highlight.
next phase
4
Summaries that point at the page
Every sentence shown is either a quotation or a generated sentence with a span behind it.
1. It works offline, costs nothing, and sets the anchoring contract.
  Files src/summary/extractive.ts
2. The characters are there or the sentence does not ship.
  Files src/summary/verify.tstests/span-verify.test.ts
3. A corpus-wide summarisation is a real invoice.
  Files src/summary/estimate.ts
done when · tick each as it passes
watch out
- Asking a model for a summary and then trying to attach citations afterwards by searching for similar text produces confident-looking anchors on sentences that were invented, because near-enough matching always finds something. Require the spans as part of the generation contract, verify that the quoted characters actually appear at those offsets, and discard any sentence that fails rather than repairing it.
5
Getting citations out, and getting everything outproduct builder
References export in formats a reference manager reads, and the library is portable.
1. Refuse rather than fill. This is the rule with teeth.
  Files src/export/bibtex.tssrc/export/csl.tstests/missing-field.test.ts
2. An export that does not reimport is a filename, not an exit.
  Files src/export/library.tstests/roundtrip.test.ts
3. The cost page is the only one that gets read later.
  Files src/cost/ledger.ts
done when · tick each as it passes
watch out
- An export that silently omits a missing year or invents a container title produces a bibliography that compiles and is wrong in a way nobody notices until a copy editor does. Refuse the reference, list which fields are missing and for which document, and make fixing them a visible task in the interface rather than an error in a log.

what this build does not replace

Licensed scholarly metadata and citation graphs.
Logged-in automation against publisher sites.
Paywall circumvention of any kind.
Team libraries and shared annotation.
licensed scholarly metadata
publisher-specific import reliability
citation graph scale
team libraries and institutional access

after v1, if you want it

Optical character recognition for scans, offered per document from the failure list
A second metadata service as a fallback, recorded with its own source name per field
Cross-document claim search, once enough summaries exist for it to be worth ranking

Need the files? The project pack on the verdict page hands your agent the whole brief · more read it later.