Vibecode Lateral
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
🪦 Lateral was sunset by its maker on June 26, 2025; no plans are sold anymore. The verdict below is now a post-mortem.
The visible research workspace loop is buildable, but a credible replacement needs more than the first screen. Lateral earns its keep through data, import reliability, so expect a weekend or multi-day build and a narrower personal scope.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Lateral. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Lateral · indie build A local research library whose hard problems are getting citation metadata from a registry rather than from a web page, keeping highlights anchored when a file is replaced, and refusing to answer or to export anything it cannot show you the source of. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Files in a folder you own with a SQLite index beside them | a library you cannot open without this application is a library you will lose | | Metadata | Resolved from an identifier through the registry, with contact details on every request | a citation scraped off a landing page is wrong in ways that propagate into other people's work | | Text | Extraction per page, with the page number kept | a quote without a page number cannot be cited, so it is not really a quote | | Search | SQLite full text search over page text, highlights and notes | a few thousand documents is not a vector database problem | | Import | Local files and addresses you can already reach, never an access bypass | the limit is access you already have, and it is not a technical limit | | AI | Optional, answering only with a quote and a page number from your own corpus | a summary you cannot check is a rumour about a paper | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the importer, the extractor and the interface. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A PDF text extraction library that reports page boundaries** · free - Why: Page numbers are the difference between a searchable pile and a citable library, so an extractor that returns one blob of text is the wrong one. - Get it: Any maintained extraction library for your runtime that exposes per-page text. - Verify: A known PDF yields the correct text for a named page number - [ ] **A contact address to send with registry requests** · free - Why: Identifying your client is what gets you into the pool reserved for polite clients and lets the registry warn you instead of blocking you. - Get it: Pick an address you actually read and record it in configuration. - Verify: A request carries it and the response includes rate limit headers - [ ] **A folder of real papers including a scanned one** · free - Why: The scanned paper is the case that silently disappears from search, and a corpus of clean born-digital PDFs will never show you that bug. - Get it: Your own reading, including at least one image-only scan and one paper with no identifier. - Verify: The corpus contains at least one scan and one source with no identifier - [ ] **A reference manager you can import into** · free - Why: The only way to know a bibliography export is correct is to import it somewhere you did not write. - Get it: Any mature open-source reference manager. - Verify: A BibTeX export from this build imports with authors and years intact - [ ] **A local or hosted model endpoint for the answering panel** (optional) · free locally, pennies per question hosted - Why: It is optional and the suite must pass without it, so treat it as an addition rather than a dependency. - Get it: A local runtime with a downloaded model, or any API key you are content sending paper text to. - Verify: One answer returns with quotes and page numbers, and removing it leaves the app working ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any access bypass or automated institutional login. - Citation graph coverage at scale. - Team libraries and shared annotation. - Systematic review screening workflows. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Lateral, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Lateral The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private research library that imports papers, gets their metadata right, and searches everything you have read to replace Lateral. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Files in a folder you own with a SQLite index beside them · a library you cannot open without this application is a library you will lose - Metadata: Resolved from an identifier through the registry, with contact details on every request · a citation scraped off a landing page is wrong in ways that propagate into other people's work - Text: Extraction per page, with the page number kept · a quote without a page number cannot be cited, so it is not really a quote - Search: SQLite full text search over page text, highlights and notes · a few thousand documents is not a vector database problem - Import: Local files and addresses you can already reach, never an access bypass · the limit is access you already have, and it is not a technical limit - AI: Optional, answering only with a quote and a page number from your own corpus · a summary you cannot check is a rumour about a paper ### Data model (create this before Phase 1) - sources(id, kind, title, doi, authors_json, container, year, url, added_at, metadata_state, metadata_source) · state is confirmed, from_registry, from_file or unknown - files(source_id, path, sha256, mime, pages, extracted_at, extractor_version, ocr_used) · the extractor version matters, because changing it moves the text - pages(source_id, page_no, text, char_count) · page level, so every result and every citation carries a number - highlights(id, source_id, page_no, quote, prefix, suffix, comment, colour, created_at) · anchored by the text and its neighbours, never by a character offset - collections(id, name) and collection_items(collection_id, source_id, added_at) - import_log(at, input, outcome, reason) · every refusal with its reason, because most failed imports fail for a dull reason worth reading - answers(question, answer, cited_page_ids_json, model, at) · every answer keeps the pages it quoted - Nothing whose metadata_state is unknown is ever written into an exported bibliography. ### Phase 1 · Importing what you are allowed to import Build: Accept a local file, an identifier, or an address you can already reach. Resolve the identifier against the metadata registry and take title, authors, container and year from there. Identify your client to the registry with a contact address, either as a query parameter or in the user agent, which is what puts you in the pool reserved for polite clients, and read the rate limit headers the response returns rather than hard-coding a rate. Never attempt to get past an access control of any kind. Trap: Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. Done when: An identifier import produces title, authors, container and year from the registry, tested against a known record. Every registry request carries a contact address and the client honours the rate limit headers the response supplies rather than a fixed interval, tested with a stubbed header. A source with no identifier is stored as unknown and is excluded from the bibliography export, asserted. A failed import writes its reason to the import log rather than disappearing. Do not build yet: No text extraction, highlights, search or export. ### Phase 2 · Text with page numbers, and the scans that have none Build: Extract text per page and store the character count alongside it. Mark a file whose pages fall below a threshold as needing text rather than indexing it as empty, and offer optional recognition for those. Record the extractor and its version so a later re-extraction is detectable. Trap: A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. Done when: An image-only PDF is marked as needing text rather than indexed as empty, tested with a scanned file. Extracted text is stored per page and every search result reports the page it came from. The extractor version is recorded so a re-extraction is detectable and highlights can be re-anchored. A file whose reported page count disagrees with the extracted pages is flagged rather than accepted. Do not build yet: No highlights, search or model answering yet. ### Phase 3 · Highlights that survive a better copy of the paper Build: Anchor each highlight by its quote plus a short prefix and suffix and its page number. After any re-extraction or file replacement, re-anchor by matching that text, and list the highlights that could not be re-anchored instead of dropping or moving them. Trap: Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. Done when: Re-extracting with a different extractor re-anchors every highlight or lists the ones it could not, tested by swapping extractors. A highlight carries its quote, page and neighbouring text and renders identically after a restart. Replacing a file with a higher quality scan preserves at least a stated proportion of highlights and reports the remainder by name. Deleting a source removes its highlights and states how many went. Do not build yet: No search, model answering or export yet. ### Phase 4 · Search, and an answer that has to quote Build: Full text search across pages, highlights and notes, with filters for collection, year and author, returning snippets with page numbers. With a model configured, answer a question only with quotes retrieved from your own corpus, each rendered beside the claim it supports, and refuse when retrieval returns nothing above a stated threshold. Trap: A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. Done when: Search returns results with page numbers and highlights the matched text, tested. A question with no relevant sources is refused with the threshold named rather than answered. Every answer sentence renders beside the quote and page it came from, and following it opens that page. Removing the model configuration leaves search, highlights and export fully working, tested by unsetting it. Do not build yet: No export, deduplication or backup yet. ### Phase 5 · Leaving with your bibliography intact Build: Export the library as BibTeX and as CSL-JSON, excluding every source whose metadata state is unknown and handing back a list of what was left out and why. Detect duplicates by identifier and by file hash and offer a merge rather than a second row. Keep files where they are, add a backup command, and make deletion remove the file, its pages and its highlights together. Trap: Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning. Done when: BibTeX and CSL-JSON exports import cleanly into a reference manager, verified by importing them rather than by reading them. Sources with unknown metadata are excluded and reported by name in the export result. A duplicate import of the same identifier or the same file hash is detected and offered as a merge. A fresh install pointed at the folder rebuilds the index and reproduces the same search results. Do not build yet: Nothing after this. ### Out of scope (and why) - Getting past a paywall or automating an institutional login, which is the part you are not permitted to build and is why the paid tools have publisher agreements instead. - Citation graph coverage of references and citing works, which needs a licensed dataset rather than a clever importer. - Team libraries and shared annotation, which need accounts, permissions and a server holding other people's reading. - Systematic review screening with dual coding and conflict resolution, which is a methodology with published standards rather than a feature. ### README must contain - That citation metadata comes from a registry record, never from a publisher page, and what unknown state means - That requests to the registry carry a contact address and respect the rate limit headers it returns - That an image-only file is marked as needing text rather than indexed as empty - That highlights are anchored by their quote and neighbours, so replacing a file re-anchors rather than moves them - That an export excludes unconfirmed sources and hands back the list of what it left out ===== AGENTS.md ===== # Agent instructions · Lateral indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Files in a folder you own with a SQLite index beside them, Resolved from an identifier through the registry, with contact details on every request, Extraction per page, with the page number kept, SQLite full text search over page text, highlights and notes, Local files and addresses you can already reach, never an access bypass, Optional, answering only with a quote and a page number from your own corpus. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. ===== BUILD_PLAN.md ===== # Build plan · Lateral A local research library whose hard problems are getting citation metadata from a registry rather than from a web page, keeping highlights anchored when a file is replaced, and refusing to answer or to export anything it cannot show you the source of. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Importing what you are allowed to import A source arrives with metadata from a registry rather than from a page's markup. ### Steps 1. Resolve the identifier, do not read the page The registry record is the citation; the page is marketing. Files: `src/import/resolve.ts`, `tests/metadata.test.ts` 2. Identify yourself and read the rate limit headers It is how the registry can warn you rather than block you. Files: `src/import/registry-client.ts` 3. Log every refusal with its reason Most import failures are boring and fixable once you can see them. Files: `src/import/log.ts` ### Done when - [ ] An identifier yields registry metadata - [ ] Requests carry contact details and honour the headers - [ ] Unknown-state sources cannot be exported - [ ] Failed imports record a readable reason ### Watch out - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. ## Phase 2 · Text with page numbers, and the scans that have none Every indexed page reports how much text it actually yielded. ### Steps 1. Store text per page with a character count The count is what tells an empty page from a blank one. Files: `src/extract/pages.ts` 2. Mark low-yield files as needing text A visible state, not a warning nobody reads. Files: `src/extract/quality.ts`, `tests/scanned.test.ts` 3. Record the extractor version Because changing it silently moves every character offset. Files: `src/extract/version.ts` ### Done when - [ ] Image-only files are marked, not silently empty - [ ] Results report a page number - [ ] The extractor version is recorded - [ ] Page count mismatches are flagged ### Watch out - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. ## Phase 3 · Highlights that survive a better copy of the paper Replacing a file or an extractor does not move every highlight. ### Steps 1. Anchor by quote, prefix and suffix Text moves less than offsets do. Files: `src/highlights/anchor.ts`, `tests/reanchor.test.ts` 2. Re-anchor after any re-extraction And treat a failure as something to show, not to hide. Files: `src/highlights/reanchor.ts` 3. List the failures with their quotes The person who made the highlight can fix it in seconds. Files: `src/ui/LostHighlights.tsx` ### Done when - [ ] A different extractor re-anchors or reports - [ ] Highlights render identically after a restart - [ ] A better scan preserves the stated proportion - [ ] Deleting a source reports what it removed ### Watch out - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. ## Phase 4 · Search, and an answer that has to quote Search reaches every page and note, and an answer without a quote is refused. ### Steps 1. Index pages, highlights and notes together You will search for something you wrote as often as something you read. Files: `src/search/index.ts` 2. Refuse below the retrieval threshold The refusal is the feature and it needs a number. Files: `src/ai/answer.ts`, `tests/refusal.test.ts` 3. Render each claim beside its quote A citation you cannot read is just a claim with a number. Files: `src/ui/Answer.tsx` ### Done when - [ ] Results carry page numbers and highlights - [ ] An empty corpus produces a refusal - [ ] Every claim sits beside its quote - [ ] The app is complete with no model ### Watch out - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. ## Not in this build - Any access bypass or automated institutional login. - Citation graph coverage at scale. - Team libraries and shared annotation. - Systematic review screening workflows. ## After v1, if you want it - A browser extension that saves an address you can already read, which reuses the same importer - Per-collection exports with a chosen citation style, once the bibliography export has been proved against a reference manager - Local embeddings over the page text, added only after the refusal threshold has been tuned on real questions ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where the files live. They stay readable without this application. LIBRARY_DIR=~/Documents/library # Required. Pages, highlights and metadata. Rebuildable from the folder. INDEX_DB=~/.local/share/library/index.db # Required. Sent with every metadata request. It is how a registry can warn you rather than block you. REGISTRY_CONTACT=you@example.com # Required. Below this a page counts as yielding no text and the file is marked as needing recognition. MIN_CHARS_PER_PAGE=50 # Required. Optional. Off means scanned files are marked rather than processed. OCR_ENABLED=false # Optional. Optional. Absent means search only, which is a complete mode. AI_ENDPOINT=http://localhost:11434 # Optional · secret. Only if the endpoint needs one. Paper text is sent there, so choose deliberately. AI_API_KEY=sk-...redacted # Required. Below this the answering panel refuses rather than guesses. RETRIEVAL_THRESHOLD=0.35
You are building a lean indie version of Lateral. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Lateral · indie build A local research library whose hard problems are getting citation metadata from a registry rather than from a web page, keeping highlights anchored when a file is replaced, and refusing to answer or to export anything it cannot show you the source of. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Files in a folder you own with a SQLite index beside them | a library you cannot open without this application is a library you will lose | | Metadata | Resolved from an identifier through the registry, with contact details on every request | a citation scraped off a landing page is wrong in ways that propagate into other people's work | | Text | Extraction per page, with the page number kept | a quote without a page number cannot be cited, so it is not really a quote | | Search | SQLite full text search over page text, highlights and notes | a few thousand documents is not a vector database problem | | Import | Local files and addresses you can already reach, never an access bypass | the limit is access you already have, and it is not a technical limit | | AI | Optional, answering only with a quote and a page number from your own corpus | a summary you cannot check is a rumour about a paper | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the importer, the extractor and the interface. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A PDF text extraction library that reports page boundaries** · free - Why: Page numbers are the difference between a searchable pile and a citable library, so an extractor that returns one blob of text is the wrong one. - Get it: Any maintained extraction library for your runtime that exposes per-page text. - Verify: A known PDF yields the correct text for a named page number - [ ] **A contact address to send with registry requests** · free - Why: Identifying your client is what gets you into the pool reserved for polite clients and lets the registry warn you instead of blocking you. - Get it: Pick an address you actually read and record it in configuration. - Verify: A request carries it and the response includes rate limit headers - [ ] **A folder of real papers including a scanned one** · free - Why: The scanned paper is the case that silently disappears from search, and a corpus of clean born-digital PDFs will never show you that bug. - Get it: Your own reading, including at least one image-only scan and one paper with no identifier. - Verify: The corpus contains at least one scan and one source with no identifier - [ ] **A reference manager you can import into** · free - Why: The only way to know a bibliography export is correct is to import it somewhere you did not write. - Get it: Any mature open-source reference manager. - Verify: A BibTeX export from this build imports with authors and years intact - [ ] **A local or hosted model endpoint for the answering panel** (optional) · free locally, pennies per question hosted - Why: It is optional and the suite must pass without it, so treat it as an addition rather than a dependency. - Get it: A local runtime with a downloaded model, or any API key you are content sending paper text to. - Verify: One answer returns with quotes and page numbers, and removing it leaves the app working ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any access bypass or automated institutional login. - Citation graph coverage at scale. - Team libraries and shared annotation. - Systematic review screening workflows. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Lateral, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Lateral The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private research library that imports papers, gets their metadata right, and searches everything you have read to replace Lateral. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Files in a folder you own with a SQLite index beside them · a library you cannot open without this application is a library you will lose - Metadata: Resolved from an identifier through the registry, with contact details on every request · a citation scraped off a landing page is wrong in ways that propagate into other people's work - Text: Extraction per page, with the page number kept · a quote without a page number cannot be cited, so it is not really a quote - Search: SQLite full text search over page text, highlights and notes · a few thousand documents is not a vector database problem - Import: Local files and addresses you can already reach, never an access bypass · the limit is access you already have, and it is not a technical limit - AI: Optional, answering only with a quote and a page number from your own corpus · a summary you cannot check is a rumour about a paper ### Data model (create this before Phase 1) - sources(id, kind, title, doi, authors_json, container, year, url, added_at, metadata_state, metadata_source) · state is confirmed, from_registry, from_file or unknown - files(source_id, path, sha256, mime, pages, extracted_at, extractor_version, ocr_used) · the extractor version matters, because changing it moves the text - pages(source_id, page_no, text, char_count) · page level, so every result and every citation carries a number - highlights(id, source_id, page_no, quote, prefix, suffix, comment, colour, created_at) · anchored by the text and its neighbours, never by a character offset - collections(id, name) and collection_items(collection_id, source_id, added_at) - import_log(at, input, outcome, reason) · every refusal with its reason, because most failed imports fail for a dull reason worth reading - answers(question, answer, cited_page_ids_json, model, at) · every answer keeps the pages it quoted - Nothing whose metadata_state is unknown is ever written into an exported bibliography. ### Phase 1 · Importing what you are allowed to import Build: Accept a local file, an identifier, or an address you can already reach. Resolve the identifier against the metadata registry and take title, authors, container and year from there. Identify your client to the registry with a contact address, either as a query parameter or in the user agent, which is what puts you in the pool reserved for polite clients, and read the rate limit headers the response returns rather than hard-coding a rate. Never attempt to get past an access control of any kind. Trap: Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. Done when: An identifier import produces title, authors, container and year from the registry, tested against a known record. Every registry request carries a contact address and the client honours the rate limit headers the response supplies rather than a fixed interval, tested with a stubbed header. A source with no identifier is stored as unknown and is excluded from the bibliography export, asserted. A failed import writes its reason to the import log rather than disappearing. Do not build yet: No text extraction, highlights, search or export. ### Phase 2 · Text with page numbers, and the scans that have none Build: Extract text per page and store the character count alongside it. Mark a file whose pages fall below a threshold as needing text rather than indexing it as empty, and offer optional recognition for those. Record the extractor and its version so a later re-extraction is detectable. Trap: A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. Done when: An image-only PDF is marked as needing text rather than indexed as empty, tested with a scanned file. Extracted text is stored per page and every search result reports the page it came from. The extractor version is recorded so a re-extraction is detectable and highlights can be re-anchored. A file whose reported page count disagrees with the extracted pages is flagged rather than accepted. Do not build yet: No highlights, search or model answering yet. ### Phase 3 · Highlights that survive a better copy of the paper Build: Anchor each highlight by its quote plus a short prefix and suffix and its page number. After any re-extraction or file replacement, re-anchor by matching that text, and list the highlights that could not be re-anchored instead of dropping or moving them. Trap: Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. Done when: Re-extracting with a different extractor re-anchors every highlight or lists the ones it could not, tested by swapping extractors. A highlight carries its quote, page and neighbouring text and renders identically after a restart. Replacing a file with a higher quality scan preserves at least a stated proportion of highlights and reports the remainder by name. Deleting a source removes its highlights and states how many went. Do not build yet: No search, model answering or export yet. ### Phase 4 · Search, and an answer that has to quote Build: Full text search across pages, highlights and notes, with filters for collection, year and author, returning snippets with page numbers. With a model configured, answer a question only with quotes retrieved from your own corpus, each rendered beside the claim it supports, and refuse when retrieval returns nothing above a stated threshold. Trap: A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. Done when: Search returns results with page numbers and highlights the matched text, tested. A question with no relevant sources is refused with the threshold named rather than answered. Every answer sentence renders beside the quote and page it came from, and following it opens that page. Removing the model configuration leaves search, highlights and export fully working, tested by unsetting it. Do not build yet: No export, deduplication or backup yet. ### Phase 5 · Leaving with your bibliography intact Build: Export the library as BibTeX and as CSL-JSON, excluding every source whose metadata state is unknown and handing back a list of what was left out and why. Detect duplicates by identifier and by file hash and offer a merge rather than a second row. Keep files where they are, add a backup command, and make deletion remove the file, its pages and its highlights together. Trap: Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning. Done when: BibTeX and CSL-JSON exports import cleanly into a reference manager, verified by importing them rather than by reading them. Sources with unknown metadata are excluded and reported by name in the export result. A duplicate import of the same identifier or the same file hash is detected and offered as a merge. A fresh install pointed at the folder rebuilds the index and reproduces the same search results. Do not build yet: Nothing after this. ### Out of scope (and why) - Getting past a paywall or automating an institutional login, which is the part you are not permitted to build and is why the paid tools have publisher agreements instead. - Citation graph coverage of references and citing works, which needs a licensed dataset rather than a clever importer. - Team libraries and shared annotation, which need accounts, permissions and a server holding other people's reading. - Systematic review screening with dual coding and conflict resolution, which is a methodology with published standards rather than a feature. ### README must contain - That citation metadata comes from a registry record, never from a publisher page, and what unknown state means - That requests to the registry carry a contact address and respect the rate limit headers it returns - That an image-only file is marked as needing text rather than indexed as empty - That highlights are anchored by their quote and neighbours, so replacing a file re-anchors rather than moves them - That an export excludes unconfirmed sources and hands back the list of what it left out ===== AGENTS.md ===== # Agent instructions · Lateral indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Files in a folder you own with a SQLite index beside them, Resolved from an identifier through the registry, with contact details on every request, Extraction per page, with the page number kept, SQLite full text search over page text, highlights and notes, Local files and addresses you can already reach, never an access bypass, Optional, answering only with a quote and a page number from your own corpus. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. ===== BUILD_PLAN.md ===== # Build plan · Lateral A local research library whose hard problems are getting citation metadata from a registry rather than from a web page, keeping highlights anchored when a file is replaced, and refusing to answer or to export anything it cannot show you the source of. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Importing what you are allowed to import A source arrives with metadata from a registry rather than from a page's markup. ### Steps 1. Resolve the identifier, do not read the page The registry record is the citation; the page is marketing. Files: `src/import/resolve.ts`, `tests/metadata.test.ts` 2. Identify yourself and read the rate limit headers It is how the registry can warn you rather than block you. Files: `src/import/registry-client.ts` 3. Log every refusal with its reason Most import failures are boring and fixable once you can see them. Files: `src/import/log.ts` ### Done when - [ ] An identifier yields registry metadata - [ ] Requests carry contact details and honour the headers - [ ] Unknown-state sources cannot be exported - [ ] Failed imports record a readable reason ### Watch out - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. ## Phase 2 · Text with page numbers, and the scans that have none Every indexed page reports how much text it actually yielded. ### Steps 1. Store text per page with a character count The count is what tells an empty page from a blank one. Files: `src/extract/pages.ts` 2. Mark low-yield files as needing text A visible state, not a warning nobody reads. Files: `src/extract/quality.ts`, `tests/scanned.test.ts` 3. Record the extractor version Because changing it silently moves every character offset. Files: `src/extract/version.ts` ### Done when - [ ] Image-only files are marked, not silently empty - [ ] Results report a page number - [ ] The extractor version is recorded - [ ] Page count mismatches are flagged ### Watch out - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. ## Phase 3 · Highlights that survive a better copy of the paper Replacing a file or an extractor does not move every highlight. ### Steps 1. Anchor by quote, prefix and suffix Text moves less than offsets do. Files: `src/highlights/anchor.ts`, `tests/reanchor.test.ts` 2. Re-anchor after any re-extraction And treat a failure as something to show, not to hide. Files: `src/highlights/reanchor.ts` 3. List the failures with their quotes The person who made the highlight can fix it in seconds. Files: `src/ui/LostHighlights.tsx` ### Done when - [ ] A different extractor re-anchors or reports - [ ] Highlights render identically after a restart - [ ] A better scan preserves the stated proportion - [ ] Deleting a source reports what it removed ### Watch out - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. ## Phase 4 · Search, and an answer that has to quote Search reaches every page and note, and an answer without a quote is refused. ### Steps 1. Index pages, highlights and notes together You will search for something you wrote as often as something you read. Files: `src/search/index.ts` 2. Refuse below the retrieval threshold The refusal is the feature and it needs a number. Files: `src/ai/answer.ts`, `tests/refusal.test.ts` 3. Render each claim beside its quote A citation you cannot read is just a claim with a number. Files: `src/ui/Answer.tsx` ### Done when - [ ] Results carry page numbers and highlights - [ ] An empty corpus produces a refusal - [ ] Every claim sits beside its quote - [ ] The app is complete with no model ### Watch out - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. ## Not in this build - Any access bypass or automated institutional login. - Citation graph coverage at scale. - Team libraries and shared annotation. - Systematic review screening workflows. ## After v1, if you want it - A browser extension that saves an address you can already read, which reuses the same importer - Per-collection exports with a chosen citation style, once the bibliography export has been proved against a reference manager - Local embeddings over the page text, added only after the refusal threshold has been tuned on real questions ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where the files live. They stay readable without this application. LIBRARY_DIR=~/Documents/library # Required. Pages, highlights and metadata. Rebuildable from the folder. INDEX_DB=~/.local/share/library/index.db # Required. Sent with every metadata request. It is how a registry can warn you rather than block you. REGISTRY_CONTACT=you@example.com # Required. Below this a page counts as yielding no text and the file is marked as needing recognition. MIN_CHARS_PER_PAGE=50 # Required. Optional. Off means scanned files are marked rather than processed. OCR_ENABLED=false # Optional. Optional. Absent means search only, which is a complete mode. AI_ENDPOINT=http://localhost:11434 # Optional · secret. Only if the endpoint needs one. Paper text is sent there, so choose deliberately. AI_API_KEY=sk-...redacted # Required. Below this the answering panel refuses rather than guesses. RETRIEVAL_THRESHOLD=0.35
You are building a production product version of Lateral. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Lateral · product brief ## Problem The visible research workspace loop is buildable, but a credible replacement needs more than the first screen. Lateral earns its keep through data, import reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A library whose citations come from a registry rather than a web page, whose scanned papers announce themselves instead of vanishing from search, whose highlights survive a better copy of the file, and whose export never quietly includes something you never confirmed. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - browser automation for user-authorized imports - optional OpenAI API key stored in .env ## Explicit non-goals for v1 - Any access bypass or automated institutional login. - Citation graph coverage at scale. - Team libraries and shared annotation. - Systematic review screening workflows. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata ## Success criteria - Registry metadata is confirmed used instead of page markup - Requests are confirmed to carry contact details and honour rate limit headers - An image-only file is confirmed marked rather than indexed empty - Highlights are confirmed to re-anchor after an extractor change - Unconfirmed sources are confirmed excluded from export and reported ===== BRIEF.md ===== # Build brief · Lateral The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private research library that imports papers, gets their metadata right, and searches everything you have read to replace Lateral. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Files in a folder you own with a SQLite index beside them · a library you cannot open without this application is a library you will lose - Metadata: Resolved from an identifier through the registry, with contact details on every request · a citation scraped off a landing page is wrong in ways that propagate into other people's work - Text: Extraction per page, with the page number kept · a quote without a page number cannot be cited, so it is not really a quote - Search: SQLite full text search over page text, highlights and notes · a few thousand documents is not a vector database problem - Import: Local files and addresses you can already reach, never an access bypass · the limit is access you already have, and it is not a technical limit - AI: Optional, answering only with a quote and a page number from your own corpus · a summary you cannot check is a rumour about a paper ### Data model (create this before Phase 1) - sources(id, kind, title, doi, authors_json, container, year, url, added_at, metadata_state, metadata_source) · state is confirmed, from_registry, from_file or unknown - files(source_id, path, sha256, mime, pages, extracted_at, extractor_version, ocr_used) · the extractor version matters, because changing it moves the text - pages(source_id, page_no, text, char_count) · page level, so every result and every citation carries a number - highlights(id, source_id, page_no, quote, prefix, suffix, comment, colour, created_at) · anchored by the text and its neighbours, never by a character offset - collections(id, name) and collection_items(collection_id, source_id, added_at) - import_log(at, input, outcome, reason) · every refusal with its reason, because most failed imports fail for a dull reason worth reading - answers(question, answer, cited_page_ids_json, model, at) · every answer keeps the pages it quoted - Nothing whose metadata_state is unknown is ever written into an exported bibliography. ### Phase 1 · Importing what you are allowed to import Build: Accept a local file, an identifier, or an address you can already reach. Resolve the identifier against the metadata registry and take title, authors, container and year from there. Identify your client to the registry with a contact address, either as a query parameter or in the user agent, which is what puts you in the pool reserved for polite clients, and read the rate limit headers the response returns rather than hard-coding a rate. Never attempt to get past an access control of any kind. Trap: Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. Done when: An identifier import produces title, authors, container and year from the registry, tested against a known record. Every registry request carries a contact address and the client honours the rate limit headers the response supplies rather than a fixed interval, tested with a stubbed header. A source with no identifier is stored as unknown and is excluded from the bibliography export, asserted. A failed import writes its reason to the import log rather than disappearing. Do not build yet: No text extraction, highlights, search or export. ### Phase 2 · Text with page numbers, and the scans that have none Build: Extract text per page and store the character count alongside it. Mark a file whose pages fall below a threshold as needing text rather than indexing it as empty, and offer optional recognition for those. Record the extractor and its version so a later re-extraction is detectable. Trap: A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. Done when: An image-only PDF is marked as needing text rather than indexed as empty, tested with a scanned file. Extracted text is stored per page and every search result reports the page it came from. The extractor version is recorded so a re-extraction is detectable and highlights can be re-anchored. A file whose reported page count disagrees with the extracted pages is flagged rather than accepted. Do not build yet: No highlights, search or model answering yet. ### Phase 3 · Highlights that survive a better copy of the paper Build: Anchor each highlight by its quote plus a short prefix and suffix and its page number. After any re-extraction or file replacement, re-anchor by matching that text, and list the highlights that could not be re-anchored instead of dropping or moving them. Trap: Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. Done when: Re-extracting with a different extractor re-anchors every highlight or lists the ones it could not, tested by swapping extractors. A highlight carries its quote, page and neighbouring text and renders identically after a restart. Replacing a file with a higher quality scan preserves at least a stated proportion of highlights and reports the remainder by name. Deleting a source removes its highlights and states how many went. Do not build yet: No search, model answering or export yet. ### Phase 4 · Search, and an answer that has to quote Build: Full text search across pages, highlights and notes, with filters for collection, year and author, returning snippets with page numbers. With a model configured, answer a question only with quotes retrieved from your own corpus, each rendered beside the claim it supports, and refuse when retrieval returns nothing above a stated threshold. Trap: A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. Done when: Search returns results with page numbers and highlights the matched text, tested. A question with no relevant sources is refused with the threshold named rather than answered. Every answer sentence renders beside the quote and page it came from, and following it opens that page. Removing the model configuration leaves search, highlights and export fully working, tested by unsetting it. Do not build yet: No export, deduplication or backup yet. ### Phase 5 · Leaving with your bibliography intact Build: Export the library as BibTeX and as CSL-JSON, excluding every source whose metadata state is unknown and handing back a list of what was left out and why. Detect duplicates by identifier and by file hash and offer a merge rather than a second row. Keep files where they are, add a backup command, and make deletion remove the file, its pages and its highlights together. Trap: Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning. Done when: BibTeX and CSL-JSON exports import cleanly into a reference manager, verified by importing them rather than by reading them. Sources with unknown metadata are excluded and reported by name in the export result. A duplicate import of the same identifier or the same file hash is detected and offered as a merge. A fresh install pointed at the folder rebuilds the index and reproduces the same search results. Do not build yet: Nothing after this. ### Out of scope (and why) - Getting past a paywall or automating an institutional login, which is the part you are not permitted to build and is why the paid tools have publisher agreements instead. - Citation graph coverage of references and citing works, which needs a licensed dataset rather than a clever importer. - Team libraries and shared annotation, which need accounts, permissions and a server holding other people's reading. - Systematic review screening with dual coding and conflict resolution, which is a methodology with published standards rather than a feature. ### README must contain - That citation metadata comes from a registry record, never from a publisher page, and what unknown state means - That requests to the registry carry a contact address and respect the rate limit headers it returns - That an image-only file is marked as needing text rather than indexed as empty - That highlights are anchored by their quote and neighbours, so replacing a file re-anchors rather than moves them - That an export excludes unconfirmed sources and hands back the list of what it left out ===== ARCHITECTURE.md ===== # Architecture · Lateral ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Files in a folder you own with a SQLite index beside them | a library you cannot open without this application is a library you will lose | | Metadata | Resolved from an identifier through the registry, with contact details on every request | a citation scraped off a landing page is wrong in ways that propagate into other people's work | | Text | Extraction per page, with the page number kept | a quote without a page number cannot be cited, so it is not really a quote | | Search | SQLite full text search over page text, highlights and notes | a few thousand documents is not a vector database problem | | Import | Local files and addresses you can already reach, never an access bypass | the limit is access you already have, and it is not a technical limit | | AI | Optional, answering only with a quote and a page number from your own corpus | a summary you cannot check is a rumour about a paper | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Import | Identifier resolution, polite requests and the refusal log. | Any registry, provided the client identifies itself and reads the rate limit headers. | | Extraction | Per-page text, quality marks and extractor versions. | Any extractor, provided page boundaries and character counts survive. | | Highlights | Text anchors and re-anchoring. | Not swappable for character offsets, which move whenever the file does. | | Search | Full text over pages, highlights and notes. | Any index, provided results still carry page numbers. | | Export | Bibliography formats and the exclusion report. | Any format, provided unconfirmed metadata is excluded and listed. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `LIBRARY_DIR` · required · Where the files live. They stay readable without this application. - `INDEX_DB` · required · Pages, highlights and metadata. Rebuildable from the folder. - `REGISTRY_CONTACT` · required · Sent with every metadata request. It is how a registry can warn you rather than block you. - `MIN_CHARS_PER_PAGE` · required · Below this a page counts as yielding no text and the file is marked as needing recognition. - `OCR_ENABLED` · required · Optional. Off means scanned files are marked rather than processed. - `AI_ENDPOINT` · optional · Optional. Absent means search only, which is a complete mode. - `AI_API_KEY` · optional, secret · Only if the endpoint needs one. Paper text is sent there, so choose deliberately. - `RETRIEVAL_THRESHOLD` · required · Below this the answering panel refuses rather than guesses. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Lateral product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Files in a folder you own with a SQLite index beside them, Resolved from an identifier through the registry, with contact details on every request, Extraction per page, with the page number kept, SQLite full text search over page text, highlights and notes, Local files and addresses you can already reach, never an access bypass, Optional, answering only with a quote and a page number from your own corpus. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. - Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning. ===== MILESTONES.md ===== # Delivery milestones · Lateral Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Importing what you are allowed to import A source arrives with metadata from a registry rather than from a page's markup. ### Steps 1. Resolve the identifier, do not read the page The registry record is the citation; the page is marketing. Files: `src/import/resolve.ts`, `tests/metadata.test.ts` 2. Identify yourself and read the rate limit headers It is how the registry can warn you rather than block you. Files: `src/import/registry-client.ts` 3. Log every refusal with its reason Most import failures are boring and fixable once you can see them. Files: `src/import/log.ts` ### Done when - [ ] An identifier yields registry metadata - [ ] Requests carry contact details and honour the headers - [ ] Unknown-state sources cannot be exported - [ ] Failed imports record a readable reason ### Watch out - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. ## M2 · Text with page numbers, and the scans that have none Every indexed page reports how much text it actually yielded. ### Steps 1. Store text per page with a character count The count is what tells an empty page from a blank one. Files: `src/extract/pages.ts` 2. Mark low-yield files as needing text A visible state, not a warning nobody reads. Files: `src/extract/quality.ts`, `tests/scanned.test.ts` 3. Record the extractor version Because changing it silently moves every character offset. Files: `src/extract/version.ts` ### Done when - [ ] Image-only files are marked, not silently empty - [ ] Results report a page number - [ ] The extractor version is recorded - [ ] Page count mismatches are flagged ### Watch out - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. ## M3 · Highlights that survive a better copy of the paper Replacing a file or an extractor does not move every highlight. ### Steps 1. Anchor by quote, prefix and suffix Text moves less than offsets do. Files: `src/highlights/anchor.ts`, `tests/reanchor.test.ts` 2. Re-anchor after any re-extraction And treat a failure as something to show, not to hide. Files: `src/highlights/reanchor.ts` 3. List the failures with their quotes The person who made the highlight can fix it in seconds. Files: `src/ui/LostHighlights.tsx` ### Done when - [ ] A different extractor re-anchors or reports - [ ] Highlights render identically after a restart - [ ] A better scan preserves the stated proportion - [ ] Deleting a source reports what it removed ### Watch out - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. ## M4 · Search, and an answer that has to quote Search reaches every page and note, and an answer without a quote is refused. ### Steps 1. Index pages, highlights and notes together You will search for something you wrote as often as something you read. Files: `src/search/index.ts` 2. Refuse below the retrieval threshold The refusal is the feature and it needs a number. Files: `src/ai/answer.ts`, `tests/refusal.test.ts` 3. Render each claim beside its quote A citation you cannot read is just a claim with a number. Files: `src/ui/Answer.tsx` ### Done when - [ ] Results carry page numbers and highlights - [ ] An empty corpus produces a refusal - [ ] Every claim sits beside its quote - [ ] The app is complete with no model ### Watch out - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. ## M5 · Leaving with your bibliography intact (production only) Export produces citations you can trust and files you can read without this application. ### Steps 1. Exclude unknown metadata and say what was excluded A silent omission is as bad as a wrong entry. Files: `src/export/bibliography.ts`, `tests/export.test.ts` 2. Detect duplicates by identifier and by hash The same paper arrives twice under two file names. Files: `src/import/dedupe.ts` 3. Prove a fresh install rebuilds from the folder That is the portability claim, so it is a test. Files: `tests/fresh-install.test.ts` ### Done when - [ ] Exports import into a reference manager - [ ] Unknown metadata is excluded and reported - [ ] Duplicates are detected and merged - [ ] A fresh install rebuilds from the folder ### Watch out - Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning. ===== OPERATIONS.md ===== # Operations · Lateral ## Backup The library folder and the index together, though only the folder is essential. Highlights and notes live in the index, so a backup of the files alone loses years of reading, which is the opposite of what most people assume. ## Restore Restore the folder and the index, then re-anchor highlights, because a restored file may have come from a different copy. The re-anchor report is the thing to read after any restore. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring For a local tool the checks live in the suite: registry requests carrying contact details, scanned files marked rather than indexed empty, re-anchoring after an extractor change, and an export that excludes unknowns. Count of unknown-state sources belongs in the interface. ## Incident checklist A wrong citation in a manuscript is the failure this build exists to prevent, and the metadata_state column says whether the entry was ever confirmed. A paper you cannot find is nearly always a scan that yielded no text, which the needing-text state makes visible. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Registry metadata is confirmed used instead of page markup - [ ] Requests are confirmed to carry contact details and honour rate limit headers - [ ] An image-only file is confirmed marked rather than indexed empty - [ ] Highlights are confirmed to re-anchor after an extractor change - [ ] Unconfirmed sources are confirmed excluded from export and reported ## Launch constraint Do not market omitted Lateral capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where the files live. They stay readable without this application. LIBRARY_DIR=~/Documents/library # Required. Pages, highlights and metadata. Rebuildable from the folder. INDEX_DB=~/.local/share/library/index.db # Required. Sent with every metadata request. It is how a registry can warn you rather than block you. REGISTRY_CONTACT=you@example.com # Required. Below this a page counts as yielding no text and the file is marked as needing recognition. MIN_CHARS_PER_PAGE=50 # Required. Optional. Off means scanned files are marked rather than processed. OCR_ENABLED=false # Optional. Optional. Absent means search only, which is a complete mode. AI_ENDPOINT=http://localhost:11434 # Optional · secret. Only if the endpoint needs one. Paper text is sent there, so choose deliberately. AI_API_KEY=sk-...redacted # Required. Below this the answering panel refuses rather than guesses. RETRIEVAL_THRESHOLD=0.35
README.md
# Lateral · indie build A local research library whose hard problems are getting citation metadata from a registry rather than from a web page, keeping highlights anchored when a file is replaced, and refusing to answer or to export anything it cannot show you the source of. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Files in a folder you own with a SQLite index beside them | a library you cannot open without this application is a library you will lose | | Metadata | Resolved from an identifier through the registry, with contact details on every request | a citation scraped off a landing page is wrong in ways that propagate into other people's work | | Text | Extraction per page, with the page number kept | a quote without a page number cannot be cited, so it is not really a quote | | Search | SQLite full text search over page text, highlights and notes | a few thousand documents is not a vector database problem | | Import | Local files and addresses you can already reach, never an access bypass | the limit is access you already have, and it is not a technical limit | | AI | Optional, answering only with a quote and a page number from your own corpus | a summary you cannot check is a rumour about a paper | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the importer, the extractor and the interface. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A PDF text extraction library that reports page boundaries** · free - Why: Page numbers are the difference between a searchable pile and a citable library, so an extractor that returns one blob of text is the wrong one. - Get it: Any maintained extraction library for your runtime that exposes per-page text. - Verify: A known PDF yields the correct text for a named page number - [ ] **A contact address to send with registry requests** · free - Why: Identifying your client is what gets you into the pool reserved for polite clients and lets the registry warn you instead of blocking you. - Get it: Pick an address you actually read and record it in configuration. - Verify: A request carries it and the response includes rate limit headers - [ ] **A folder of real papers including a scanned one** · free - Why: The scanned paper is the case that silently disappears from search, and a corpus of clean born-digital PDFs will never show you that bug. - Get it: Your own reading, including at least one image-only scan and one paper with no identifier. - Verify: The corpus contains at least one scan and one source with no identifier - [ ] **A reference manager you can import into** · free - Why: The only way to know a bibliography export is correct is to import it somewhere you did not write. - Get it: Any mature open-source reference manager. - Verify: A BibTeX export from this build imports with authors and years intact - [ ] **A local or hosted model endpoint for the answering panel** (optional) · free locally, pennies per question hosted - Why: It is optional and the suite must pass without it, so treat it as an addition rather than a dependency. - Get it: A local runtime with a downloaded model, or any API key you are content sending paper text to. - Verify: One answer returns with quotes and page numbers, and removing it leaves the app working ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any access bypass or automated institutional login. - Citation graph coverage at scale. - Team libraries and shared annotation. - Systematic review screening workflows. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Lateral, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Lateral The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private research library that imports papers, gets their metadata right, and searches everything you have read to replace Lateral. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Files in a folder you own with a SQLite index beside them · a library you cannot open without this application is a library you will lose - Metadata: Resolved from an identifier through the registry, with contact details on every request · a citation scraped off a landing page is wrong in ways that propagate into other people's work - Text: Extraction per page, with the page number kept · a quote without a page number cannot be cited, so it is not really a quote - Search: SQLite full text search over page text, highlights and notes · a few thousand documents is not a vector database problem - Import: Local files and addresses you can already reach, never an access bypass · the limit is access you already have, and it is not a technical limit - AI: Optional, answering only with a quote and a page number from your own corpus · a summary you cannot check is a rumour about a paper ### Data model (create this before Phase 1) - sources(id, kind, title, doi, authors_json, container, year, url, added_at, metadata_state, metadata_source) · state is confirmed, from_registry, from_file or unknown - files(source_id, path, sha256, mime, pages, extracted_at, extractor_version, ocr_used) · the extractor version matters, because changing it moves the text - pages(source_id, page_no, text, char_count) · page level, so every result and every citation carries a number - highlights(id, source_id, page_no, quote, prefix, suffix, comment, colour, created_at) · anchored by the text and its neighbours, never by a character offset - collections(id, name) and collection_items(collection_id, source_id, added_at) - import_log(at, input, outcome, reason) · every refusal with its reason, because most failed imports fail for a dull reason worth reading - answers(question, answer, cited_page_ids_json, model, at) · every answer keeps the pages it quoted - Nothing whose metadata_state is unknown is ever written into an exported bibliography. ### Phase 1 · Importing what you are allowed to import Build: Accept a local file, an identifier, or an address you can already reach. Resolve the identifier against the metadata registry and take title, authors, container and year from there. Identify your client to the registry with a contact address, either as a query parameter or in the user agent, which is what puts you in the pool reserved for polite clients, and read the rate limit headers the response returns rather than hard-coding a rate. Never attempt to get past an access control of any kind. Trap: Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. Done when: An identifier import produces title, authors, container and year from the registry, tested against a known record. Every registry request carries a contact address and the client honours the rate limit headers the response supplies rather than a fixed interval, tested with a stubbed header. A source with no identifier is stored as unknown and is excluded from the bibliography export, asserted. A failed import writes its reason to the import log rather than disappearing. Do not build yet: No text extraction, highlights, search or export. ### Phase 2 · Text with page numbers, and the scans that have none Build: Extract text per page and store the character count alongside it. Mark a file whose pages fall below a threshold as needing text rather than indexing it as empty, and offer optional recognition for those. Record the extractor and its version so a later re-extraction is detectable. Trap: A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. Done when: An image-only PDF is marked as needing text rather than indexed as empty, tested with a scanned file. Extracted text is stored per page and every search result reports the page it came from. The extractor version is recorded so a re-extraction is detectable and highlights can be re-anchored. A file whose reported page count disagrees with the extracted pages is flagged rather than accepted. Do not build yet: No highlights, search or model answering yet. ### Phase 3 · Highlights that survive a better copy of the paper Build: Anchor each highlight by its quote plus a short prefix and suffix and its page number. After any re-extraction or file replacement, re-anchor by matching that text, and list the highlights that could not be re-anchored instead of dropping or moving them. Trap: Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. Done when: Re-extracting with a different extractor re-anchors every highlight or lists the ones it could not, tested by swapping extractors. A highlight carries its quote, page and neighbouring text and renders identically after a restart. Replacing a file with a higher quality scan preserves at least a stated proportion of highlights and reports the remainder by name. Deleting a source removes its highlights and states how many went. Do not build yet: No search, model answering or export yet. ### Phase 4 · Search, and an answer that has to quote Build: Full text search across pages, highlights and notes, with filters for collection, year and author, returning snippets with page numbers. With a model configured, answer a question only with quotes retrieved from your own corpus, each rendered beside the claim it supports, and refuse when retrieval returns nothing above a stated threshold. Trap: A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. Done when: Search returns results with page numbers and highlights the matched text, tested. A question with no relevant sources is refused with the threshold named rather than answered. Every answer sentence renders beside the quote and page it came from, and following it opens that page. Removing the model configuration leaves search, highlights and export fully working, tested by unsetting it. Do not build yet: No export, deduplication or backup yet. ### Phase 5 · Leaving with your bibliography intact Build: Export the library as BibTeX and as CSL-JSON, excluding every source whose metadata state is unknown and handing back a list of what was left out and why. Detect duplicates by identifier and by file hash and offer a merge rather than a second row. Keep files where they are, add a backup command, and make deletion remove the file, its pages and its highlights together. Trap: Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning. Done when: BibTeX and CSL-JSON exports import cleanly into a reference manager, verified by importing them rather than by reading them. Sources with unknown metadata are excluded and reported by name in the export result. A duplicate import of the same identifier or the same file hash is detected and offered as a merge. A fresh install pointed at the folder rebuilds the index and reproduces the same search results. Do not build yet: Nothing after this. ### Out of scope (and why) - Getting past a paywall or automating an institutional login, which is the part you are not permitted to build and is why the paid tools have publisher agreements instead. - Citation graph coverage of references and citing works, which needs a licensed dataset rather than a clever importer. - Team libraries and shared annotation, which need accounts, permissions and a server holding other people's reading. - Systematic review screening with dual coding and conflict resolution, which is a methodology with published standards rather than a feature. ### README must contain - That citation metadata comes from a registry record, never from a publisher page, and what unknown state means - That requests to the registry carry a contact address and respect the rate limit headers it returns - That an image-only file is marked as needing text rather than indexed as empty - That highlights are anchored by their quote and neighbours, so replacing a file re-anchors rather than moves them - That an export excludes unconfirmed sources and hands back the list of what it left out
AGENTS.md
# Agent instructions · Lateral indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Files in a folder you own with a SQLite index beside them, Resolved from an identifier through the registry, with contact details on every request, Extraction per page, with the page number kept, SQLite full text search over page text, highlights and notes, Local files and addresses you can already reach, never an access bypass, Optional, answering only with a quote and a page number from your own corpus. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold.
BUILD_PLAN.md
# Build plan · Lateral A local research library whose hard problems are getting citation metadata from a registry rather than from a web page, keeping highlights anchored when a file is replaced, and refusing to answer or to export anything it cannot show you the source of. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Importing what you are allowed to import A source arrives with metadata from a registry rather than from a page's markup. ### Steps 1. Resolve the identifier, do not read the page The registry record is the citation; the page is marketing. Files: `src/import/resolve.ts`, `tests/metadata.test.ts` 2. Identify yourself and read the rate limit headers It is how the registry can warn you rather than block you. Files: `src/import/registry-client.ts` 3. Log every refusal with its reason Most import failures are boring and fixable once you can see them. Files: `src/import/log.ts` ### Done when - [ ] An identifier yields registry metadata - [ ] Requests carry contact details and honour the headers - [ ] Unknown-state sources cannot be exported - [ ] Failed imports record a readable reason ### Watch out - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. ## Phase 2 · Text with page numbers, and the scans that have none Every indexed page reports how much text it actually yielded. ### Steps 1. Store text per page with a character count The count is what tells an empty page from a blank one. Files: `src/extract/pages.ts` 2. Mark low-yield files as needing text A visible state, not a warning nobody reads. Files: `src/extract/quality.ts`, `tests/scanned.test.ts` 3. Record the extractor version Because changing it silently moves every character offset. Files: `src/extract/version.ts` ### Done when - [ ] Image-only files are marked, not silently empty - [ ] Results report a page number - [ ] The extractor version is recorded - [ ] Page count mismatches are flagged ### Watch out - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. ## Phase 3 · Highlights that survive a better copy of the paper Replacing a file or an extractor does not move every highlight. ### Steps 1. Anchor by quote, prefix and suffix Text moves less than offsets do. Files: `src/highlights/anchor.ts`, `tests/reanchor.test.ts` 2. Re-anchor after any re-extraction And treat a failure as something to show, not to hide. Files: `src/highlights/reanchor.ts` 3. List the failures with their quotes The person who made the highlight can fix it in seconds. Files: `src/ui/LostHighlights.tsx` ### Done when - [ ] A different extractor re-anchors or reports - [ ] Highlights render identically after a restart - [ ] A better scan preserves the stated proportion - [ ] Deleting a source reports what it removed ### Watch out - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. ## Phase 4 · Search, and an answer that has to quote Search reaches every page and note, and an answer without a quote is refused. ### Steps 1. Index pages, highlights and notes together You will search for something you wrote as often as something you read. Files: `src/search/index.ts` 2. Refuse below the retrieval threshold The refusal is the feature and it needs a number. Files: `src/ai/answer.ts`, `tests/refusal.test.ts` 3. Render each claim beside its quote A citation you cannot read is just a claim with a number. Files: `src/ui/Answer.tsx` ### Done when - [ ] Results carry page numbers and highlights - [ ] An empty corpus produces a refusal - [ ] Every claim sits beside its quote - [ ] The app is complete with no model ### Watch out - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. ## Not in this build - Any access bypass or automated institutional login. - Citation graph coverage at scale. - Team libraries and shared annotation. - Systematic review screening workflows. ## After v1, if you want it - A browser extension that saves an address you can already read, which reuses the same importer - Per-collection exports with a chosen citation style, once the bibliography export has been proved against a reference manager - Local embeddings over the page text, added only after the refusal threshold has been tuned on real questions
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where the files live. They stay readable without this application. LIBRARY_DIR=~/Documents/library # Required. Pages, highlights and metadata. Rebuildable from the folder. INDEX_DB=~/.local/share/library/index.db # Required. Sent with every metadata request. It is how a registry can warn you rather than block you. REGISTRY_CONTACT=you@example.com # Required. Below this a page counts as yielding no text and the file is marked as needing recognition. MIN_CHARS_PER_PAGE=50 # Required. Optional. Off means scanned files are marked rather than processed. OCR_ENABLED=false # Optional. Optional. Absent means search only, which is a complete mode. AI_ENDPOINT=http://localhost:11434 # Optional · secret. Only if the endpoint needs one. Paper text is sent there, so choose deliberately. AI_API_KEY=sk-...redacted # Required. Below this the answering panel refuses rather than guesses. RETRIEVAL_THRESHOLD=0.35
PRODUCT.md
# Lateral · product brief ## Problem The visible research workspace loop is buildable, but a credible replacement needs more than the first screen. Lateral earns its keep through data, import reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A library whose citations come from a registry rather than a web page, whose scanned papers announce themselves instead of vanishing from search, whose highlights survive a better copy of the file, and whose export never quietly includes something you never confirmed. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - browser automation for user-authorized imports - optional OpenAI API key stored in .env ## Explicit non-goals for v1 - Any access bypass or automated institutional login. - Citation graph coverage at scale. - Team libraries and shared annotation. - Systematic review screening workflows. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata ## Success criteria - Registry metadata is confirmed used instead of page markup - Requests are confirmed to carry contact details and honour rate limit headers - An image-only file is confirmed marked rather than indexed empty - Highlights are confirmed to re-anchor after an extractor change - Unconfirmed sources are confirmed excluded from export and reported
BRIEF.md
# Build brief · Lateral The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private research library that imports papers, gets their metadata right, and searches everything you have read to replace Lateral. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Files in a folder you own with a SQLite index beside them · a library you cannot open without this application is a library you will lose - Metadata: Resolved from an identifier through the registry, with contact details on every request · a citation scraped off a landing page is wrong in ways that propagate into other people's work - Text: Extraction per page, with the page number kept · a quote without a page number cannot be cited, so it is not really a quote - Search: SQLite full text search over page text, highlights and notes · a few thousand documents is not a vector database problem - Import: Local files and addresses you can already reach, never an access bypass · the limit is access you already have, and it is not a technical limit - AI: Optional, answering only with a quote and a page number from your own corpus · a summary you cannot check is a rumour about a paper ### Data model (create this before Phase 1) - sources(id, kind, title, doi, authors_json, container, year, url, added_at, metadata_state, metadata_source) · state is confirmed, from_registry, from_file or unknown - files(source_id, path, sha256, mime, pages, extracted_at, extractor_version, ocr_used) · the extractor version matters, because changing it moves the text - pages(source_id, page_no, text, char_count) · page level, so every result and every citation carries a number - highlights(id, source_id, page_no, quote, prefix, suffix, comment, colour, created_at) · anchored by the text and its neighbours, never by a character offset - collections(id, name) and collection_items(collection_id, source_id, added_at) - import_log(at, input, outcome, reason) · every refusal with its reason, because most failed imports fail for a dull reason worth reading - answers(question, answer, cited_page_ids_json, model, at) · every answer keeps the pages it quoted - Nothing whose metadata_state is unknown is ever written into an exported bibliography. ### Phase 1 · Importing what you are allowed to import Build: Accept a local file, an identifier, or an address you can already reach. Resolve the identifier against the metadata registry and take title, authors, container and year from there. Identify your client to the registry with a contact address, either as a query parameter or in the user agent, which is what puts you in the pool reserved for polite clients, and read the rate limit headers the response returns rather than hard-coding a rate. Never attempt to get past an access control of any kind. Trap: Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. Done when: An identifier import produces title, authors, container and year from the registry, tested against a known record. Every registry request carries a contact address and the client honours the rate limit headers the response supplies rather than a fixed interval, tested with a stubbed header. A source with no identifier is stored as unknown and is excluded from the bibliography export, asserted. A failed import writes its reason to the import log rather than disappearing. Do not build yet: No text extraction, highlights, search or export. ### Phase 2 · Text with page numbers, and the scans that have none Build: Extract text per page and store the character count alongside it. Mark a file whose pages fall below a threshold as needing text rather than indexing it as empty, and offer optional recognition for those. Record the extractor and its version so a later re-extraction is detectable. Trap: A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. Done when: An image-only PDF is marked as needing text rather than indexed as empty, tested with a scanned file. Extracted text is stored per page and every search result reports the page it came from. The extractor version is recorded so a re-extraction is detectable and highlights can be re-anchored. A file whose reported page count disagrees with the extracted pages is flagged rather than accepted. Do not build yet: No highlights, search or model answering yet. ### Phase 3 · Highlights that survive a better copy of the paper Build: Anchor each highlight by its quote plus a short prefix and suffix and its page number. After any re-extraction or file replacement, re-anchor by matching that text, and list the highlights that could not be re-anchored instead of dropping or moving them. Trap: Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. Done when: Re-extracting with a different extractor re-anchors every highlight or lists the ones it could not, tested by swapping extractors. A highlight carries its quote, page and neighbouring text and renders identically after a restart. Replacing a file with a higher quality scan preserves at least a stated proportion of highlights and reports the remainder by name. Deleting a source removes its highlights and states how many went. Do not build yet: No search, model answering or export yet. ### Phase 4 · Search, and an answer that has to quote Build: Full text search across pages, highlights and notes, with filters for collection, year and author, returning snippets with page numbers. With a model configured, answer a question only with quotes retrieved from your own corpus, each rendered beside the claim it supports, and refuse when retrieval returns nothing above a stated threshold. Trap: A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. Done when: Search returns results with page numbers and highlights the matched text, tested. A question with no relevant sources is refused with the threshold named rather than answered. Every answer sentence renders beside the quote and page it came from, and following it opens that page. Removing the model configuration leaves search, highlights and export fully working, tested by unsetting it. Do not build yet: No export, deduplication or backup yet. ### Phase 5 · Leaving with your bibliography intact Build: Export the library as BibTeX and as CSL-JSON, excluding every source whose metadata state is unknown and handing back a list of what was left out and why. Detect duplicates by identifier and by file hash and offer a merge rather than a second row. Keep files where they are, add a backup command, and make deletion remove the file, its pages and its highlights together. Trap: Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning. Done when: BibTeX and CSL-JSON exports import cleanly into a reference manager, verified by importing them rather than by reading them. Sources with unknown metadata are excluded and reported by name in the export result. A duplicate import of the same identifier or the same file hash is detected and offered as a merge. A fresh install pointed at the folder rebuilds the index and reproduces the same search results. Do not build yet: Nothing after this. ### Out of scope (and why) - Getting past a paywall or automating an institutional login, which is the part you are not permitted to build and is why the paid tools have publisher agreements instead. - Citation graph coverage of references and citing works, which needs a licensed dataset rather than a clever importer. - Team libraries and shared annotation, which need accounts, permissions and a server holding other people's reading. - Systematic review screening with dual coding and conflict resolution, which is a methodology with published standards rather than a feature. ### README must contain - That citation metadata comes from a registry record, never from a publisher page, and what unknown state means - That requests to the registry carry a contact address and respect the rate limit headers it returns - That an image-only file is marked as needing text rather than indexed as empty - That highlights are anchored by their quote and neighbours, so replacing a file re-anchors rather than moves them - That an export excludes unconfirmed sources and hands back the list of what it left out
ARCHITECTURE.md
# Architecture · Lateral ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Files in a folder you own with a SQLite index beside them | a library you cannot open without this application is a library you will lose | | Metadata | Resolved from an identifier through the registry, with contact details on every request | a citation scraped off a landing page is wrong in ways that propagate into other people's work | | Text | Extraction per page, with the page number kept | a quote without a page number cannot be cited, so it is not really a quote | | Search | SQLite full text search over page text, highlights and notes | a few thousand documents is not a vector database problem | | Import | Local files and addresses you can already reach, never an access bypass | the limit is access you already have, and it is not a technical limit | | AI | Optional, answering only with a quote and a page number from your own corpus | a summary you cannot check is a rumour about a paper | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Import | Identifier resolution, polite requests and the refusal log. | Any registry, provided the client identifies itself and reads the rate limit headers. | | Extraction | Per-page text, quality marks and extractor versions. | Any extractor, provided page boundaries and character counts survive. | | Highlights | Text anchors and re-anchoring. | Not swappable for character offsets, which move whenever the file does. | | Search | Full text over pages, highlights and notes. | Any index, provided results still carry page numbers. | | Export | Bibliography formats and the exclusion report. | Any format, provided unconfirmed metadata is excluded and listed. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `LIBRARY_DIR` · required · Where the files live. They stay readable without this application. - `INDEX_DB` · required · Pages, highlights and metadata. Rebuildable from the folder. - `REGISTRY_CONTACT` · required · Sent with every metadata request. It is how a registry can warn you rather than block you. - `MIN_CHARS_PER_PAGE` · required · Below this a page counts as yielding no text and the file is marked as needing recognition. - `OCR_ENABLED` · required · Optional. Off means scanned files are marked rather than processed. - `AI_ENDPOINT` · optional · Optional. Absent means search only, which is a complete mode. - `AI_API_KEY` · optional, secret · Only if the endpoint needs one. Paper text is sent there, so choose deliberately. - `RETRIEVAL_THRESHOLD` · required · Below this the answering panel refuses rather than guesses. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Lateral product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Files in a folder you own with a SQLite index beside them, Resolved from an identifier through the registry, with contact details on every request, Extraction per page, with the page number kept, SQLite full text search over page text, highlights and notes, Local files and addresses you can already reach, never an access bypass, Optional, answering only with a quote and a page number from your own corpus. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. - Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning.
MILESTONES.md
# Delivery milestones · Lateral Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Importing what you are allowed to import A source arrives with metadata from a registry rather than from a page's markup. ### Steps 1. Resolve the identifier, do not read the page The registry record is the citation; the page is marketing. Files: `src/import/resolve.ts`, `tests/metadata.test.ts` 2. Identify yourself and read the rate limit headers It is how the registry can warn you rather than block you. Files: `src/import/registry-client.ts` 3. Log every refusal with its reason Most import failures are boring and fixable once you can see them. Files: `src/import/log.ts` ### Done when - [ ] An identifier yields registry metadata - [ ] Requests carry contact details and honour the headers - [ ] Unknown-state sources cannot be exported - [ ] Failed imports record a readable reason ### Watch out - Taking a citation from a publisher's landing page puts the journal's name in the title field and the cookie notice in the abstract, and because the result looks like a citation it gets copied into a bibliography and then into somebody else's. Resolve the identifier against the registry, take the metadata from there, and mark anything with no identifier as unknown so it cannot reach an export by accident. ## M2 · Text with page numbers, and the scans that have none Every indexed page reports how much text it actually yielded. ### Steps 1. Store text per page with a character count The count is what tells an empty page from a blank one. Files: `src/extract/pages.ts` 2. Mark low-yield files as needing text A visible state, not a warning nobody reads. Files: `src/extract/quality.ts`, `tests/scanned.test.ts` 3. Record the extractor version Because changing it silently moves every character offset. Files: `src/extract/version.ts` ### Done when - [ ] Image-only files are marked, not silently empty - [ ] Results report a page number - [ ] The extractor version is recorded - [ ] Page count mismatches are flagged ### Watch out - A scanned PDF extracts zero characters, is indexed as an empty document, and then quietly fails to appear in any search while still sitting in the library looking perfectly imported, so the one paper you clearly remember reading is the one you can never find again. Count characters per page, mark anything below the threshold as needing text, and put that state in the list rather than in a log. ## M3 · Highlights that survive a better copy of the paper Replacing a file or an extractor does not move every highlight. ### Steps 1. Anchor by quote, prefix and suffix Text moves less than offsets do. Files: `src/highlights/anchor.ts`, `tests/reanchor.test.ts` 2. Re-anchor after any re-extraction And treat a failure as something to show, not to hide. Files: `src/highlights/reanchor.ts` 3. List the failures with their quotes The person who made the highlight can fix it in seconds. Files: `src/ui/LostHighlights.tsx` ### Done when - [ ] A different extractor re-anchors or reports - [ ] Highlights render identically after a restart - [ ] A better scan preserves the stated proportion - [ ] Deleting a source reports what it removed ### Watch out - Anchoring a highlight to a character offset means replacing a PDF with a cleaner scan, or upgrading the extraction library, shifts every highlight by a few hundred characters at once, silently, so your quotes now point at the wrong sentences and you find out when you cite one. Store the quote with its neighbouring text, re-anchor by matching, and show what failed. ## M4 · Search, and an answer that has to quote Search reaches every page and note, and an answer without a quote is refused. ### Steps 1. Index pages, highlights and notes together You will search for something you wrote as often as something you read. Files: `src/search/index.ts` 2. Refuse below the retrieval threshold The refusal is the feature and it needs a number. Files: `src/ai/answer.ts`, `tests/refusal.test.ts` 3. Render each claim beside its quote A citation you cannot read is just a claim with a number. Files: `src/ui/Answer.tsx` ### Done when - [ ] Results carry page numbers and highlights - [ ] An empty corpus produces a refusal - [ ] Every claim sits beside its quote - [ ] The app is complete with no model ### Watch out - A model asked to summarise a paper from two retrieved paragraphs produces a confident finding the paper does not contain, and in a literature review that error is copied into a table and then into a manuscript, where it is somebody else's problem for years. Put every claim beside the quote and page it came from, and refuse to answer below the retrieval threshold. ## M5 · Leaving with your bibliography intact (production only) Export produces citations you can trust and files you can read without this application. ### Steps 1. Exclude unknown metadata and say what was excluded A silent omission is as bad as a wrong entry. Files: `src/export/bibliography.ts`, `tests/export.test.ts` 2. Detect duplicates by identifier and by hash The same paper arrives twice under two file names. Files: `src/import/dedupe.ts` 3. Prove a fresh install rebuilds from the folder That is the portability claim, so it is a test. Files: `tests/fresh-install.test.ts` ### Done when - [ ] Exports import into a reference manager - [ ] Unknown metadata is excluded and reported - [ ] Duplicates are detected and merged - [ ] A fresh install rebuilds from the folder ### Watch out - Exporting a bibliography that quietly includes the sources whose metadata you never confirmed puts a wrong author and a wrong year into a manuscript, and if you are lucky the correction happens at the proof stage rather than after publication. Exclude unknown-state sources and return the list of exclusions as part of the export rather than as a warning.
OPERATIONS.md
# Operations · Lateral ## Backup The library folder and the index together, though only the folder is essential. Highlights and notes live in the index, so a backup of the files alone loses years of reading, which is the opposite of what most people assume. ## Restore Restore the folder and the index, then re-anchor highlights, because a restored file may have come from a different copy. The re-anchor report is the thing to read after any restore. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring For a local tool the checks live in the suite: registry requests carrying contact details, scanned files marked rather than indexed empty, re-anchoring after an extractor change, and an export that excludes unknowns. Count of unknown-state sources belongs in the interface. ## Incident checklist A wrong citation in a manuscript is the failure this build exists to prevent, and the metadata_state column says whether the entry was ever confirmed. A paper you cannot find is nearly always a scan that yielded no text, which the needing-text state makes visible. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Registry metadata is confirmed used instead of page markup - [ ] Requests are confirmed to carry contact details and honour rate limit headers - [ ] An image-only file is confirmed marked rather than indexed empty - [ ] Highlights are confirmed to re-anchor after an extractor change - [ ] Unconfirmed sources are confirmed excluded from export and reported ## Launch constraint Do not market omitted Lateral capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where the files live. They stay readable without this application. LIBRARY_DIR=~/Documents/library # Required. Pages, highlights and metadata. Rebuildable from the folder. INDEX_DB=~/.local/share/library/index.db # Required. Sent with every metadata request. It is how a registry can warn you rather than block you. REGISTRY_CONTACT=you@example.com # Required. Below this a page counts as yielding no text and the file is marked as needing recognition. MIN_CHARS_PER_PAGE=50 # Required. Optional. Off means scanned files are marked rather than processed. OCR_ENABLED=false # Optional. Optional. Absent means search only, which is a complete mode. AI_ENDPOINT=http://localhost:11434 # Optional · secret. Only if the endpoint needs one. Paper text is sent there, so choose deliberately. AI_API_KEY=sk-...redacted # Required. Below this the answering panel refuses rather than guesses. RETRIEVAL_THRESHOLD=0.35
$ choose a build depth, inspect the files, then open the complete pack in your agent
Lateral: Researchers pay for correct metadata, resilient importers, citation coverage, and workflows that survive publisher and browser changes.
xpublisher-specific import reliability
xcitation graph scale
xteam libraries and institutional access
xlicensed scholarly metadata
share on X ↗"I just replaced Lateral with one prompt"
Don't feel like building it? These folks already made it free.
OOpen NotebookSources, notes, search and grounded chat in one self-hosted notebook.open source↗CCADIMAA sober evidence-organizing workflow for literature reviews, without the AI garnish.free↗
NotebookLMRead, compare and question a paper set with citations; screening remains manual.free↗all 3 free alternatives to Lateral →· no votes, no pay-to-list · just what's real
Vibecode Lateral
Kinda. The core of Lateral is buildable in a weekend with the prompt on this page, but there are real gaps: publisher-specific import reliability, citation graph scale. Read the honest list above before committing.
How much does Lateral cost?
Nothing anymore: Lateral was sunset by its maker on June 26, 2025; no plans are sold anymore.
What do I lose by replacing Lateral?
Honestly: publisher-specific import reliability; citation graph scale; team libraries and institutional access; licensed scholarly metadata. Since the product is gone, these gaps are now everyone's problem.
Is there an open-source alternative to Lateral?
Yes: Open Notebook (Sources, notes, search and grounded chat in one self-hosted notebook.) CADIMA (A sober evidence-organizing workflow for literature reviews, without the AI garnish.) NotebookLM (Read, compare and question a paper set with citations; screening remains manual.) All 3 curated free alternatives are at vibecodeit.com/lateral/alternatives. The prompt is for when you want it exactly your way.