Vibecode GoodLinks
track this build4 phases, 12 steps, beginner friendly0%YES · one-shottable
The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For GoodLinks, save readable pages locally across Apple devices with tags and highlights. The hard boundary is native apple polish, icloud sync, shortcuts, and mobile capture, plus capture polish, sync, and content partnerships.
the project pack5 files · written for this build, step by step
You are building a lean indie version of GoodLinks. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # GoodLinks · indie build A read-later archive whose hard problems are keeping something readable when the page is gone, deciding when two addresses are the same article without breaking the ones that are not, and highlights that survive the page being rewritten. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Next.js 15 with TypeScript and a local background worker | fetching has to keep happening while you are reading something else | | Storage | SQLite for metadata, files on disk for raw bytes and extracted text | the raw bytes are the archive and a database row is an index over it | | Extraction | A maintained readability implementation, versioned and re-runnable | extraction improves, so it must be something you can redo without refetching | | Search | Full-text search over extracted text, rebuildable from the files | an index you can throw away is an index you can fix | | Fetching | Conditional requests, per-host concurrency limits and a clear identifying agent | an archive that gets itself blocked stops being an archive without telling you | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the application, the worker and the consistency check. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **Somewhere with room to keep raw pages** · free, if you already back up a disk - Why: Keeping the bytes is the entire premise, and the archive grows steadily rather than dramatically, so the number to know is the average page size times your saving rate. - Get it: A directory on a disk you back up, sized from a week of real saving. - Verify: The disk report shows the total and the largest items - [ ] **The agent string and the contact address in it** · free - Why: An archive that fetches anonymously gives a site operator no way to reach you and no reason not to block you. - Get it: Decide the name and a contact address, and put both in the configuration rather than the code. - Verify: A request log on a site you control shows the string and the address - [ ] **Your existing subscriptions as OPML** (optional) · free - Why: Feed handling only shows its problems on real feeds, where one is gone, one redirects and one ignores conditional requests entirely. - Get it: Export OPML from whatever reader you use now. - Verify: Import produces the expected feed count and the first poll succeeds ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Paywall or authentication bypass. - Native mobile clients with background sync. - Licensed full-text feeds and recommendations. - Full-page archival with every asset. - native Apple polish, iCloud sync, Shortcuts, and mobile capture - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds If one of those is essential to you, that is the reason to keep paying for GoodLinks, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · GoodLinks The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal reading archive that keeps a copy of the page rather than a link to it to replace GoodLinks. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Next.js 15 with TypeScript and a local background worker · fetching has to keep happening while you are reading something else - Storage: SQLite for metadata, files on disk for raw bytes and extracted text · the raw bytes are the archive and a database row is an index over it - Extraction: A maintained readability implementation, versioned and re-runnable · extraction improves, so it must be something you can redo without refetching - Search: Full-text search over extracted text, rebuildable from the files · an index you can throw away is an index you can fix - Fetching: Conditional requests, per-host concurrency limits and a clear identifying agent · an archive that gets itself blocked stops being an archive without telling you ### Data model (create this before Phase 1) - items(id, url, canonical_url, url_key, title, author, published_at, saved_at, state, read_at, source) · url_key is the normalised key and is the only thing deduplication looks at - captures(id, item_id, at, http_status, etag, last_modified, raw_path, extracted_path, extractor_version, word_count, failed_reason) · more than one per item over time - highlights(id, item_id, capture_id, quote, start_offset, end_offset, note, at, orphaned_at) · anchored to a capture, because the text changes underneath you - tags(id, name), item_tags(item_id, tag_id) · flat, because a hierarchy is a second product - feeds(id, url, title, etag, last_modified, last_polled_at, consecutive_failures, disabled_at) - merges(id, kept_item_id, merged_item_id, at, undone_at) · merging two items is reversible because deduplication is a guess - The raw bytes of every capture are kept. The extractor will get better and the page will not be there to fetch again. ### Phase 1 · Save the page, not the link Build: On save, fetch the page, store the raw response bytes on disk, record the status and the validators, then run extraction into a separate artefact. An extraction failure still leaves a saved item backed by the raw copy, marked as failed rather than hidden. Re-running extraction with a newer version creates a new artefact without refetching. Trap: Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. Done when: Every successful capture stores the raw response bytes and a separate extracted artefact, checked on disk. An extraction failure produces a saved item that still opens, marked failed with its reason, tested with a page the extractor cannot handle. Re-running extraction at a new version writes a new artefact and leaves the old one, tested. The capture records status, entity tag and last modified where the server supplied them. Do not build yet: No deduplication, highlights, feeds or export yet. ### Phase 2 · One item per article, however the link arrived Build: Normalise the address into a key: lowercase the host, drop the fragment, and strip only parameters on a named list you can read. Read the canonical link but honour it for deduplication only when it stays on the same registrable domain. Merging two items keeps both captures and both sets of highlights, and the merge can be undone. Trap: Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. Done when: Two addresses differing only by parameters on the strip list produce one item, tested. A page whose content depends on a query parameter is not collapsed with another, tested with a fixture pair. A canonical link to a different registrable domain is recorded and not used for deduplication, tested. Undoing a merge restores both items with their captures and highlights intact, tested. Do not build yet: No highlights, feeds, search or export yet. ### Phase 3 · Highlights that survive the page being rewritten Build: Anchor a highlight to a capture with both the quoted text and its offsets. When a newer capture arrives, re-anchor by searching for the quote near the old offset. If it cannot be found, mark the highlight orphaned and keep it attached to the capture it was made on, which is still on disk. Never move a highlight to a plausible-looking nearby position. Trap: Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. Done when: A highlight re-anchors correctly against a capture where the text shifted position, tested with a modified fixture. A quote appearing twice anchors to the occurrence nearest its original offset, tested. A highlight whose quote has vanished is marked orphaned and still opens on its original capture, tested. No highlight is ever silently repositioned, asserted by the absence of a fallback path in the re-anchor function. Do not build yet: No feeds, polite fetching or export yet. ### Phase 4 · Feeds, and fetching without becoming a nuisance Build: Import subscriptions from OPML. Poll with the entity tag and last modified you were given, cap concurrency per host, and send an identifying agent string with a contact address. Back off on repeated failure and disable a feed that has failed for a configured run of attempts, surfacing it rather than retrying forever. Never attempt to get past authentication or a paywall. Trap: Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. Done when: A poll sends the stored validators and a response indicating no change stores no new capture, tested against a fixture server. Concurrent requests to one host never exceed the configured limit, measured in a test. A feed failing repeatedly is disabled after the configured run and appears in the interface, tested. No code path sends credentials, cookies from a session, or attempts a paywall bypass, asserted by review and a test over the fetch options. Do not build yet: No export, disk budget or consistency check yet. ### Phase 5 · An archive that outlives this application Build: Export a directory per item containing the raw bytes, the extracted text as Markdown, and the highlights as a sidecar file, alongside a JSON bundle of all metadata. Export OPML and a bookmarks file as well. Verify hashes on import. Run a consistency check reporting items whose files are missing, and show a disk budget with the largest items listed. Trap: An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared. Done when: Export writes raw bytes, extracted Markdown and highlight sidecars per item plus a metadata bundle, and reimports into an empty installation with matching item, capture and highlight counts, tested. Import verifies the recorded hash for every file and refuses a mismatch with the item named, tested by corrupting one. The consistency check reports items with missing files and exits non-zero, tested. The disk report lists the largest items and the total, checked by hand. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything that gets past a paywall or an authentication wall, which is the publisher's decision and not a feature this build is missing. - Native mobile clients with system sharing and background sync, which is most of what the paid application sells and is a per-platform project each. - Licensed full-text feeds and a recommendation graph, which are content deals rather than code. - Full-page archival with every asset and script, which is a different discipline with its own formats and a much larger disk budget. ### README must contain - That a saved item is a copy rather than a link, with the measured share of pages that disappear and the source and date for that figure - That the raw bytes are kept permanently and the readable version is regenerated, and how to re-extract everything - How addresses are normalised, which parameters are stripped, and why a canonical link across domains is ignored for deduplication - How highlights are anchored, what an orphan means, and why a highlight is never quietly moved - The fetching manners: conditional requests, per-host limits, the agent string, and that nothing bypasses authentication ===== AGENTS.md ===== # Agent instructions · GoodLinks indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Next.js 15 with TypeScript and a local background worker, SQLite for metadata, files on disk for raw bytes and extracted text, A maintained readability implementation, versioned and re-runnable, Full-text search over extracted text, rebuildable from the files, Conditional requests, per-host concurrency limits and a clear identifying agent. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. ===== BUILD_PLAN.md ===== # Build plan · GoodLinks A read-later archive whose hard problems are keeping something readable when the page is gone, deciding when two addresses are the same article without breaking the ones that are not, and highlights that survive the page being rewritten. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Save the page, not the link Saving fetches immediately, keeps the raw bytes, and stays useful when extraction fails. ### Steps 1. Fetch at save time, always The page you can read today may not exist next year. Files: `src/capture/fetch.ts`, `tests/capture.test.ts` 2. Keep the raw bytes as the archive Extraction is derived and should be regenerable. Files: `src/capture/store.ts` 3. Make a failed extraction a visible state A saved page you can still read beats a hidden error. Files: `src/capture/extract.ts`, `src/ui/ItemState.tsx` ### Done when - [ ] Raw bytes and extracted text are stored separately - [ ] A failed extraction still yields a readable item - [ ] Re-extraction creates a new artefact without refetching - [ ] Captures record status and validators ### Watch out - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. ## Phase 2 · One item per article, however the link arrived The same article saved from three places becomes one item, and two different pages never do. ### Steps 1. Strip from a named list, not by rule The list is readable and arguable; a rule is neither. Files: `config/strip-params.txt`, `src/url/normalise.ts` 2. Restrict canonical to the same registrable domain A cross-domain canonical is somebody else's claim about your page. Files: `src/url/canonical.ts`, `tests/canonical.test.ts` 3. Make every merge reversible Deduplication is a guess and guesses need an undo. Files: `src/items/merge.ts`, `tests/merge.test.ts` ### Done when - [ ] Listed parameters collapse to one item - [ ] Content-bearing parameters do not collapse - [ ] Cross-domain canonicals are recorded, not followed - [ ] A merge can be undone completely ### Watch out - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. ## Phase 3 · Highlights that survive the page being rewritten A highlight either points at the words it was made on or says it is orphaned. ### Steps 1. Store the quote and the offsets together Either one alone fails in a predictable way. Files: `src/highlights/anchor.ts`, `tests/anchor.test.ts` 2. Re-anchor near the old offset Nearest occurrence, not first occurrence. Files: `src/highlights/reanchor.ts` 3. Make orphaned a visible state with a way back The old capture is still on disk; link to it. Files: `src/ui/Highlight.tsx` ### Done when - [ ] A shifted quote re-anchors correctly - [ ] A repeated quote picks the nearest occurrence - [ ] A vanished quote becomes a visible orphan - [ ] No highlight is silently repositioned ### Watch out - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. ## Phase 4 · Feeds, and fetching without becoming a nuisance Subscriptions poll conditionally, back off when refused, and say when a feed has stopped working. ### Steps 1. Send back the validators you were given It is the difference between polite and rude at scale. Files: `src/feeds/poll.ts`, `tests/conditional.test.ts` 2. Cap concurrency per host and back off One slow site should not become forty simultaneous requests. Files: `src/fetch/limiter.ts` 3. Surface a failing feed instead of retrying it Silence is how an archive stops working unnoticed. Files: `src/ui/Feeds.tsx` ### Done when - [ ] An unchanged response stores no new capture - [ ] Per-host concurrency stays within the limit - [ ] A persistently failing feed is disabled and shown - [ ] Nothing attempts authentication or paywall bypass ### Watch out - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. ## Not in this build - Paywall or authentication bypass. - Native mobile clients with background sync. - Licensed full-text feeds and recommendations. - Full-page archival with every asset. ## After v1, if you want it - Scheduled re-capture for items you starred, which is the only way to notice a page changing under a highlight - A reading-time estimate from the extracted word count, which costs nothing and changes what you open - An offline bundle for one tag, so a journey has a reading list that needs no network ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Items, captures, highlights, tags and feeds. The index over the archive. DATABASE_PATH=./data/reading.db # Required. Raw bytes and extracted text. This directory is the archive itself. ARCHIVE_DIR=./data/archive # Required. Sent on every fetch. Include an address somebody can write to. USER_AGENT=my-reading-archive/1.0 (+https://example.com/contact) # Required. Simultaneous requests to one host. Low is the right answer. PER_HOST_CONCURRENCY=2 # Required. Consecutive failures before a feed is disabled and surfaced. FEED_FAIL_DISABLE_AFTER=10 # Required. Stamped on every extracted artefact so a re-extraction is traceable. EXTRACTOR_VERSION=1
You are building a lean indie version of GoodLinks. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # GoodLinks · indie build A read-later archive whose hard problems are keeping something readable when the page is gone, deciding when two addresses are the same article without breaking the ones that are not, and highlights that survive the page being rewritten. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Next.js 15 with TypeScript and a local background worker | fetching has to keep happening while you are reading something else | | Storage | SQLite for metadata, files on disk for raw bytes and extracted text | the raw bytes are the archive and a database row is an index over it | | Extraction | A maintained readability implementation, versioned and re-runnable | extraction improves, so it must be something you can redo without refetching | | Search | Full-text search over extracted text, rebuildable from the files | an index you can throw away is an index you can fix | | Fetching | Conditional requests, per-host concurrency limits and a clear identifying agent | an archive that gets itself blocked stops being an archive without telling you | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the application, the worker and the consistency check. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **Somewhere with room to keep raw pages** · free, if you already back up a disk - Why: Keeping the bytes is the entire premise, and the archive grows steadily rather than dramatically, so the number to know is the average page size times your saving rate. - Get it: A directory on a disk you back up, sized from a week of real saving. - Verify: The disk report shows the total and the largest items - [ ] **The agent string and the contact address in it** · free - Why: An archive that fetches anonymously gives a site operator no way to reach you and no reason not to block you. - Get it: Decide the name and a contact address, and put both in the configuration rather than the code. - Verify: A request log on a site you control shows the string and the address - [ ] **Your existing subscriptions as OPML** (optional) · free - Why: Feed handling only shows its problems on real feeds, where one is gone, one redirects and one ignores conditional requests entirely. - Get it: Export OPML from whatever reader you use now. - Verify: Import produces the expected feed count and the first poll succeeds ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Paywall or authentication bypass. - Native mobile clients with background sync. - Licensed full-text feeds and recommendations. - Full-page archival with every asset. - native Apple polish, iCloud sync, Shortcuts, and mobile capture - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds If one of those is essential to you, that is the reason to keep paying for GoodLinks, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · GoodLinks The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal reading archive that keeps a copy of the page rather than a link to it to replace GoodLinks. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Next.js 15 with TypeScript and a local background worker · fetching has to keep happening while you are reading something else - Storage: SQLite for metadata, files on disk for raw bytes and extracted text · the raw bytes are the archive and a database row is an index over it - Extraction: A maintained readability implementation, versioned and re-runnable · extraction improves, so it must be something you can redo without refetching - Search: Full-text search over extracted text, rebuildable from the files · an index you can throw away is an index you can fix - Fetching: Conditional requests, per-host concurrency limits and a clear identifying agent · an archive that gets itself blocked stops being an archive without telling you ### Data model (create this before Phase 1) - items(id, url, canonical_url, url_key, title, author, published_at, saved_at, state, read_at, source) · url_key is the normalised key and is the only thing deduplication looks at - captures(id, item_id, at, http_status, etag, last_modified, raw_path, extracted_path, extractor_version, word_count, failed_reason) · more than one per item over time - highlights(id, item_id, capture_id, quote, start_offset, end_offset, note, at, orphaned_at) · anchored to a capture, because the text changes underneath you - tags(id, name), item_tags(item_id, tag_id) · flat, because a hierarchy is a second product - feeds(id, url, title, etag, last_modified, last_polled_at, consecutive_failures, disabled_at) - merges(id, kept_item_id, merged_item_id, at, undone_at) · merging two items is reversible because deduplication is a guess - The raw bytes of every capture are kept. The extractor will get better and the page will not be there to fetch again. ### Phase 1 · Save the page, not the link Build: On save, fetch the page, store the raw response bytes on disk, record the status and the validators, then run extraction into a separate artefact. An extraction failure still leaves a saved item backed by the raw copy, marked as failed rather than hidden. Re-running extraction with a newer version creates a new artefact without refetching. Trap: Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. Done when: Every successful capture stores the raw response bytes and a separate extracted artefact, checked on disk. An extraction failure produces a saved item that still opens, marked failed with its reason, tested with a page the extractor cannot handle. Re-running extraction at a new version writes a new artefact and leaves the old one, tested. The capture records status, entity tag and last modified where the server supplied them. Do not build yet: No deduplication, highlights, feeds or export yet. ### Phase 2 · One item per article, however the link arrived Build: Normalise the address into a key: lowercase the host, drop the fragment, and strip only parameters on a named list you can read. Read the canonical link but honour it for deduplication only when it stays on the same registrable domain. Merging two items keeps both captures and both sets of highlights, and the merge can be undone. Trap: Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. Done when: Two addresses differing only by parameters on the strip list produce one item, tested. A page whose content depends on a query parameter is not collapsed with another, tested with a fixture pair. A canonical link to a different registrable domain is recorded and not used for deduplication, tested. Undoing a merge restores both items with their captures and highlights intact, tested. Do not build yet: No highlights, feeds, search or export yet. ### Phase 3 · Highlights that survive the page being rewritten Build: Anchor a highlight to a capture with both the quoted text and its offsets. When a newer capture arrives, re-anchor by searching for the quote near the old offset. If it cannot be found, mark the highlight orphaned and keep it attached to the capture it was made on, which is still on disk. Never move a highlight to a plausible-looking nearby position. Trap: Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. Done when: A highlight re-anchors correctly against a capture where the text shifted position, tested with a modified fixture. A quote appearing twice anchors to the occurrence nearest its original offset, tested. A highlight whose quote has vanished is marked orphaned and still opens on its original capture, tested. No highlight is ever silently repositioned, asserted by the absence of a fallback path in the re-anchor function. Do not build yet: No feeds, polite fetching or export yet. ### Phase 4 · Feeds, and fetching without becoming a nuisance Build: Import subscriptions from OPML. Poll with the entity tag and last modified you were given, cap concurrency per host, and send an identifying agent string with a contact address. Back off on repeated failure and disable a feed that has failed for a configured run of attempts, surfacing it rather than retrying forever. Never attempt to get past authentication or a paywall. Trap: Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. Done when: A poll sends the stored validators and a response indicating no change stores no new capture, tested against a fixture server. Concurrent requests to one host never exceed the configured limit, measured in a test. A feed failing repeatedly is disabled after the configured run and appears in the interface, tested. No code path sends credentials, cookies from a session, or attempts a paywall bypass, asserted by review and a test over the fetch options. Do not build yet: No export, disk budget or consistency check yet. ### Phase 5 · An archive that outlives this application Build: Export a directory per item containing the raw bytes, the extracted text as Markdown, and the highlights as a sidecar file, alongside a JSON bundle of all metadata. Export OPML and a bookmarks file as well. Verify hashes on import. Run a consistency check reporting items whose files are missing, and show a disk budget with the largest items listed. Trap: An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared. Done when: Export writes raw bytes, extracted Markdown and highlight sidecars per item plus a metadata bundle, and reimports into an empty installation with matching item, capture and highlight counts, tested. Import verifies the recorded hash for every file and refuses a mismatch with the item named, tested by corrupting one. The consistency check reports items with missing files and exits non-zero, tested. The disk report lists the largest items and the total, checked by hand. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything that gets past a paywall or an authentication wall, which is the publisher's decision and not a feature this build is missing. - Native mobile clients with system sharing and background sync, which is most of what the paid application sells and is a per-platform project each. - Licensed full-text feeds and a recommendation graph, which are content deals rather than code. - Full-page archival with every asset and script, which is a different discipline with its own formats and a much larger disk budget. ### README must contain - That a saved item is a copy rather than a link, with the measured share of pages that disappear and the source and date for that figure - That the raw bytes are kept permanently and the readable version is regenerated, and how to re-extract everything - How addresses are normalised, which parameters are stripped, and why a canonical link across domains is ignored for deduplication - How highlights are anchored, what an orphan means, and why a highlight is never quietly moved - The fetching manners: conditional requests, per-host limits, the agent string, and that nothing bypasses authentication ===== AGENTS.md ===== # Agent instructions · GoodLinks indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Next.js 15 with TypeScript and a local background worker, SQLite for metadata, files on disk for raw bytes and extracted text, A maintained readability implementation, versioned and re-runnable, Full-text search over extracted text, rebuildable from the files, Conditional requests, per-host concurrency limits and a clear identifying agent. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. ===== BUILD_PLAN.md ===== # Build plan · GoodLinks A read-later archive whose hard problems are keeping something readable when the page is gone, deciding when two addresses are the same article without breaking the ones that are not, and highlights that survive the page being rewritten. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Save the page, not the link Saving fetches immediately, keeps the raw bytes, and stays useful when extraction fails. ### Steps 1. Fetch at save time, always The page you can read today may not exist next year. Files: `src/capture/fetch.ts`, `tests/capture.test.ts` 2. Keep the raw bytes as the archive Extraction is derived and should be regenerable. Files: `src/capture/store.ts` 3. Make a failed extraction a visible state A saved page you can still read beats a hidden error. Files: `src/capture/extract.ts`, `src/ui/ItemState.tsx` ### Done when - [ ] Raw bytes and extracted text are stored separately - [ ] A failed extraction still yields a readable item - [ ] Re-extraction creates a new artefact without refetching - [ ] Captures record status and validators ### Watch out - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. ## Phase 2 · One item per article, however the link arrived The same article saved from three places becomes one item, and two different pages never do. ### Steps 1. Strip from a named list, not by rule The list is readable and arguable; a rule is neither. Files: `config/strip-params.txt`, `src/url/normalise.ts` 2. Restrict canonical to the same registrable domain A cross-domain canonical is somebody else's claim about your page. Files: `src/url/canonical.ts`, `tests/canonical.test.ts` 3. Make every merge reversible Deduplication is a guess and guesses need an undo. Files: `src/items/merge.ts`, `tests/merge.test.ts` ### Done when - [ ] Listed parameters collapse to one item - [ ] Content-bearing parameters do not collapse - [ ] Cross-domain canonicals are recorded, not followed - [ ] A merge can be undone completely ### Watch out - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. ## Phase 3 · Highlights that survive the page being rewritten A highlight either points at the words it was made on or says it is orphaned. ### Steps 1. Store the quote and the offsets together Either one alone fails in a predictable way. Files: `src/highlights/anchor.ts`, `tests/anchor.test.ts` 2. Re-anchor near the old offset Nearest occurrence, not first occurrence. Files: `src/highlights/reanchor.ts` 3. Make orphaned a visible state with a way back The old capture is still on disk; link to it. Files: `src/ui/Highlight.tsx` ### Done when - [ ] A shifted quote re-anchors correctly - [ ] A repeated quote picks the nearest occurrence - [ ] A vanished quote becomes a visible orphan - [ ] No highlight is silently repositioned ### Watch out - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. ## Phase 4 · Feeds, and fetching without becoming a nuisance Subscriptions poll conditionally, back off when refused, and say when a feed has stopped working. ### Steps 1. Send back the validators you were given It is the difference between polite and rude at scale. Files: `src/feeds/poll.ts`, `tests/conditional.test.ts` 2. Cap concurrency per host and back off One slow site should not become forty simultaneous requests. Files: `src/fetch/limiter.ts` 3. Surface a failing feed instead of retrying it Silence is how an archive stops working unnoticed. Files: `src/ui/Feeds.tsx` ### Done when - [ ] An unchanged response stores no new capture - [ ] Per-host concurrency stays within the limit - [ ] A persistently failing feed is disabled and shown - [ ] Nothing attempts authentication or paywall bypass ### Watch out - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. ## Not in this build - Paywall or authentication bypass. - Native mobile clients with background sync. - Licensed full-text feeds and recommendations. - Full-page archival with every asset. ## After v1, if you want it - Scheduled re-capture for items you starred, which is the only way to notice a page changing under a highlight - A reading-time estimate from the extracted word count, which costs nothing and changes what you open - An offline bundle for one tag, so a journey has a reading list that needs no network ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Items, captures, highlights, tags and feeds. The index over the archive. DATABASE_PATH=./data/reading.db # Required. Raw bytes and extracted text. This directory is the archive itself. ARCHIVE_DIR=./data/archive # Required. Sent on every fetch. Include an address somebody can write to. USER_AGENT=my-reading-archive/1.0 (+https://example.com/contact) # Required. Simultaneous requests to one host. Low is the right answer. PER_HOST_CONCURRENCY=2 # Required. Consecutive failures before a feed is disabled and surfaced. FEED_FAIL_DISABLE_AFTER=10 # Required. Stamped on every extracted artefact so a re-extraction is traceable. EXTRACTOR_VERSION=1
You are building a production product version of GoodLinks. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # GoodLinks · product brief ## Problem The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For GoodLinks, save readable pages locally across Apple devices with tags and highlights. The hard boundary is native apple polish, icloud sync, shortcuts, and mobile capture, plus capture polish, sync, and content partnerships. ## Product outcome A reading archive whose items are copies rather than links, whose deduplication can be undone, whose highlights either point at their words or admit they do not, and whose export carries the bytes with the metadata. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - local disk - optional browser bookmarklet - user-supplied RSS feeds ## Explicit non-goals for v1 - Paywall or authentication bypass. - Native mobile clients with background sync. - Licensed full-text feeds and recommendations. - Full-page archival with every asset. - native Apple polish, iCloud sync, Shortcuts, and mobile capture - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds ## Success criteria - A failed extraction is confirmed to leave a readable item - Content-bearing parameters are confirmed not to collapse items - A vanished quote is confirmed to become a visible orphan - An unchanged conditional response is confirmed to store no capture - Export and reimport are confirmed to preserve counts and hashes ===== BRIEF.md ===== # Build brief · GoodLinks The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal reading archive that keeps a copy of the page rather than a link to it to replace GoodLinks. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Next.js 15 with TypeScript and a local background worker · fetching has to keep happening while you are reading something else - Storage: SQLite for metadata, files on disk for raw bytes and extracted text · the raw bytes are the archive and a database row is an index over it - Extraction: A maintained readability implementation, versioned and re-runnable · extraction improves, so it must be something you can redo without refetching - Search: Full-text search over extracted text, rebuildable from the files · an index you can throw away is an index you can fix - Fetching: Conditional requests, per-host concurrency limits and a clear identifying agent · an archive that gets itself blocked stops being an archive without telling you ### Data model (create this before Phase 1) - items(id, url, canonical_url, url_key, title, author, published_at, saved_at, state, read_at, source) · url_key is the normalised key and is the only thing deduplication looks at - captures(id, item_id, at, http_status, etag, last_modified, raw_path, extracted_path, extractor_version, word_count, failed_reason) · more than one per item over time - highlights(id, item_id, capture_id, quote, start_offset, end_offset, note, at, orphaned_at) · anchored to a capture, because the text changes underneath you - tags(id, name), item_tags(item_id, tag_id) · flat, because a hierarchy is a second product - feeds(id, url, title, etag, last_modified, last_polled_at, consecutive_failures, disabled_at) - merges(id, kept_item_id, merged_item_id, at, undone_at) · merging two items is reversible because deduplication is a guess - The raw bytes of every capture are kept. The extractor will get better and the page will not be there to fetch again. ### Phase 1 · Save the page, not the link Build: On save, fetch the page, store the raw response bytes on disk, record the status and the validators, then run extraction into a separate artefact. An extraction failure still leaves a saved item backed by the raw copy, marked as failed rather than hidden. Re-running extraction with a newer version creates a new artefact without refetching. Trap: Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. Done when: Every successful capture stores the raw response bytes and a separate extracted artefact, checked on disk. An extraction failure produces a saved item that still opens, marked failed with its reason, tested with a page the extractor cannot handle. Re-running extraction at a new version writes a new artefact and leaves the old one, tested. The capture records status, entity tag and last modified where the server supplied them. Do not build yet: No deduplication, highlights, feeds or export yet. ### Phase 2 · One item per article, however the link arrived Build: Normalise the address into a key: lowercase the host, drop the fragment, and strip only parameters on a named list you can read. Read the canonical link but honour it for deduplication only when it stays on the same registrable domain. Merging two items keeps both captures and both sets of highlights, and the merge can be undone. Trap: Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. Done when: Two addresses differing only by parameters on the strip list produce one item, tested. A page whose content depends on a query parameter is not collapsed with another, tested with a fixture pair. A canonical link to a different registrable domain is recorded and not used for deduplication, tested. Undoing a merge restores both items with their captures and highlights intact, tested. Do not build yet: No highlights, feeds, search or export yet. ### Phase 3 · Highlights that survive the page being rewritten Build: Anchor a highlight to a capture with both the quoted text and its offsets. When a newer capture arrives, re-anchor by searching for the quote near the old offset. If it cannot be found, mark the highlight orphaned and keep it attached to the capture it was made on, which is still on disk. Never move a highlight to a plausible-looking nearby position. Trap: Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. Done when: A highlight re-anchors correctly against a capture where the text shifted position, tested with a modified fixture. A quote appearing twice anchors to the occurrence nearest its original offset, tested. A highlight whose quote has vanished is marked orphaned and still opens on its original capture, tested. No highlight is ever silently repositioned, asserted by the absence of a fallback path in the re-anchor function. Do not build yet: No feeds, polite fetching or export yet. ### Phase 4 · Feeds, and fetching without becoming a nuisance Build: Import subscriptions from OPML. Poll with the entity tag and last modified you were given, cap concurrency per host, and send an identifying agent string with a contact address. Back off on repeated failure and disable a feed that has failed for a configured run of attempts, surfacing it rather than retrying forever. Never attempt to get past authentication or a paywall. Trap: Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. Done when: A poll sends the stored validators and a response indicating no change stores no new capture, tested against a fixture server. Concurrent requests to one host never exceed the configured limit, measured in a test. A feed failing repeatedly is disabled after the configured run and appears in the interface, tested. No code path sends credentials, cookies from a session, or attempts a paywall bypass, asserted by review and a test over the fetch options. Do not build yet: No export, disk budget or consistency check yet. ### Phase 5 · An archive that outlives this application Build: Export a directory per item containing the raw bytes, the extracted text as Markdown, and the highlights as a sidecar file, alongside a JSON bundle of all metadata. Export OPML and a bookmarks file as well. Verify hashes on import. Run a consistency check reporting items whose files are missing, and show a disk budget with the largest items listed. Trap: An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared. Done when: Export writes raw bytes, extracted Markdown and highlight sidecars per item plus a metadata bundle, and reimports into an empty installation with matching item, capture and highlight counts, tested. Import verifies the recorded hash for every file and refuses a mismatch with the item named, tested by corrupting one. The consistency check reports items with missing files and exits non-zero, tested. The disk report lists the largest items and the total, checked by hand. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything that gets past a paywall or an authentication wall, which is the publisher's decision and not a feature this build is missing. - Native mobile clients with system sharing and background sync, which is most of what the paid application sells and is a per-platform project each. - Licensed full-text feeds and a recommendation graph, which are content deals rather than code. - Full-page archival with every asset and script, which is a different discipline with its own formats and a much larger disk budget. ### README must contain - That a saved item is a copy rather than a link, with the measured share of pages that disappear and the source and date for that figure - That the raw bytes are kept permanently and the readable version is regenerated, and how to re-extract everything - How addresses are normalised, which parameters are stripped, and why a canonical link across domains is ignored for deduplication - How highlights are anchored, what an orphan means, and why a highlight is never quietly moved - The fetching manners: conditional requests, per-host limits, the agent string, and that nothing bypasses authentication ===== ARCHITECTURE.md ===== # Architecture · GoodLinks ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Next.js 15 with TypeScript and a local background worker | fetching has to keep happening while you are reading something else | | Storage | SQLite for metadata, files on disk for raw bytes and extracted text | the raw bytes are the archive and a database row is an index over it | | Extraction | A maintained readability implementation, versioned and re-runnable | extraction improves, so it must be something you can redo without refetching | | Search | Full-text search over extracted text, rebuildable from the files | an index you can throw away is an index you can fix | | Fetching | Conditional requests, per-host concurrency limits and a clear identifying agent | an archive that gets itself blocked stops being an archive without telling you | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Capture | Fetching, raw storage, extraction and failure states. | Any extractor, provided raw bytes are kept and extraction is redoable. | | Identity | Address normalisation, canonical handling and merges. | Any rules, provided they are readable and merges are reversible. | | Highlights | Anchoring, re-anchoring and orphans. | Any anchoring, provided nothing is silently repositioned. | | Feeds | Polling, conditional requests, limits and backoff. | Any scheduler, provided validators are sent and failures surface. | | Portability | Export, import, hash verification and the consistency check. | Any format, provided the bytes travel with the metadata. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Items, captures, highlights, tags and feeds. The index over the archive. - `ARCHIVE_DIR` · required · Raw bytes and extracted text. This directory is the archive itself. - `USER_AGENT` · required · Sent on every fetch. Include an address somebody can write to. - `PER_HOST_CONCURRENCY` · required · Simultaneous requests to one host. Low is the right answer. - `FEED_FAIL_DISABLE_AFTER` · required · Consecutive failures before a feed is disabled and surfaced. - `EXTRACTOR_VERSION` · required · Stamped on every extracted artefact so a re-extraction is traceable. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · GoodLinks product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Next.js 15 with TypeScript and a local background worker, SQLite for metadata, files on disk for raw bytes and extracted text, A maintained readability implementation, versioned and re-runnable, Full-text search over extracted text, rebuildable from the files, Conditional requests, per-host concurrency limits and a clear identifying agent. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. - An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared. ===== MILESTONES.md ===== # Delivery milestones · GoodLinks Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Save the page, not the link Saving fetches immediately, keeps the raw bytes, and stays useful when extraction fails. ### Steps 1. Fetch at save time, always The page you can read today may not exist next year. Files: `src/capture/fetch.ts`, `tests/capture.test.ts` 2. Keep the raw bytes as the archive Extraction is derived and should be regenerable. Files: `src/capture/store.ts` 3. Make a failed extraction a visible state A saved page you can still read beats a hidden error. Files: `src/capture/extract.ts`, `src/ui/ItemState.tsx` ### Done when - [ ] Raw bytes and extracted text are stored separately - [ ] A failed extraction still yields a readable item - [ ] Re-extraction creates a new artefact without refetching - [ ] Captures record status and validators ### Watch out - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. ## M2 · One item per article, however the link arrived The same article saved from three places becomes one item, and two different pages never do. ### Steps 1. Strip from a named list, not by rule The list is readable and arguable; a rule is neither. Files: `config/strip-params.txt`, `src/url/normalise.ts` 2. Restrict canonical to the same registrable domain A cross-domain canonical is somebody else's claim about your page. Files: `src/url/canonical.ts`, `tests/canonical.test.ts` 3. Make every merge reversible Deduplication is a guess and guesses need an undo. Files: `src/items/merge.ts`, `tests/merge.test.ts` ### Done when - [ ] Listed parameters collapse to one item - [ ] Content-bearing parameters do not collapse - [ ] Cross-domain canonicals are recorded, not followed - [ ] A merge can be undone completely ### Watch out - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. ## M3 · Highlights that survive the page being rewritten A highlight either points at the words it was made on or says it is orphaned. ### Steps 1. Store the quote and the offsets together Either one alone fails in a predictable way. Files: `src/highlights/anchor.ts`, `tests/anchor.test.ts` 2. Re-anchor near the old offset Nearest occurrence, not first occurrence. Files: `src/highlights/reanchor.ts` 3. Make orphaned a visible state with a way back The old capture is still on disk; link to it. Files: `src/ui/Highlight.tsx` ### Done when - [ ] A shifted quote re-anchors correctly - [ ] A repeated quote picks the nearest occurrence - [ ] A vanished quote becomes a visible orphan - [ ] No highlight is silently repositioned ### Watch out - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. ## M4 · Feeds, and fetching without becoming a nuisance Subscriptions poll conditionally, back off when refused, and say when a feed has stopped working. ### Steps 1. Send back the validators you were given It is the difference between polite and rude at scale. Files: `src/feeds/poll.ts`, `tests/conditional.test.ts` 2. Cap concurrency per host and back off One slow site should not become forty simultaneous requests. Files: `src/fetch/limiter.ts` 3. Surface a failing feed instead of retrying it Silence is how an archive stops working unnoticed. Files: `src/ui/Feeds.tsx` ### Done when - [ ] An unchanged response stores no new capture - [ ] Per-host concurrency stays within the limit - [ ] A persistently failing feed is disabled and shown - [ ] Nothing attempts authentication or paywall bypass ### Watch out - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. ## M5 · An archive that outlives this application (production only) Everything leaves in open formats, and a consistency check proves the files are still there. ### Steps 1. Export the bytes beside the metadata A manifest pointing at files nobody exported is not a backup. Files: `src/export/bundle.ts`, `tests/export.test.ts` 2. Verify hashes on import and name the failure Silent corruption is the only kind that matters here. Files: `src/export/import.ts`, `tests/hash.test.ts` 3. Run a consistency check that can fail a script A non-zero exit is how this ends up in a cron entry. Files: `scripts/check.ts` ### Done when - [ ] Export and reimport preserve items, captures and highlights - [ ] A corrupted file is refused and named on import - [ ] The consistency check reports missing files and exits non-zero - [ ] The disk report lists the largest items and the total ### Watch out - An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared. ===== OPERATIONS.md ===== # Operations · GoodLinks ## Backup The archive directory and the database, in that order of importance. The database can be rebuilt from the files with some loss of reading state; the files cannot be rebuilt from anything, because the pages they came from are the thing that disappears. ## Restore Restore both, run the consistency check, and rebuild the search index from the extracted artefacts. A restore is correct when the consistency check reports zero missing files and the item count matches. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Disk growth per week, extraction failure rate by host, feeds disabled for repeated failure, and the consistency check exit code on a schedule. A rising extraction failure rate for one host usually means that site changed its markup, which is a re-extraction job rather than a bug. ## Incident checklist The quiet failure is an archive that stopped saving: a blocked fetcher, a full disk, a worker that died. Each of those shows up as saves without captures, so the check that matters is the count of items with no successful capture in the last day. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A failed extraction is confirmed to leave a readable item - [ ] Content-bearing parameters are confirmed not to collapse items - [ ] A vanished quote is confirmed to become a visible orphan - [ ] An unchanged conditional response is confirmed to store no capture - [ ] Export and reimport are confirmed to preserve counts and hashes ## Launch constraint Do not market omitted GoodLinks capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Items, captures, highlights, tags and feeds. The index over the archive. DATABASE_PATH=./data/reading.db # Required. Raw bytes and extracted text. This directory is the archive itself. ARCHIVE_DIR=./data/archive # Required. Sent on every fetch. Include an address somebody can write to. USER_AGENT=my-reading-archive/1.0 (+https://example.com/contact) # Required. Simultaneous requests to one host. Low is the right answer. PER_HOST_CONCURRENCY=2 # Required. Consecutive failures before a feed is disabled and surfaced. FEED_FAIL_DISABLE_AFTER=10 # Required. Stamped on every extracted artefact so a re-extraction is traceable. EXTRACTOR_VERSION=1
README.md
# GoodLinks · indie build A read-later archive whose hard problems are keeping something readable when the page is gone, deciding when two addresses are the same article without breaking the ones that are not, and highlights that survive the page being rewritten. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Next.js 15 with TypeScript and a local background worker | fetching has to keep happening while you are reading something else | | Storage | SQLite for metadata, files on disk for raw bytes and extracted text | the raw bytes are the archive and a database row is an index over it | | Extraction | A maintained readability implementation, versioned and re-runnable | extraction improves, so it must be something you can redo without refetching | | Search | Full-text search over extracted text, rebuildable from the files | an index you can throw away is an index you can fix | | Fetching | Conditional requests, per-host concurrency limits and a clear identifying agent | an archive that gets itself blocked stops being an archive without telling you | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the application, the worker and the consistency check. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **Somewhere with room to keep raw pages** · free, if you already back up a disk - Why: Keeping the bytes is the entire premise, and the archive grows steadily rather than dramatically, so the number to know is the average page size times your saving rate. - Get it: A directory on a disk you back up, sized from a week of real saving. - Verify: The disk report shows the total and the largest items - [ ] **The agent string and the contact address in it** · free - Why: An archive that fetches anonymously gives a site operator no way to reach you and no reason not to block you. - Get it: Decide the name and a contact address, and put both in the configuration rather than the code. - Verify: A request log on a site you control shows the string and the address - [ ] **Your existing subscriptions as OPML** (optional) · free - Why: Feed handling only shows its problems on real feeds, where one is gone, one redirects and one ignores conditional requests entirely. - Get it: Export OPML from whatever reader you use now. - Verify: Import produces the expected feed count and the first poll succeeds ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Paywall or authentication bypass. - Native mobile clients with background sync. - Licensed full-text feeds and recommendations. - Full-page archival with every asset. - native Apple polish, iCloud sync, Shortcuts, and mobile capture - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds If one of those is essential to you, that is the reason to keep paying for GoodLinks, and the README should say so rather than pretend.
BRIEF.md
# Build brief · GoodLinks The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal reading archive that keeps a copy of the page rather than a link to it to replace GoodLinks. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Next.js 15 with TypeScript and a local background worker · fetching has to keep happening while you are reading something else - Storage: SQLite for metadata, files on disk for raw bytes and extracted text · the raw bytes are the archive and a database row is an index over it - Extraction: A maintained readability implementation, versioned and re-runnable · extraction improves, so it must be something you can redo without refetching - Search: Full-text search over extracted text, rebuildable from the files · an index you can throw away is an index you can fix - Fetching: Conditional requests, per-host concurrency limits and a clear identifying agent · an archive that gets itself blocked stops being an archive without telling you ### Data model (create this before Phase 1) - items(id, url, canonical_url, url_key, title, author, published_at, saved_at, state, read_at, source) · url_key is the normalised key and is the only thing deduplication looks at - captures(id, item_id, at, http_status, etag, last_modified, raw_path, extracted_path, extractor_version, word_count, failed_reason) · more than one per item over time - highlights(id, item_id, capture_id, quote, start_offset, end_offset, note, at, orphaned_at) · anchored to a capture, because the text changes underneath you - tags(id, name), item_tags(item_id, tag_id) · flat, because a hierarchy is a second product - feeds(id, url, title, etag, last_modified, last_polled_at, consecutive_failures, disabled_at) - merges(id, kept_item_id, merged_item_id, at, undone_at) · merging two items is reversible because deduplication is a guess - The raw bytes of every capture are kept. The extractor will get better and the page will not be there to fetch again. ### Phase 1 · Save the page, not the link Build: On save, fetch the page, store the raw response bytes on disk, record the status and the validators, then run extraction into a separate artefact. An extraction failure still leaves a saved item backed by the raw copy, marked as failed rather than hidden. Re-running extraction with a newer version creates a new artefact without refetching. Trap: Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. Done when: Every successful capture stores the raw response bytes and a separate extracted artefact, checked on disk. An extraction failure produces a saved item that still opens, marked failed with its reason, tested with a page the extractor cannot handle. Re-running extraction at a new version writes a new artefact and leaves the old one, tested. The capture records status, entity tag and last modified where the server supplied them. Do not build yet: No deduplication, highlights, feeds or export yet. ### Phase 2 · One item per article, however the link arrived Build: Normalise the address into a key: lowercase the host, drop the fragment, and strip only parameters on a named list you can read. Read the canonical link but honour it for deduplication only when it stays on the same registrable domain. Merging two items keeps both captures and both sets of highlights, and the merge can be undone. Trap: Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. Done when: Two addresses differing only by parameters on the strip list produce one item, tested. A page whose content depends on a query parameter is not collapsed with another, tested with a fixture pair. A canonical link to a different registrable domain is recorded and not used for deduplication, tested. Undoing a merge restores both items with their captures and highlights intact, tested. Do not build yet: No highlights, feeds, search or export yet. ### Phase 3 · Highlights that survive the page being rewritten Build: Anchor a highlight to a capture with both the quoted text and its offsets. When a newer capture arrives, re-anchor by searching for the quote near the old offset. If it cannot be found, mark the highlight orphaned and keep it attached to the capture it was made on, which is still on disk. Never move a highlight to a plausible-looking nearby position. Trap: Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. Done when: A highlight re-anchors correctly against a capture where the text shifted position, tested with a modified fixture. A quote appearing twice anchors to the occurrence nearest its original offset, tested. A highlight whose quote has vanished is marked orphaned and still opens on its original capture, tested. No highlight is ever silently repositioned, asserted by the absence of a fallback path in the re-anchor function. Do not build yet: No feeds, polite fetching or export yet. ### Phase 4 · Feeds, and fetching without becoming a nuisance Build: Import subscriptions from OPML. Poll with the entity tag and last modified you were given, cap concurrency per host, and send an identifying agent string with a contact address. Back off on repeated failure and disable a feed that has failed for a configured run of attempts, surfacing it rather than retrying forever. Never attempt to get past authentication or a paywall. Trap: Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. Done when: A poll sends the stored validators and a response indicating no change stores no new capture, tested against a fixture server. Concurrent requests to one host never exceed the configured limit, measured in a test. A feed failing repeatedly is disabled after the configured run and appears in the interface, tested. No code path sends credentials, cookies from a session, or attempts a paywall bypass, asserted by review and a test over the fetch options. Do not build yet: No export, disk budget or consistency check yet. ### Phase 5 · An archive that outlives this application Build: Export a directory per item containing the raw bytes, the extracted text as Markdown, and the highlights as a sidecar file, alongside a JSON bundle of all metadata. Export OPML and a bookmarks file as well. Verify hashes on import. Run a consistency check reporting items whose files are missing, and show a disk budget with the largest items listed. Trap: An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared. Done when: Export writes raw bytes, extracted Markdown and highlight sidecars per item plus a metadata bundle, and reimports into an empty installation with matching item, capture and highlight counts, tested. Import verifies the recorded hash for every file and refuses a mismatch with the item named, tested by corrupting one. The consistency check reports items with missing files and exits non-zero, tested. The disk report lists the largest items and the total, checked by hand. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything that gets past a paywall or an authentication wall, which is the publisher's decision and not a feature this build is missing. - Native mobile clients with system sharing and background sync, which is most of what the paid application sells and is a per-platform project each. - Licensed full-text feeds and a recommendation graph, which are content deals rather than code. - Full-page archival with every asset and script, which is a different discipline with its own formats and a much larger disk budget. ### README must contain - That a saved item is a copy rather than a link, with the measured share of pages that disappear and the source and date for that figure - That the raw bytes are kept permanently and the readable version is regenerated, and how to re-extract everything - How addresses are normalised, which parameters are stripped, and why a canonical link across domains is ignored for deduplication - How highlights are anchored, what an orphan means, and why a highlight is never quietly moved - The fetching manners: conditional requests, per-host limits, the agent string, and that nothing bypasses authentication
AGENTS.md
# Agent instructions · GoodLinks indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Next.js 15 with TypeScript and a local background worker, SQLite for metadata, files on disk for raw bytes and extracted text, A maintained readability implementation, versioned and re-runnable, Full-text search over extracted text, rebuildable from the files, Conditional requests, per-host concurrency limits and a clear identifying agent. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log.
BUILD_PLAN.md
# Build plan · GoodLinks A read-later archive whose hard problems are keeping something readable when the page is gone, deciding when two addresses are the same article without breaking the ones that are not, and highlights that survive the page being rewritten. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Save the page, not the link Saving fetches immediately, keeps the raw bytes, and stays useful when extraction fails. ### Steps 1. Fetch at save time, always The page you can read today may not exist next year. Files: `src/capture/fetch.ts`, `tests/capture.test.ts` 2. Keep the raw bytes as the archive Extraction is derived and should be regenerable. Files: `src/capture/store.ts` 3. Make a failed extraction a visible state A saved page you can still read beats a hidden error. Files: `src/capture/extract.ts`, `src/ui/ItemState.tsx` ### Done when - [ ] Raw bytes and extracted text are stored separately - [ ] A failed extraction still yields a readable item - [ ] Re-extraction creates a new artefact without refetching - [ ] Captures record status and validators ### Watch out - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. ## Phase 2 · One item per article, however the link arrived The same article saved from three places becomes one item, and two different pages never do. ### Steps 1. Strip from a named list, not by rule The list is readable and arguable; a rule is neither. Files: `config/strip-params.txt`, `src/url/normalise.ts` 2. Restrict canonical to the same registrable domain A cross-domain canonical is somebody else's claim about your page. Files: `src/url/canonical.ts`, `tests/canonical.test.ts` 3. Make every merge reversible Deduplication is a guess and guesses need an undo. Files: `src/items/merge.ts`, `tests/merge.test.ts` ### Done when - [ ] Listed parameters collapse to one item - [ ] Content-bearing parameters do not collapse - [ ] Cross-domain canonicals are recorded, not followed - [ ] A merge can be undone completely ### Watch out - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. ## Phase 3 · Highlights that survive the page being rewritten A highlight either points at the words it was made on or says it is orphaned. ### Steps 1. Store the quote and the offsets together Either one alone fails in a predictable way. Files: `src/highlights/anchor.ts`, `tests/anchor.test.ts` 2. Re-anchor near the old offset Nearest occurrence, not first occurrence. Files: `src/highlights/reanchor.ts` 3. Make orphaned a visible state with a way back The old capture is still on disk; link to it. Files: `src/ui/Highlight.tsx` ### Done when - [ ] A shifted quote re-anchors correctly - [ ] A repeated quote picks the nearest occurrence - [ ] A vanished quote becomes a visible orphan - [ ] No highlight is silently repositioned ### Watch out - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. ## Phase 4 · Feeds, and fetching without becoming a nuisance Subscriptions poll conditionally, back off when refused, and say when a feed has stopped working. ### Steps 1. Send back the validators you were given It is the difference between polite and rude at scale. Files: `src/feeds/poll.ts`, `tests/conditional.test.ts` 2. Cap concurrency per host and back off One slow site should not become forty simultaneous requests. Files: `src/fetch/limiter.ts` 3. Surface a failing feed instead of retrying it Silence is how an archive stops working unnoticed. Files: `src/ui/Feeds.tsx` ### Done when - [ ] An unchanged response stores no new capture - [ ] Per-host concurrency stays within the limit - [ ] A persistently failing feed is disabled and shown - [ ] Nothing attempts authentication or paywall bypass ### Watch out - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. ## Not in this build - Paywall or authentication bypass. - Native mobile clients with background sync. - Licensed full-text feeds and recommendations. - Full-page archival with every asset. ## After v1, if you want it - Scheduled re-capture for items you starred, which is the only way to notice a page changing under a highlight - A reading-time estimate from the extracted word count, which costs nothing and changes what you open - An offline bundle for one tag, so a journey has a reading list that needs no network
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Items, captures, highlights, tags and feeds. The index over the archive. DATABASE_PATH=./data/reading.db # Required. Raw bytes and extracted text. This directory is the archive itself. ARCHIVE_DIR=./data/archive # Required. Sent on every fetch. Include an address somebody can write to. USER_AGENT=my-reading-archive/1.0 (+https://example.com/contact) # Required. Simultaneous requests to one host. Low is the right answer. PER_HOST_CONCURRENCY=2 # Required. Consecutive failures before a feed is disabled and surfaced. FEED_FAIL_DISABLE_AFTER=10 # Required. Stamped on every extracted artefact so a re-extraction is traceable. EXTRACTOR_VERSION=1
PRODUCT.md
# GoodLinks · product brief ## Problem The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For GoodLinks, save readable pages locally across Apple devices with tags and highlights. The hard boundary is native apple polish, icloud sync, shortcuts, and mobile capture, plus capture polish, sync, and content partnerships. ## Product outcome A reading archive whose items are copies rather than links, whose deduplication can be undone, whose highlights either point at their words or admit they do not, and whose export carries the bytes with the metadata. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - local disk - optional browser bookmarklet - user-supplied RSS feeds ## Explicit non-goals for v1 - Paywall or authentication bypass. - Native mobile clients with background sync. - Licensed full-text feeds and recommendations. - Full-page archival with every asset. - native Apple polish, iCloud sync, Shortcuts, and mobile capture - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds ## Success criteria - A failed extraction is confirmed to leave a readable item - Content-bearing parameters are confirmed not to collapse items - A vanished quote is confirmed to become a visible orphan - An unchanged conditional response is confirmed to store no capture - Export and reimport are confirmed to preserve counts and hashes
BRIEF.md
# Build brief · GoodLinks The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal reading archive that keeps a copy of the page rather than a link to it to replace GoodLinks. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Next.js 15 with TypeScript and a local background worker · fetching has to keep happening while you are reading something else - Storage: SQLite for metadata, files on disk for raw bytes and extracted text · the raw bytes are the archive and a database row is an index over it - Extraction: A maintained readability implementation, versioned and re-runnable · extraction improves, so it must be something you can redo without refetching - Search: Full-text search over extracted text, rebuildable from the files · an index you can throw away is an index you can fix - Fetching: Conditional requests, per-host concurrency limits and a clear identifying agent · an archive that gets itself blocked stops being an archive without telling you ### Data model (create this before Phase 1) - items(id, url, canonical_url, url_key, title, author, published_at, saved_at, state, read_at, source) · url_key is the normalised key and is the only thing deduplication looks at - captures(id, item_id, at, http_status, etag, last_modified, raw_path, extracted_path, extractor_version, word_count, failed_reason) · more than one per item over time - highlights(id, item_id, capture_id, quote, start_offset, end_offset, note, at, orphaned_at) · anchored to a capture, because the text changes underneath you - tags(id, name), item_tags(item_id, tag_id) · flat, because a hierarchy is a second product - feeds(id, url, title, etag, last_modified, last_polled_at, consecutive_failures, disabled_at) - merges(id, kept_item_id, merged_item_id, at, undone_at) · merging two items is reversible because deduplication is a guess - The raw bytes of every capture are kept. The extractor will get better and the page will not be there to fetch again. ### Phase 1 · Save the page, not the link Build: On save, fetch the page, store the raw response bytes on disk, record the status and the validators, then run extraction into a separate artefact. An extraction failure still leaves a saved item backed by the raw copy, marked as failed rather than hidden. Re-running extraction with a newer version creates a new artefact without refetching. Trap: Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. Done when: Every successful capture stores the raw response bytes and a separate extracted artefact, checked on disk. An extraction failure produces a saved item that still opens, marked failed with its reason, tested with a page the extractor cannot handle. Re-running extraction at a new version writes a new artefact and leaves the old one, tested. The capture records status, entity tag and last modified where the server supplied them. Do not build yet: No deduplication, highlights, feeds or export yet. ### Phase 2 · One item per article, however the link arrived Build: Normalise the address into a key: lowercase the host, drop the fragment, and strip only parameters on a named list you can read. Read the canonical link but honour it for deduplication only when it stays on the same registrable domain. Merging two items keeps both captures and both sets of highlights, and the merge can be undone. Trap: Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. Done when: Two addresses differing only by parameters on the strip list produce one item, tested. A page whose content depends on a query parameter is not collapsed with another, tested with a fixture pair. A canonical link to a different registrable domain is recorded and not used for deduplication, tested. Undoing a merge restores both items with their captures and highlights intact, tested. Do not build yet: No highlights, feeds, search or export yet. ### Phase 3 · Highlights that survive the page being rewritten Build: Anchor a highlight to a capture with both the quoted text and its offsets. When a newer capture arrives, re-anchor by searching for the quote near the old offset. If it cannot be found, mark the highlight orphaned and keep it attached to the capture it was made on, which is still on disk. Never move a highlight to a plausible-looking nearby position. Trap: Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. Done when: A highlight re-anchors correctly against a capture where the text shifted position, tested with a modified fixture. A quote appearing twice anchors to the occurrence nearest its original offset, tested. A highlight whose quote has vanished is marked orphaned and still opens on its original capture, tested. No highlight is ever silently repositioned, asserted by the absence of a fallback path in the re-anchor function. Do not build yet: No feeds, polite fetching or export yet. ### Phase 4 · Feeds, and fetching without becoming a nuisance Build: Import subscriptions from OPML. Poll with the entity tag and last modified you were given, cap concurrency per host, and send an identifying agent string with a contact address. Back off on repeated failure and disable a feed that has failed for a configured run of attempts, surfacing it rather than retrying forever. Never attempt to get past authentication or a paywall. Trap: Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. Done when: A poll sends the stored validators and a response indicating no change stores no new capture, tested against a fixture server. Concurrent requests to one host never exceed the configured limit, measured in a test. A feed failing repeatedly is disabled after the configured run and appears in the interface, tested. No code path sends credentials, cookies from a session, or attempts a paywall bypass, asserted by review and a test over the fetch options. Do not build yet: No export, disk budget or consistency check yet. ### Phase 5 · An archive that outlives this application Build: Export a directory per item containing the raw bytes, the extracted text as Markdown, and the highlights as a sidecar file, alongside a JSON bundle of all metadata. Export OPML and a bookmarks file as well. Verify hashes on import. Run a consistency check reporting items whose files are missing, and show a disk budget with the largest items listed. Trap: An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared. Done when: Export writes raw bytes, extracted Markdown and highlight sidecars per item plus a metadata bundle, and reimports into an empty installation with matching item, capture and highlight counts, tested. Import verifies the recorded hash for every file and refuses a mismatch with the item named, tested by corrupting one. The consistency check reports items with missing files and exits non-zero, tested. The disk report lists the largest items and the total, checked by hand. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything that gets past a paywall or an authentication wall, which is the publisher's decision and not a feature this build is missing. - Native mobile clients with system sharing and background sync, which is most of what the paid application sells and is a per-platform project each. - Licensed full-text feeds and a recommendation graph, which are content deals rather than code. - Full-page archival with every asset and script, which is a different discipline with its own formats and a much larger disk budget. ### README must contain - That a saved item is a copy rather than a link, with the measured share of pages that disappear and the source and date for that figure - That the raw bytes are kept permanently and the readable version is regenerated, and how to re-extract everything - How addresses are normalised, which parameters are stripped, and why a canonical link across domains is ignored for deduplication - How highlights are anchored, what an orphan means, and why a highlight is never quietly moved - The fetching manners: conditional requests, per-host limits, the agent string, and that nothing bypasses authentication
ARCHITECTURE.md
# Architecture · GoodLinks ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Next.js 15 with TypeScript and a local background worker | fetching has to keep happening while you are reading something else | | Storage | SQLite for metadata, files on disk for raw bytes and extracted text | the raw bytes are the archive and a database row is an index over it | | Extraction | A maintained readability implementation, versioned and re-runnable | extraction improves, so it must be something you can redo without refetching | | Search | Full-text search over extracted text, rebuildable from the files | an index you can throw away is an index you can fix | | Fetching | Conditional requests, per-host concurrency limits and a clear identifying agent | an archive that gets itself blocked stops being an archive without telling you | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Capture | Fetching, raw storage, extraction and failure states. | Any extractor, provided raw bytes are kept and extraction is redoable. | | Identity | Address normalisation, canonical handling and merges. | Any rules, provided they are readable and merges are reversible. | | Highlights | Anchoring, re-anchoring and orphans. | Any anchoring, provided nothing is silently repositioned. | | Feeds | Polling, conditional requests, limits and backoff. | Any scheduler, provided validators are sent and failures surface. | | Portability | Export, import, hash verification and the consistency check. | Any format, provided the bytes travel with the metadata. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Items, captures, highlights, tags and feeds. The index over the archive. - `ARCHIVE_DIR` · required · Raw bytes and extracted text. This directory is the archive itself. - `USER_AGENT` · required · Sent on every fetch. Include an address somebody can write to. - `PER_HOST_CONCURRENCY` · required · Simultaneous requests to one host. Low is the right answer. - `FEED_FAIL_DISABLE_AFTER` · required · Consecutive failures before a feed is disabled and surfaced. - `EXTRACTOR_VERSION` · required · Stamped on every extracted artefact so a re-extraction is traceable. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · GoodLinks product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Next.js 15 with TypeScript and a local background worker, SQLite for metadata, files on disk for raw bytes and extracted text, A maintained readability implementation, versioned and re-runnable, Full-text search over extracted text, rebuildable from the files, Conditional requests, per-host concurrency limits and a clear identifying agent. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. - An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared.
MILESTONES.md
# Delivery milestones · GoodLinks Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Save the page, not the link Saving fetches immediately, keeps the raw bytes, and stays useful when extraction fails. ### Steps 1. Fetch at save time, always The page you can read today may not exist next year. Files: `src/capture/fetch.ts`, `tests/capture.test.ts` 2. Keep the raw bytes as the archive Extraction is derived and should be regenerable. Files: `src/capture/store.ts` 3. Make a failed extraction a visible state A saved page you can still read beats a hidden error. Files: `src/capture/extract.ts`, `src/ui/ItemState.tsx` ### Done when - [ ] Raw bytes and extracted text are stored separately - [ ] A failed extraction still yields a readable item - [ ] Re-extraction creates a new artefact without refetching - [ ] Captures record status and validators ### Watch out - Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate. ## M2 · One item per article, however the link arrived The same article saved from three places becomes one item, and two different pages never do. ### Steps 1. Strip from a named list, not by rule The list is readable and arguable; a rule is neither. Files: `config/strip-params.txt`, `src/url/normalise.ts` 2. Restrict canonical to the same registrable domain A cross-domain canonical is somebody else's claim about your page. Files: `src/url/canonical.ts`, `tests/canonical.test.ts` 3. Make every merge reversible Deduplication is a guess and guesses need an undo. Files: `src/items/merge.ts`, `tests/merge.test.ts` ### Done when - [ ] Listed parameters collapse to one item - [ ] Content-bearing parameters do not collapse - [ ] Cross-domain canonicals are recorded, not followed - [ ] A merge can be undone completely ### Watch out - Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess. ## M3 · Highlights that survive the page being rewritten A highlight either points at the words it was made on or says it is orphaned. ### Steps 1. Store the quote and the offsets together Either one alone fails in a predictable way. Files: `src/highlights/anchor.ts`, `tests/anchor.test.ts` 2. Re-anchor near the old offset Nearest occurrence, not first occurrence. Files: `src/highlights/reanchor.ts` 3. Make orphaned a visible state with a way back The old capture is still on disk; link to it. Files: `src/ui/Highlight.tsx` ### Done when - [ ] A shifted quote re-anchors correctly - [ ] A repeated quote picks the nearest occurrence - [ ] A vanished quote becomes a visible orphan - [ ] No highlight is silently repositioned ### Watch out - Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense. ## M4 · Feeds, and fetching without becoming a nuisance Subscriptions poll conditionally, back off when refused, and say when a feed has stopped working. ### Steps 1. Send back the validators you were given It is the difference between polite and rude at scale. Files: `src/feeds/poll.ts`, `tests/conditional.test.ts` 2. Cap concurrency per host and back off One slow site should not become forty simultaneous requests. Files: `src/fetch/limiter.ts` 3. Surface a failing feed instead of retrying it Silence is how an archive stops working unnoticed. Files: `src/ui/Feeds.tsx` ### Done when - [ ] An unchanged response stores no new capture - [ ] Per-host concurrency stays within the limit - [ ] A persistently failing feed is disabled and shown - [ ] Nothing attempts authentication or paywall bypass ### Watch out - Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log. ## M5 · An archive that outlives this application (production only) Everything leaves in open formats, and a consistency check proves the files are still there. ### Steps 1. Export the bytes beside the metadata A manifest pointing at files nobody exported is not a backup. Files: `src/export/bundle.ts`, `tests/export.test.ts` 2. Verify hashes on import and name the failure Silent corruption is the only kind that matters here. Files: `src/export/import.ts`, `tests/hash.test.ts` 3. Run a consistency check that can fail a script A non-zero exit is how this ends up in a cron entry. Files: `scripts/check.ts` ### Done when - [ ] Export and reimport preserve items, captures and highlights - [ ] A corrupted file is refused and named on import - [ ] The consistency check reports missing files and exits non-zero - [ ] The disk report lists the largest items and the total ### Watch out - An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared.
OPERATIONS.md
# Operations · GoodLinks ## Backup The archive directory and the database, in that order of importance. The database can be rebuilt from the files with some loss of reading state; the files cannot be rebuilt from anything, because the pages they came from are the thing that disappears. ## Restore Restore both, run the consistency check, and rebuild the search index from the extracted artefacts. A restore is correct when the consistency check reports zero missing files and the item count matches. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Disk growth per week, extraction failure rate by host, feeds disabled for repeated failure, and the consistency check exit code on a schedule. A rising extraction failure rate for one host usually means that site changed its markup, which is a re-extraction job rather than a bug. ## Incident checklist The quiet failure is an archive that stopped saving: a blocked fetcher, a full disk, a worker that died. Each of those shows up as saves without captures, so the check that matters is the count of items with no successful capture in the last day. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A failed extraction is confirmed to leave a readable item - [ ] Content-bearing parameters are confirmed not to collapse items - [ ] A vanished quote is confirmed to become a visible orphan - [ ] An unchanged conditional response is confirmed to store no capture - [ ] Export and reimport are confirmed to preserve counts and hashes ## Launch constraint Do not market omitted GoodLinks capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Items, captures, highlights, tags and feeds. The index over the archive. DATABASE_PATH=./data/reading.db # Required. Raw bytes and extracted text. This directory is the archive itself. ARCHIVE_DIR=./data/archive # Required. Sent on every fetch. Include an address somebody can write to. USER_AGENT=my-reading-archive/1.0 (+https://example.com/contact) # Required. Simultaneous requests to one host. Low is the right answer. PER_HOST_CONCURRENCY=2 # Required. Consecutive failures before a feed is disabled and surfaced. FEED_FAIL_DISABLE_AFTER=10 # Required. Stamped on every extracted artefact so a re-extraction is traceable. EXTRACTOR_VERSION=1
$ choose a build depth, inspect the files, then open the complete pack in your agent
People still pay for GoodLinks because people pay when capture works everywhere, reading state syncs instantly, and the archive remains clean without constant parser repairs. The recurring cost buys feed quirks, parsing, paywalls, canonical URLs, deduplication, images, search, sync, browser extensions, and publisher changes, not just the visible interface.
xnative Apple polish, iCloud sync, Shortcuts, and mobile capture
xpublisher bypasses
xcross-device mobile capture
xhigh-quality recommendation graph
xlicensed full-text feeds
share on X ↗"I just replaced GoodLinks ($0.42/mo) with one prompt"
Don't feel like building it? These folks already made it free.
KarakeepA polished read-later archive with highlights and mobile apps, provided you host it.open source↗
wallabagReadable saved pages, tags and highlights across devices; local Apple elegance becomes Docker.open source↗no votes, no pay-to-list · just what's real
GoodLinks pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| goodlinks app | custom | — | 1 universal Apple-platform purchase; includes all current features and 1 year of newly released premium features. |
| annual feature upgrade | — | $0.42 | Adds new premium features released during each additional 1-year period; previously unlocked features remain permanently available. |
free tierno free tier and no trial
billingone-time $9.99 app purchase + optional $4.99/year feature-upgrade subscription; no required monthly plan
hidden costssummaries can require the user's own external LLM API key, creating separate provider charges; Apple regional pricing and tax may vary
verified 2026-08-13 · source ↗
Vibecode GoodLinks
Yes. A competent AI coding agent (Claude Code, Codex, Cursor) can build a usable personal GoodLinks replacement in one session with the prompt on this page. It runs on your own machine or server with no subscription.
How much does GoodLinks cost?
GoodLinks costs about $0.42/month (Annual Feature Upgrade, checked 2026-08-13), which is $5.04 per year. That's what you save by replacing it with one prompt.
What do I lose by replacing GoodLinks?
Honestly: native Apple polish, iCloud sync, Shortcuts, and mobile capture; publisher bypasses; cross-device mobile capture; high-quality recommendation graph; licensed full-text feeds. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to GoodLinks?
Yes: Karakeep (A polished read-later archive with highlights and mobile apps, provided you host it.) wallabag (Readable saved pages, tags and highlights across devices; local Apple elegance becomes Docker.) The prompt is for when you want it exactly your way.