Build GoodLinks

YESreplaces $0.42/mosaves $5.04/yrback to the verdict

0%0 of 16 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A read-later archive whose hard problems are keeping something readable when the page is gone, deciding when two addresses are the same article without breaking the ones that are not, and highlights that survive the page being rewritten.

estimated effort weekendthe files for this build are in the project pack

RuntimeNext.js 15 with TypeScript and a local background workerStorageSQLite for metadata, files on disk for raw bytes and extracted textExtractionA maintained readability implementation, versioned and re-runnableSearchFull-text search over extracted text, rebuildable from the filesFetchingConditional requests, per-host concurrency limits and a clear identifying agent

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the application, the worker and the consistency check.
Get it nodejs.org or your version manager. open ↗
Verify node --version prints v22 or higher
have readyfree, if you already back up a disk
Why Keeping the bytes is the entire premise, and the archive grows steadily rather than dramatically, so the number to know is the average page size times your saving rate.
Get it A directory on a disk you back up, sized from a week of real saving.
Verify The disk report shows the total and the largest items
decidefree
Why An archive that fetches anonymously gives a site operator no way to reach you and no reason not to block you.
Get it Decide the name and a contact address, and put both in the configuration rather than the code.
Verify A request log on a site you control shows the string and the address
have readyfree
Why Feed handling only shows its problems on real feeds, where one is gone, one redirects and one ignores conditional requests entirely.
Get it Export OPML from whatever reader you use now.
Verify Import produces the expected feed count and the first poll succeeds

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- items(id, url, canonical_url, url_key, title, author, published_at, saved_at, state, read_at, source) · url_key is the normalised key and is the only thing deduplication looks at
- captures(id, item_id, at, http_status, etag, last_modified, raw_path, extracted_path, extractor_version, word_count, failed_reason) · more than one per item over time
- highlights(id, item_id, capture_id, quote, start_offset, end_offset, note, at, orphaned_at) · anchored to a capture, because the text changes underneath you
- tags(id, name), item_tags(item_id, tag_id) · flat, because a hierarchy is a second product
- feeds(id, url, title, etag, last_modified, last_polled_at, consecutive_failures, disabled_at)
- merges(id, kept_item_id, merged_item_id, at, undone_at) · merging two items is reversible because deduplication is a guess
- The raw bytes of every capture are kept. The extractor will get better and the page will not be there to fetch again.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_PATH`	required	`./data/reading.db`	Items, captures, highlights, tags and feeds. The index over the archive.
`ARCHIVE_DIR`	required	`./data/archive`	Raw bytes and extracted text. This directory is the archive itself.
`USER_AGENT`	required	`my-reading-archive/1.0 (+https://example.com/contact)`	Sent on every fetch. Include an address somebody can write to.
`PER_HOST_CONCURRENCY`	required	`2`	Simultaneous requests to one host. Low is the right answer.
`FEED_FAIL_DISABLE_AFTER`	required	`10`	Consecutive failures before a feed is disabled and surfaced.
`EXTRACTOR_VERSION`	required	`1`	Stamped on every extracted artefact so a re-extraction is traceable.

The build, in order

1
Save the page, not the link
Saving fetches immediately, keeps the raw bytes, and stays useful when extraction fails.
1. The page you can read today may not exist next year.
  Files src/capture/fetch.tstests/capture.test.ts
2. Extraction is derived and should be regenerable.
  Files src/capture/store.ts
3. A saved page you can still read beats a hidden error.
  Files src/capture/extract.tssrc/ui/ItemState.tsx
done when · tick each as it passes
watch out
- Saving the address and fetching it when you get round to reading is the design that quietly empties the archive: Pew Research Center measured in May 2024 that 38 percent of webpages that existed in 2013 were no longer reachable in 2023, that 8 percent of pages from 2023 were already gone, and that 23 percent of news pages contained at least one broken link. Fetch at save time, keep the raw bytes permanently, and treat the readable version as something derived that you can regenerate.
next phase
2
One item per article, however the link arrived
The same article saved from three places becomes one item, and two different pages never do.
1. The list is readable and arguable; a rule is neither.
  Files config/strip-params.txtsrc/url/normalise.ts
2. A cross-domain canonical is somebody else's claim about your page.
  Files src/url/canonical.tstests/canonical.test.ts
3. Deduplication is a guess and guesses need an undo.
  Files src/items/merge.tstests/merge.test.ts
done when · tick each as it passes
watch out
- Stripping every query parameter to get a tidy key merges pages whose parameter is the content, so a search result page and an article page become one item and one of them is lost; trusting a canonical link that points at another domain lets a syndicating site collapse your saved article into somebody else's page. Strip from a named list, restrict the canonical to the same registrable domain, and keep every merge reversible because deduplication is always a guess.
next phase
3
Highlights that survive the page being rewritten
A highlight either points at the words it was made on or says it is orphaned.
1. Either one alone fails in a predictable way.
  Files src/highlights/anchor.tstests/anchor.test.ts
2. Nearest occurrence, not first occurrence.
  Files src/highlights/reanchor.ts
3. The old capture is still on disk; link to it.
  Files src/ui/Highlight.tsx
done when · tick each as it passes
watch out
- Anchoring to a character offset in the live page means the day the site adds a consent banner paragraph every highlight points at the wrong sentence, while anchoring by quoted text alone breaks on any phrase that appears twice. Store both, re-anchor against a new capture by searching near the old offset, and let an orphan be visibly an orphan with a link to the capture where it still makes sense.
next phase
4
Feeds, and fetching without becoming a nuisance
Subscriptions poll conditionally, back off when refused, and say when a feed has stopped working.
1. It is the difference between polite and rude at scale.
  Files src/feeds/poll.tstests/conditional.test.ts
2. One slow site should not become forty simultaneous requests.
  Files src/fetch/limiter.ts
3. Silence is how an archive stops working unnoticed.
  Files src/ui/Feeds.tsx
done when · tick each as it passes
watch out
- Polling forty feeds every few minutes with no conditional request and no backoff makes your archive a small nuisance to every site you read and eventually gets it blocked, and a blocked archive stops saving without ever saying so. Send back the validators you were given, cap concurrency per host, escalate the interval on failure, and put a failing feed on the front page instead of in a log.
5
An archive that outlives this applicationproduct builder
Everything leaves in open formats, and a consistency check proves the files are still there.
1. A manifest pointing at files nobody exported is not a backup.
  Files src/export/bundle.tstests/export.test.ts
2. Silent corruption is the only kind that matters here.
  Files src/export/import.tstests/hash.test.ts
3. A non-zero exit is how this ends up in a cron entry.
  Files scripts/check.ts
done when · tick each as it passes
watch out
- An export that writes a JSON file referencing captures it did not include is a backup that restores into an archive full of missing pages, and you find out years later when the originals are also gone. Export the bytes beside the metadata, record a hash per file, verify those hashes on import, and run a periodic consistency check that names the items whose files have disappeared.

what this build does not replace

Paywall or authentication bypass.
Native mobile clients with background sync.
Licensed full-text feeds and recommendations.
Full-page archival with every asset.
native Apple polish, iCloud sync, Shortcuts, and mobile capture
publisher bypasses
cross-device mobile capture
high-quality recommendation graph
licensed full-text feeds

after v1, if you want it

Scheduled re-capture for items you starred, which is the only way to notice a page changing under a highlight
A reading-time estimate from the extracted word count, which costs nothing and changes what you open
An offline bundle for one tag, so a journey has a reading list that needs no network

Need the files? The project pack on the verdict page hands your agent the whole brief · more reading.