Vibecode Pinboard
track this build5 phases, 15 steps, beginner friendly0%YES · one-shottable
The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For Pinboard, run a fast private bookmark archive with tags, notes, and full-text search. The hard boundary is hosted longevity, simple api, archive service, and zero-maintenance reliability, plus capture polish, sync, and content partnerships.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Pinboard. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Pinboard · indie build Save a page from a bookmarklet or the command line, extract and store its readable text so it survives the site going away, tag and annotate it, search the full text instantly, and export the whole archive as plain files you could read without this application. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with a small HTTP server | one person's archive, where the work is fetching and indexing rather than serving traffic | | Extraction | Readability plus a sanitiser | the article without the navigation, the cookie banner and the newsletter prompt | | Storage | SQLite with full text search, plus the extracted text on disk | search wants an index; longevity wants files you can read without the index | | Capture | A bookmarklet and a command line entry point | no browser extension to maintain across browsers and their review processes | | Snapshots | Text and a single-file HTML snapshot, images optional | a full page archive with every asset is enormous and mostly worthless a year later | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the server, the fetcher and the indexer. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **Disk space proportional to your reading** · free - Why: Text snapshots are small, but ten thousand articles still add up, and the index roughly doubles it. - Get it: Estimate from your current bookmark count. Text-only archiving is modest; enabling images changes the picture entirely. - Verify: Free space comfortably exceeds your estimate - [ ] **Five awkward URLs to test extraction** · free - Why: Phase 2 is only meaningful against real pages: a clean article, a JavaScript-only page, one with an aggressive content security policy, one behind a soft wall and a very long page. - Get it: Collect them from your own reading. Keep them as fixtures so extraction changes can be compared. - Verify: All five are recorded with their expected extraction outcome - [ ] **A small server, for the product phase** (optional) · 5 to 10 USD a month, or free on hardware you own - Why: Only needed if you want the archive reachable from more than one machine. - Get it: Any small virtual server, or a machine at home with a tunnel. This archive is private, so authentication comes before exposure. - Verify: The archive is reachable over HTTPS and refuses unauthenticated requests ## Quick start ```sh npm install @mozilla/readability jsdom dompurify ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Archiving pages behind paywalls or logins. - Full asset archiving with images and stylesheets. - The proven longevity of a service that has run for years. - Convenient mobile capture without a browser extension. - hosted longevity, simple API, archive service, and zero-maintenance reliability - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds If one of those is essential to you, that is the reason to keep paying for Pinboard, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Pinboard The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private bookmark archive that saves the readable page, not just the link to replace Pinboard. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with a small HTTP server · one person's archive, where the work is fetching and indexing rather than serving traffic - Extraction: Readability plus a sanitiser · the article without the navigation, the cookie banner and the newsletter prompt - Storage: SQLite with full text search, plus the extracted text on disk · search wants an index; longevity wants files you can read without the index - Capture: A bookmarklet and a command line entry point · no browser extension to maintain across browsers and their review processes - Snapshots: Text and a single-file HTML snapshot, images optional · a full page archive with every asset is enormous and mostly worthless a year later ### Data model (create this before Phase 1) - bookmarks(id, url, canonical_url, title, description, added_at, read_at, starred, source) - tags(id, name), bookmark_tags(bookmark_id, tag_id) - snapshots(id, bookmark_id, kind, path, bytes, fetched_at, status) · kind is text, html or pdf - notes(id, bookmark_id, body, created_at) - bookmarks_fts · full text over title, description, notes and extracted text - The URL is normalised on save so the same page saved from two places is one bookmark, not two. ### Phase 1 · Save a link and normalise it Build: An endpoint that accepts a URL, title and tags. Normalise the URL: strip tracking parameters, resolve the canonical link where the page declares one, and unify the trivial variations. Saving an existing URL updates rather than duplicates. Trap: Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. Done when: The same article saved with and without tracking parameters produces one bookmark. A URL differing only by a trailing slash or scheme produces one bookmark. A URL whose query string genuinely identifies the content is not collapsed into another page. Do not build yet: No fetching, no extraction, no search. ### Phase 2 · Archiving the readable page Build: Fetch the page with a timeout and a byte cap, extract the readable content, sanitise it, and store both plain text and a self-contained HTML snapshot. Record failures as failures rather than leaving an empty snapshot. Trap: Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. Done when: A saved article's text is readable from the archive with the network disabled. A page requiring JavaScript to render is recorded as a failed extraction with the reason rather than an empty snapshot. Opening a stored snapshot makes no outbound network request, verified in developer tools. Do not build yet: No search and no tagging yet. ### Phase 3 · Tags, notes and fast search Build: Tagging with autocomplete, notes per bookmark, and full text search across titles, descriptions, notes and extracted text. Test at the size an archive reaches after a few years, not the size it is today. Trap: Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. Done when: Search across ten thousand bookmarks with archived text returns in well under a hundred milliseconds. Searching a phrase that appears only in an article body finds it. Tag filtering combines with text search. Rebuilding the index from the stored text produces identical results. Do not build yet: No capture tooling or export yet. ### Phase 4 · Capture that takes one click Build: A bookmarklet that posts the current page with its title and selection. A command line entry point for batch saving. An import for bookmarks exported from a browser or another service. Trap: A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. Done when: The bookmarklet saves the current page from three different sites, including one with an aggressive content security policy, or fails with a visible message on that site. Importing a browser bookmarks export creates bookmarks without duplicates. The command line entry point saves a URL and returns the identifier. Do not build yet: No export yet. ### Phase 5 · Export that makes leaving easy Build: Export as JSON with everything, as the standard bookmarks HTML format, and as a directory of Markdown files with the extracted text. Verify that a JSON export imports back into an empty installation exactly. Trap: An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. Done when: A JSON export imports into an empty installation and reproduces every bookmark, tag, note and snapshot, verified by checksum. The Markdown export is readable in any editor with front matter carrying the URL and tags. The bookmarks HTML export imports into a browser. Do not build yet: Hosting and automation come next. ### Phase 6 · Hosting it, refreshing it and keeping it Build: Put it behind authentication and TLS. Periodically re-check saved URLs and mark the dead ones, which is the moment the archive proves its worth. Back up the database and the snapshot directory, and restore both onto a clean machine. Trap: Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months. Done when: An unauthenticated request cannot read or save. A link-check pass marks a deliberately broken URL as dead and leaves its archived text readable, which is the entire point. A restore onto a clean machine reproduces bookmarks and snapshots, and the date is recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - Archiving pages behind paywalls or logins. It is the obvious gap and working around access controls is not something this build does. - Full asset archiving with images, stylesheets and fonts. It multiplies storage many times over for material that is rarely what you came back for. - The hosted service's longevity. A service that has run for years has proved something your own instance has not, and the maintenance is now yours. - Mobile capture without a browser extension, which is genuinely awkward and the bookmarklet only partly solves. ### README must contain - Which tracking parameters are stripped and why unknown parameters are kept - That snapshots are sanitised at capture and make no outbound requests when opened - That a page requiring JavaScript is recorded as a failed snapshot rather than an empty one - That the Markdown export is the format designed to outlive this application - The link check interval and concurrency, and that dead links keep their archived text ===== AGENTS.md ===== # Agent instructions · Pinboard indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with a small HTTP server, Readability plus a sanitiser, SQLite with full text search, plus the extracted text on disk, A bookmarklet and a command line entry point, Text and a single-file HTML snapshot, images optional. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. ===== BUILD_PLAN.md ===== # Build plan · Pinboard Save a page from a bookmarklet or the command line, extract and store its readable text so it survives the site going away, tag and annotate it, search the full text instantly, and export the whole archive as plain files you could read without this application. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Save a link and normalise it A URL becomes a bookmark, and the same page saved twice is one entry. ### Steps 1. Build the save endpoint URL, optional title, optional tags. Return the bookmark so a caller knows whether it created or updated. Files: `src/save.js` 2. Normalise carefully Strip known tracking parameters, lowercase the host, remove a default port and a trailing slash. Do not strip unknown parameters, because some of them are the page. Files: `src/normalise.js` 3. Test the collapsing rules A table of URL pairs with expected same or different verdicts. This is where over-eager normalisation gets caught. Files: `test/normalise.test.js` ### Done when - [ ] The same article with and without tracking parameters is one bookmark - [ ] Scheme and trailing slash variations collapse to one - [ ] A content-identifying query string is preserved - [ ] Re-saving updates rather than duplicating ### Watch out - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. ## Phase 2 · Archiving the readable page The article survives the site disappearing. ### Steps 1. Fetch with limits and respect robots Timeout, byte cap enforced during streaming, and a check of robots.txt. You are a well-behaved client saving pages for yourself. Files: `src/fetch.js` 2. Extract and sanitise Readability for the article, then strip scripts, event handlers and remote references so the snapshot is inert and self-contained. Files: `src/extract.js` ```sh npm install @mozilla/readability jsdom dompurify ``` 3. Record failure honestly A page that needs JavaScript, returns an error or is behind a wall is a failed snapshot with a reason. An empty snapshot pretending to be a success is worse than nothing. Files: `src/extract.js` ### Done when - [ ] A saved article is readable offline - [ ] A JavaScript-dependent page is recorded as a failure with a reason - [ ] An opened snapshot makes no outbound requests - [ ] The byte cap is enforced during download, not after ### Watch out - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. ## Phase 3 · Tags, notes and fast search Finding something you saved two years ago takes a few seconds. ### Steps 1. Generate a realistic archive for testing Ten thousand bookmarks with article-length text. Without this fixture every performance claim is a guess. Files: `scripts/generate-archive.js` 2. Index everything searchable Title, description, notes and extracted text in one full text index, updated on save. Files: `src/search.js` 3. Make rebuild a first-class action The index is derived from the stored text, so rebuilding must be possible and must produce identical results. Files: `src/search.js` ### Done when - [ ] Search over ten thousand bookmarks returns in under a hundred milliseconds - [ ] A phrase only in an article body is findable - [ ] Tag filters combine with text search - [ ] Index rebuild produces identical results ### Watch out - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. ## Phase 4 · Capture that takes one click Saving a page is fast enough that you actually do it. ### Steps 1. Write the bookmarklet Title, URL and any selected text. Keep it small enough to paste, and use a save-only token. Files: `public/bookmarklet.js` 2. Handle sites that block it A strict content security policy will stop a bookmarklet. Detect and tell the user rather than failing silently, and offer the command line as the fallback. Files: `public/bookmarklet.js` 3. Import from browsers and services The standard bookmarks HTML export, plus at least one service's JSON. Deduplicate through the same normalisation as saving. Files: `src/import.js` ### Done when - [ ] The bookmarklet saves from three different sites - [ ] A site blocking it produces a visible message - [ ] A browser export imports without duplicates - [ ] The command line entry point saves and returns an identifier ### Watch out - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. ## Phase 5 · Export that makes leaving easy The archive is portable, and its value does not depend on this application existing. ### Steps 1. Export lossless JSON Everything including snapshot paths, with a schema version. Files: `src/export.js` 2. Export a Markdown directory One file per bookmark with front matter and the extracted text. This is the format that outlives the application. Files: `src/export.js` 3. Verify the round trip Import into an empty database and compare a normalised checksum. Files: `test/roundtrip.test.js` ### Done when - [ ] A JSON export round-trips by checksum - [ ] Markdown files are readable in any editor with URL and tags in front matter - [ ] The bookmarks HTML export imports into a browser - [ ] Snapshots are included or their absence is explicit ### Watch out - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. ## Not in this build - Archiving pages behind paywalls or logins. - Full asset archiving with images and stylesheets. - The proven longevity of a service that has run for years. - Convenient mobile capture without a browser extension. ## After v1, if you want it - A weekly digest of pages saved but never opened - Optional image archiving for a subset of tags where it actually matters - Public sharing of a single tag as a static page ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. SQLite database with bookmarks, tags, notes and the search index. DB_PATH=./data/archive.db # Required. Where extracted text and HTML snapshots are written. Back this up alongside the database. SNAPSHOT_DIR=./data/snapshots # Required. Per-page fetch timeout. A slow page should fail rather than hold up a save. FETCH_TIMEOUT_MS=10000 # Required. Streaming byte cap per page, enforced during download. FETCH_MAX_BYTES=5000000 # Required · secret. Used by the bookmarklet. Scoped to saving only, because a bookmarklet cannot keep a secret well. SAVE_TOKEN=replace-with-a-random-token # Required. How often each URL is re-checked. Link rot works on months, so checking often is rude and pointless. LINKCHECK_INTERVAL_DAYS=90 # Required. Simultaneous checks. Low, and per-host limited, so you do not look like a crawler. LINKCHECK_CONCURRENCY=2
You are building a lean indie version of Pinboard. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Pinboard · indie build Save a page from a bookmarklet or the command line, extract and store its readable text so it survives the site going away, tag and annotate it, search the full text instantly, and export the whole archive as plain files you could read without this application. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with a small HTTP server | one person's archive, where the work is fetching and indexing rather than serving traffic | | Extraction | Readability plus a sanitiser | the article without the navigation, the cookie banner and the newsletter prompt | | Storage | SQLite with full text search, plus the extracted text on disk | search wants an index; longevity wants files you can read without the index | | Capture | A bookmarklet and a command line entry point | no browser extension to maintain across browsers and their review processes | | Snapshots | Text and a single-file HTML snapshot, images optional | a full page archive with every asset is enormous and mostly worthless a year later | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the server, the fetcher and the indexer. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **Disk space proportional to your reading** · free - Why: Text snapshots are small, but ten thousand articles still add up, and the index roughly doubles it. - Get it: Estimate from your current bookmark count. Text-only archiving is modest; enabling images changes the picture entirely. - Verify: Free space comfortably exceeds your estimate - [ ] **Five awkward URLs to test extraction** · free - Why: Phase 2 is only meaningful against real pages: a clean article, a JavaScript-only page, one with an aggressive content security policy, one behind a soft wall and a very long page. - Get it: Collect them from your own reading. Keep them as fixtures so extraction changes can be compared. - Verify: All five are recorded with their expected extraction outcome - [ ] **A small server, for the product phase** (optional) · 5 to 10 USD a month, or free on hardware you own - Why: Only needed if you want the archive reachable from more than one machine. - Get it: Any small virtual server, or a machine at home with a tunnel. This archive is private, so authentication comes before exposure. - Verify: The archive is reachable over HTTPS and refuses unauthenticated requests ## Quick start ```sh npm install @mozilla/readability jsdom dompurify ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Archiving pages behind paywalls or logins. - Full asset archiving with images and stylesheets. - The proven longevity of a service that has run for years. - Convenient mobile capture without a browser extension. - hosted longevity, simple API, archive service, and zero-maintenance reliability - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds If one of those is essential to you, that is the reason to keep paying for Pinboard, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Pinboard The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private bookmark archive that saves the readable page, not just the link to replace Pinboard. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with a small HTTP server · one person's archive, where the work is fetching and indexing rather than serving traffic - Extraction: Readability plus a sanitiser · the article without the navigation, the cookie banner and the newsletter prompt - Storage: SQLite with full text search, plus the extracted text on disk · search wants an index; longevity wants files you can read without the index - Capture: A bookmarklet and a command line entry point · no browser extension to maintain across browsers and their review processes - Snapshots: Text and a single-file HTML snapshot, images optional · a full page archive with every asset is enormous and mostly worthless a year later ### Data model (create this before Phase 1) - bookmarks(id, url, canonical_url, title, description, added_at, read_at, starred, source) - tags(id, name), bookmark_tags(bookmark_id, tag_id) - snapshots(id, bookmark_id, kind, path, bytes, fetched_at, status) · kind is text, html or pdf - notes(id, bookmark_id, body, created_at) - bookmarks_fts · full text over title, description, notes and extracted text - The URL is normalised on save so the same page saved from two places is one bookmark, not two. ### Phase 1 · Save a link and normalise it Build: An endpoint that accepts a URL, title and tags. Normalise the URL: strip tracking parameters, resolve the canonical link where the page declares one, and unify the trivial variations. Saving an existing URL updates rather than duplicates. Trap: Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. Done when: The same article saved with and without tracking parameters produces one bookmark. A URL differing only by a trailing slash or scheme produces one bookmark. A URL whose query string genuinely identifies the content is not collapsed into another page. Do not build yet: No fetching, no extraction, no search. ### Phase 2 · Archiving the readable page Build: Fetch the page with a timeout and a byte cap, extract the readable content, sanitise it, and store both plain text and a self-contained HTML snapshot. Record failures as failures rather than leaving an empty snapshot. Trap: Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. Done when: A saved article's text is readable from the archive with the network disabled. A page requiring JavaScript to render is recorded as a failed extraction with the reason rather than an empty snapshot. Opening a stored snapshot makes no outbound network request, verified in developer tools. Do not build yet: No search and no tagging yet. ### Phase 3 · Tags, notes and fast search Build: Tagging with autocomplete, notes per bookmark, and full text search across titles, descriptions, notes and extracted text. Test at the size an archive reaches after a few years, not the size it is today. Trap: Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. Done when: Search across ten thousand bookmarks with archived text returns in well under a hundred milliseconds. Searching a phrase that appears only in an article body finds it. Tag filtering combines with text search. Rebuilding the index from the stored text produces identical results. Do not build yet: No capture tooling or export yet. ### Phase 4 · Capture that takes one click Build: A bookmarklet that posts the current page with its title and selection. A command line entry point for batch saving. An import for bookmarks exported from a browser or another service. Trap: A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. Done when: The bookmarklet saves the current page from three different sites, including one with an aggressive content security policy, or fails with a visible message on that site. Importing a browser bookmarks export creates bookmarks without duplicates. The command line entry point saves a URL and returns the identifier. Do not build yet: No export yet. ### Phase 5 · Export that makes leaving easy Build: Export as JSON with everything, as the standard bookmarks HTML format, and as a directory of Markdown files with the extracted text. Verify that a JSON export imports back into an empty installation exactly. Trap: An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. Done when: A JSON export imports into an empty installation and reproduces every bookmark, tag, note and snapshot, verified by checksum. The Markdown export is readable in any editor with front matter carrying the URL and tags. The bookmarks HTML export imports into a browser. Do not build yet: Hosting and automation come next. ### Phase 6 · Hosting it, refreshing it and keeping it Build: Put it behind authentication and TLS. Periodically re-check saved URLs and mark the dead ones, which is the moment the archive proves its worth. Back up the database and the snapshot directory, and restore both onto a clean machine. Trap: Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months. Done when: An unauthenticated request cannot read or save. A link-check pass marks a deliberately broken URL as dead and leaves its archived text readable, which is the entire point. A restore onto a clean machine reproduces bookmarks and snapshots, and the date is recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - Archiving pages behind paywalls or logins. It is the obvious gap and working around access controls is not something this build does. - Full asset archiving with images, stylesheets and fonts. It multiplies storage many times over for material that is rarely what you came back for. - The hosted service's longevity. A service that has run for years has proved something your own instance has not, and the maintenance is now yours. - Mobile capture without a browser extension, which is genuinely awkward and the bookmarklet only partly solves. ### README must contain - Which tracking parameters are stripped and why unknown parameters are kept - That snapshots are sanitised at capture and make no outbound requests when opened - That a page requiring JavaScript is recorded as a failed snapshot rather than an empty one - That the Markdown export is the format designed to outlive this application - The link check interval and concurrency, and that dead links keep their archived text ===== AGENTS.md ===== # Agent instructions · Pinboard indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with a small HTTP server, Readability plus a sanitiser, SQLite with full text search, plus the extracted text on disk, A bookmarklet and a command line entry point, Text and a single-file HTML snapshot, images optional. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. ===== BUILD_PLAN.md ===== # Build plan · Pinboard Save a page from a bookmarklet or the command line, extract and store its readable text so it survives the site going away, tag and annotate it, search the full text instantly, and export the whole archive as plain files you could read without this application. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Save a link and normalise it A URL becomes a bookmark, and the same page saved twice is one entry. ### Steps 1. Build the save endpoint URL, optional title, optional tags. Return the bookmark so a caller knows whether it created or updated. Files: `src/save.js` 2. Normalise carefully Strip known tracking parameters, lowercase the host, remove a default port and a trailing slash. Do not strip unknown parameters, because some of them are the page. Files: `src/normalise.js` 3. Test the collapsing rules A table of URL pairs with expected same or different verdicts. This is where over-eager normalisation gets caught. Files: `test/normalise.test.js` ### Done when - [ ] The same article with and without tracking parameters is one bookmark - [ ] Scheme and trailing slash variations collapse to one - [ ] A content-identifying query string is preserved - [ ] Re-saving updates rather than duplicating ### Watch out - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. ## Phase 2 · Archiving the readable page The article survives the site disappearing. ### Steps 1. Fetch with limits and respect robots Timeout, byte cap enforced during streaming, and a check of robots.txt. You are a well-behaved client saving pages for yourself. Files: `src/fetch.js` 2. Extract and sanitise Readability for the article, then strip scripts, event handlers and remote references so the snapshot is inert and self-contained. Files: `src/extract.js` ```sh npm install @mozilla/readability jsdom dompurify ``` 3. Record failure honestly A page that needs JavaScript, returns an error or is behind a wall is a failed snapshot with a reason. An empty snapshot pretending to be a success is worse than nothing. Files: `src/extract.js` ### Done when - [ ] A saved article is readable offline - [ ] A JavaScript-dependent page is recorded as a failure with a reason - [ ] An opened snapshot makes no outbound requests - [ ] The byte cap is enforced during download, not after ### Watch out - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. ## Phase 3 · Tags, notes and fast search Finding something you saved two years ago takes a few seconds. ### Steps 1. Generate a realistic archive for testing Ten thousand bookmarks with article-length text. Without this fixture every performance claim is a guess. Files: `scripts/generate-archive.js` 2. Index everything searchable Title, description, notes and extracted text in one full text index, updated on save. Files: `src/search.js` 3. Make rebuild a first-class action The index is derived from the stored text, so rebuilding must be possible and must produce identical results. Files: `src/search.js` ### Done when - [ ] Search over ten thousand bookmarks returns in under a hundred milliseconds - [ ] A phrase only in an article body is findable - [ ] Tag filters combine with text search - [ ] Index rebuild produces identical results ### Watch out - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. ## Phase 4 · Capture that takes one click Saving a page is fast enough that you actually do it. ### Steps 1. Write the bookmarklet Title, URL and any selected text. Keep it small enough to paste, and use a save-only token. Files: `public/bookmarklet.js` 2. Handle sites that block it A strict content security policy will stop a bookmarklet. Detect and tell the user rather than failing silently, and offer the command line as the fallback. Files: `public/bookmarklet.js` 3. Import from browsers and services The standard bookmarks HTML export, plus at least one service's JSON. Deduplicate through the same normalisation as saving. Files: `src/import.js` ### Done when - [ ] The bookmarklet saves from three different sites - [ ] A site blocking it produces a visible message - [ ] A browser export imports without duplicates - [ ] The command line entry point saves and returns an identifier ### Watch out - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. ## Phase 5 · Export that makes leaving easy The archive is portable, and its value does not depend on this application existing. ### Steps 1. Export lossless JSON Everything including snapshot paths, with a schema version. Files: `src/export.js` 2. Export a Markdown directory One file per bookmark with front matter and the extracted text. This is the format that outlives the application. Files: `src/export.js` 3. Verify the round trip Import into an empty database and compare a normalised checksum. Files: `test/roundtrip.test.js` ### Done when - [ ] A JSON export round-trips by checksum - [ ] Markdown files are readable in any editor with URL and tags in front matter - [ ] The bookmarks HTML export imports into a browser - [ ] Snapshots are included or their absence is explicit ### Watch out - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. ## Not in this build - Archiving pages behind paywalls or logins. - Full asset archiving with images and stylesheets. - The proven longevity of a service that has run for years. - Convenient mobile capture without a browser extension. ## After v1, if you want it - A weekly digest of pages saved but never opened - Optional image archiving for a subset of tags where it actually matters - Public sharing of a single tag as a static page ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. SQLite database with bookmarks, tags, notes and the search index. DB_PATH=./data/archive.db # Required. Where extracted text and HTML snapshots are written. Back this up alongside the database. SNAPSHOT_DIR=./data/snapshots # Required. Per-page fetch timeout. A slow page should fail rather than hold up a save. FETCH_TIMEOUT_MS=10000 # Required. Streaming byte cap per page, enforced during download. FETCH_MAX_BYTES=5000000 # Required · secret. Used by the bookmarklet. Scoped to saving only, because a bookmarklet cannot keep a secret well. SAVE_TOKEN=replace-with-a-random-token # Required. How often each URL is re-checked. Link rot works on months, so checking often is rude and pointless. LINKCHECK_INTERVAL_DAYS=90 # Required. Simultaneous checks. Low, and per-host limited, so you do not look like a crawler. LINKCHECK_CONCURRENCY=2
You are building a production product version of Pinboard. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Pinboard · product brief ## Problem The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For Pinboard, run a fast private bookmark archive with tags, notes, and full-text search. The hard boundary is hosted longevity, simple api, archive service, and zero-maintenance reliability, plus capture polish, sync, and content partnerships. ## Product outcome A private archive where the article survives the site, search is instant at ten thousand entries, snapshots are inert, and the whole thing exports to Markdown files that need none of this software. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - local disk - optional browser bookmarklet - user-supplied RSS feeds ## Explicit non-goals for v1 - Archiving pages behind paywalls or logins. - Full asset archiving with images and stylesheets. - The proven longevity of a service that has run for years. - Convenient mobile capture without a browser extension. - hosted longevity, simple API, archive service, and zero-maintenance reliability - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds ## Success criteria - Search latency on the ten thousand entry fixture is measured and has not regressed - A stored snapshot is confirmed to make no outbound requests - The JSON export round-trips by checksum - Markdown export is confirmed readable in a plain editor - A restore of both database and snapshots has been performed and dated ===== BRIEF.md ===== # Build brief · Pinboard The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private bookmark archive that saves the readable page, not just the link to replace Pinboard. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with a small HTTP server · one person's archive, where the work is fetching and indexing rather than serving traffic - Extraction: Readability plus a sanitiser · the article without the navigation, the cookie banner and the newsletter prompt - Storage: SQLite with full text search, plus the extracted text on disk · search wants an index; longevity wants files you can read without the index - Capture: A bookmarklet and a command line entry point · no browser extension to maintain across browsers and their review processes - Snapshots: Text and a single-file HTML snapshot, images optional · a full page archive with every asset is enormous and mostly worthless a year later ### Data model (create this before Phase 1) - bookmarks(id, url, canonical_url, title, description, added_at, read_at, starred, source) - tags(id, name), bookmark_tags(bookmark_id, tag_id) - snapshots(id, bookmark_id, kind, path, bytes, fetched_at, status) · kind is text, html or pdf - notes(id, bookmark_id, body, created_at) - bookmarks_fts · full text over title, description, notes and extracted text - The URL is normalised on save so the same page saved from two places is one bookmark, not two. ### Phase 1 · Save a link and normalise it Build: An endpoint that accepts a URL, title and tags. Normalise the URL: strip tracking parameters, resolve the canonical link where the page declares one, and unify the trivial variations. Saving an existing URL updates rather than duplicates. Trap: Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. Done when: The same article saved with and without tracking parameters produces one bookmark. A URL differing only by a trailing slash or scheme produces one bookmark. A URL whose query string genuinely identifies the content is not collapsed into another page. Do not build yet: No fetching, no extraction, no search. ### Phase 2 · Archiving the readable page Build: Fetch the page with a timeout and a byte cap, extract the readable content, sanitise it, and store both plain text and a self-contained HTML snapshot. Record failures as failures rather than leaving an empty snapshot. Trap: Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. Done when: A saved article's text is readable from the archive with the network disabled. A page requiring JavaScript to render is recorded as a failed extraction with the reason rather than an empty snapshot. Opening a stored snapshot makes no outbound network request, verified in developer tools. Do not build yet: No search and no tagging yet. ### Phase 3 · Tags, notes and fast search Build: Tagging with autocomplete, notes per bookmark, and full text search across titles, descriptions, notes and extracted text. Test at the size an archive reaches after a few years, not the size it is today. Trap: Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. Done when: Search across ten thousand bookmarks with archived text returns in well under a hundred milliseconds. Searching a phrase that appears only in an article body finds it. Tag filtering combines with text search. Rebuilding the index from the stored text produces identical results. Do not build yet: No capture tooling or export yet. ### Phase 4 · Capture that takes one click Build: A bookmarklet that posts the current page with its title and selection. A command line entry point for batch saving. An import for bookmarks exported from a browser or another service. Trap: A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. Done when: The bookmarklet saves the current page from three different sites, including one with an aggressive content security policy, or fails with a visible message on that site. Importing a browser bookmarks export creates bookmarks without duplicates. The command line entry point saves a URL and returns the identifier. Do not build yet: No export yet. ### Phase 5 · Export that makes leaving easy Build: Export as JSON with everything, as the standard bookmarks HTML format, and as a directory of Markdown files with the extracted text. Verify that a JSON export imports back into an empty installation exactly. Trap: An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. Done when: A JSON export imports into an empty installation and reproduces every bookmark, tag, note and snapshot, verified by checksum. The Markdown export is readable in any editor with front matter carrying the URL and tags. The bookmarks HTML export imports into a browser. Do not build yet: Hosting and automation come next. ### Phase 6 · Hosting it, refreshing it and keeping it Build: Put it behind authentication and TLS. Periodically re-check saved URLs and mark the dead ones, which is the moment the archive proves its worth. Back up the database and the snapshot directory, and restore both onto a clean machine. Trap: Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months. Done when: An unauthenticated request cannot read or save. A link-check pass marks a deliberately broken URL as dead and leaves its archived text readable, which is the entire point. A restore onto a clean machine reproduces bookmarks and snapshots, and the date is recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - Archiving pages behind paywalls or logins. It is the obvious gap and working around access controls is not something this build does. - Full asset archiving with images, stylesheets and fonts. It multiplies storage many times over for material that is rarely what you came back for. - The hosted service's longevity. A service that has run for years has proved something your own instance has not, and the maintenance is now yours. - Mobile capture without a browser extension, which is genuinely awkward and the bookmarklet only partly solves. ### README must contain - Which tracking parameters are stripped and why unknown parameters are kept - That snapshots are sanitised at capture and make no outbound requests when opened - That a page requiring JavaScript is recorded as a failed snapshot rather than an empty one - That the Markdown export is the format designed to outlive this application - The link check interval and concurrency, and that dead links keep their archived text ===== ARCHITECTURE.md ===== # Architecture · Pinboard ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with a small HTTP server | one person's archive, where the work is fetching and indexing rather than serving traffic | | Extraction | Readability plus a sanitiser | the article without the navigation, the cookie banner and the newsletter prompt | | Storage | SQLite with full text search, plus the extracted text on disk | search wants an index; longevity wants files you can read without the index | | Capture | A bookmarklet and a command line entry point | no browser extension to maintain across browsers and their review processes | | Snapshots | Text and a single-file HTML snapshot, images optional | a full page archive with every asset is enormous and mostly worthless a year later | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Normalise | Turning a URL into a stable identity. | A pure function with a test table. Over-eager stripping is the failure mode, so any change needs the table updated first. | | Fetch and extract | Getting the article and making it inert. | A headless browser would handle JavaScript-dependent pages at a large cost in speed and complexity. Keep it behind the same interface if you add it. | | Store | Bookmarks, snapshots and the search index. | The index is derived and rebuildable; the snapshots are not. That asymmetry decides what gets backed up. | | Capture | Bookmarklet and command line entry points. | An extension would be more capable and more secure, at the cost of maintaining it per browser. | | Export | Portability. | Never remove the Markdown export. It is the promise that the archive is yours. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DB_PATH` · required · SQLite database with bookmarks, tags, notes and the search index. - `SNAPSHOT_DIR` · required · Where extracted text and HTML snapshots are written. Back this up alongside the database. - `FETCH_TIMEOUT_MS` · required · Per-page fetch timeout. A slow page should fail rather than hold up a save. - `FETCH_MAX_BYTES` · required · Streaming byte cap per page, enforced during download. - `SAVE_TOKEN` · required, secret · Used by the bookmarklet. Scoped to saving only, because a bookmarklet cannot keep a secret well. - `LINKCHECK_INTERVAL_DAYS` · required · How often each URL is re-checked. Link rot works on months, so checking often is rude and pointless. - `LINKCHECK_CONCURRENCY` · required · Simultaneous checks. Low, and per-host limited, so you do not look like a crawler. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Pinboard product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with a small HTTP server, Readability plus a sanitiser, SQLite with full text search, plus the extracted text on disk, A bookmarklet and a command line entry point, Text and a single-file HTML snapshot, images optional. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. - Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months. ===== MILESTONES.md ===== # Delivery milestones · Pinboard Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Save a link and normalise it A URL becomes a bookmark, and the same page saved twice is one entry. ### Steps 1. Build the save endpoint URL, optional title, optional tags. Return the bookmark so a caller knows whether it created or updated. Files: `src/save.js` 2. Normalise carefully Strip known tracking parameters, lowercase the host, remove a default port and a trailing slash. Do not strip unknown parameters, because some of them are the page. Files: `src/normalise.js` 3. Test the collapsing rules A table of URL pairs with expected same or different verdicts. This is where over-eager normalisation gets caught. Files: `test/normalise.test.js` ### Done when - [ ] The same article with and without tracking parameters is one bookmark - [ ] Scheme and trailing slash variations collapse to one - [ ] A content-identifying query string is preserved - [ ] Re-saving updates rather than duplicating ### Watch out - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. ## M2 · Archiving the readable page The article survives the site disappearing. ### Steps 1. Fetch with limits and respect robots Timeout, byte cap enforced during streaming, and a check of robots.txt. You are a well-behaved client saving pages for yourself. Files: `src/fetch.js` 2. Extract and sanitise Readability for the article, then strip scripts, event handlers and remote references so the snapshot is inert and self-contained. Files: `src/extract.js` ```sh npm install @mozilla/readability jsdom dompurify ``` 3. Record failure honestly A page that needs JavaScript, returns an error or is behind a wall is a failed snapshot with a reason. An empty snapshot pretending to be a success is worse than nothing. Files: `src/extract.js` ### Done when - [ ] A saved article is readable offline - [ ] A JavaScript-dependent page is recorded as a failure with a reason - [ ] An opened snapshot makes no outbound requests - [ ] The byte cap is enforced during download, not after ### Watch out - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. ## M3 · Tags, notes and fast search Finding something you saved two years ago takes a few seconds. ### Steps 1. Generate a realistic archive for testing Ten thousand bookmarks with article-length text. Without this fixture every performance claim is a guess. Files: `scripts/generate-archive.js` 2. Index everything searchable Title, description, notes and extracted text in one full text index, updated on save. Files: `src/search.js` 3. Make rebuild a first-class action The index is derived from the stored text, so rebuilding must be possible and must produce identical results. Files: `src/search.js` ### Done when - [ ] Search over ten thousand bookmarks returns in under a hundred milliseconds - [ ] A phrase only in an article body is findable - [ ] Tag filters combine with text search - [ ] Index rebuild produces identical results ### Watch out - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. ## M4 · Capture that takes one click Saving a page is fast enough that you actually do it. ### Steps 1. Write the bookmarklet Title, URL and any selected text. Keep it small enough to paste, and use a save-only token. Files: `public/bookmarklet.js` 2. Handle sites that block it A strict content security policy will stop a bookmarklet. Detect and tell the user rather than failing silently, and offer the command line as the fallback. Files: `public/bookmarklet.js` 3. Import from browsers and services The standard bookmarks HTML export, plus at least one service's JSON. Deduplicate through the same normalisation as saving. Files: `src/import.js` ### Done when - [ ] The bookmarklet saves from three different sites - [ ] A site blocking it produces a visible message - [ ] A browser export imports without duplicates - [ ] The command line entry point saves and returns an identifier ### Watch out - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. ## M5 · Export that makes leaving easy The archive is portable, and its value does not depend on this application existing. ### Steps 1. Export lossless JSON Everything including snapshot paths, with a schema version. Files: `src/export.js` 2. Export a Markdown directory One file per bookmark with front matter and the extracted text. This is the format that outlives the application. Files: `src/export.js` 3. Verify the round trip Import into an empty database and compare a normalised checksum. Files: `test/roundtrip.test.js` ### Done when - [ ] A JSON export round-trips by checksum - [ ] Markdown files are readable in any editor with URL and tags in front matter - [ ] The bookmarks HTML export imports into a browser - [ ] Snapshots are included or their absence is explicit ### Watch out - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. ## M6 · Hosting it, refreshing it and keeping it (production only) It runs somewhere reachable, notices dead links, and you have restored it once. ### Steps 1. Authenticate and serve over TLS A private archive of what you read is sensitive. Put a login in front and get a certificate. Files: `src/auth.js`, `Caddyfile` 2. Check links gently Low concurrency, per-host limits, and a long interval. Mark dead rather than deleting, because the snapshot is now the only copy. Files: `src/linkcheck.js` 3. Back up both halves and restore The database and the snapshot directory. Restore onto a clean machine and record the date. Files: `scripts/backup.sh`, `README.md` ### Done when - [ ] Unauthenticated requests cannot read or save - [ ] A dead link is marked and its archived text is still readable - [ ] Link checking stays within polite concurrency limits - [ ] A restore has been performed on a clean machine and dated ### Watch out - Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months. ===== OPERATIONS.md ===== # Operations · Pinboard ## Backup The database and the snapshot directory together. The database alone leaves you with metadata pointing at nothing, which is exactly the situation archiving exists to prevent. ## Restore Restore both, rebuild the index if needed, and open a bookmark whose original site is dead. That is the test that the archive is real. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Failed extraction rate and dead link count. A jump in extraction failures usually means a change in the extraction library or a shift in how sites render, not a change in your reading. ## Incident checklist If your address gets blocked by a site, reduce link check frequency and concurrency first. If snapshots are lost but the database survives, the archive is metadata only and the lesson is that the two halves must be backed up together. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Search latency on the ten thousand entry fixture is measured and has not regressed - [ ] A stored snapshot is confirmed to make no outbound requests - [ ] The JSON export round-trips by checksum - [ ] Markdown export is confirmed readable in a plain editor - [ ] A restore of both database and snapshots has been performed and dated ## Launch constraint Do not market omitted Pinboard capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. SQLite database with bookmarks, tags, notes and the search index. DB_PATH=./data/archive.db # Required. Where extracted text and HTML snapshots are written. Back this up alongside the database. SNAPSHOT_DIR=./data/snapshots # Required. Per-page fetch timeout. A slow page should fail rather than hold up a save. FETCH_TIMEOUT_MS=10000 # Required. Streaming byte cap per page, enforced during download. FETCH_MAX_BYTES=5000000 # Required · secret. Used by the bookmarklet. Scoped to saving only, because a bookmarklet cannot keep a secret well. SAVE_TOKEN=replace-with-a-random-token # Required. How often each URL is re-checked. Link rot works on months, so checking often is rude and pointless. LINKCHECK_INTERVAL_DAYS=90 # Required. Simultaneous checks. Low, and per-host limited, so you do not look like a crawler. LINKCHECK_CONCURRENCY=2
README.md
# Pinboard · indie build Save a page from a bookmarklet or the command line, extract and store its readable text so it survives the site going away, tag and annotate it, search the full text instantly, and export the whole archive as plain files you could read without this application. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with a small HTTP server | one person's archive, where the work is fetching and indexing rather than serving traffic | | Extraction | Readability plus a sanitiser | the article without the navigation, the cookie banner and the newsletter prompt | | Storage | SQLite with full text search, plus the extracted text on disk | search wants an index; longevity wants files you can read without the index | | Capture | A bookmarklet and a command line entry point | no browser extension to maintain across browsers and their review processes | | Snapshots | Text and a single-file HTML snapshot, images optional | a full page archive with every asset is enormous and mostly worthless a year later | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the server, the fetcher and the indexer. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **Disk space proportional to your reading** · free - Why: Text snapshots are small, but ten thousand articles still add up, and the index roughly doubles it. - Get it: Estimate from your current bookmark count. Text-only archiving is modest; enabling images changes the picture entirely. - Verify: Free space comfortably exceeds your estimate - [ ] **Five awkward URLs to test extraction** · free - Why: Phase 2 is only meaningful against real pages: a clean article, a JavaScript-only page, one with an aggressive content security policy, one behind a soft wall and a very long page. - Get it: Collect them from your own reading. Keep them as fixtures so extraction changes can be compared. - Verify: All five are recorded with their expected extraction outcome - [ ] **A small server, for the product phase** (optional) · 5 to 10 USD a month, or free on hardware you own - Why: Only needed if you want the archive reachable from more than one machine. - Get it: Any small virtual server, or a machine at home with a tunnel. This archive is private, so authentication comes before exposure. - Verify: The archive is reachable over HTTPS and refuses unauthenticated requests ## Quick start ```sh npm install @mozilla/readability jsdom dompurify ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Archiving pages behind paywalls or logins. - Full asset archiving with images and stylesheets. - The proven longevity of a service that has run for years. - Convenient mobile capture without a browser extension. - hosted longevity, simple API, archive service, and zero-maintenance reliability - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds If one of those is essential to you, that is the reason to keep paying for Pinboard, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Pinboard The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private bookmark archive that saves the readable page, not just the link to replace Pinboard. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with a small HTTP server · one person's archive, where the work is fetching and indexing rather than serving traffic - Extraction: Readability plus a sanitiser · the article without the navigation, the cookie banner and the newsletter prompt - Storage: SQLite with full text search, plus the extracted text on disk · search wants an index; longevity wants files you can read without the index - Capture: A bookmarklet and a command line entry point · no browser extension to maintain across browsers and their review processes - Snapshots: Text and a single-file HTML snapshot, images optional · a full page archive with every asset is enormous and mostly worthless a year later ### Data model (create this before Phase 1) - bookmarks(id, url, canonical_url, title, description, added_at, read_at, starred, source) - tags(id, name), bookmark_tags(bookmark_id, tag_id) - snapshots(id, bookmark_id, kind, path, bytes, fetched_at, status) · kind is text, html or pdf - notes(id, bookmark_id, body, created_at) - bookmarks_fts · full text over title, description, notes and extracted text - The URL is normalised on save so the same page saved from two places is one bookmark, not two. ### Phase 1 · Save a link and normalise it Build: An endpoint that accepts a URL, title and tags. Normalise the URL: strip tracking parameters, resolve the canonical link where the page declares one, and unify the trivial variations. Saving an existing URL updates rather than duplicates. Trap: Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. Done when: The same article saved with and without tracking parameters produces one bookmark. A URL differing only by a trailing slash or scheme produces one bookmark. A URL whose query string genuinely identifies the content is not collapsed into another page. Do not build yet: No fetching, no extraction, no search. ### Phase 2 · Archiving the readable page Build: Fetch the page with a timeout and a byte cap, extract the readable content, sanitise it, and store both plain text and a self-contained HTML snapshot. Record failures as failures rather than leaving an empty snapshot. Trap: Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. Done when: A saved article's text is readable from the archive with the network disabled. A page requiring JavaScript to render is recorded as a failed extraction with the reason rather than an empty snapshot. Opening a stored snapshot makes no outbound network request, verified in developer tools. Do not build yet: No search and no tagging yet. ### Phase 3 · Tags, notes and fast search Build: Tagging with autocomplete, notes per bookmark, and full text search across titles, descriptions, notes and extracted text. Test at the size an archive reaches after a few years, not the size it is today. Trap: Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. Done when: Search across ten thousand bookmarks with archived text returns in well under a hundred milliseconds. Searching a phrase that appears only in an article body finds it. Tag filtering combines with text search. Rebuilding the index from the stored text produces identical results. Do not build yet: No capture tooling or export yet. ### Phase 4 · Capture that takes one click Build: A bookmarklet that posts the current page with its title and selection. A command line entry point for batch saving. An import for bookmarks exported from a browser or another service. Trap: A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. Done when: The bookmarklet saves the current page from three different sites, including one with an aggressive content security policy, or fails with a visible message on that site. Importing a browser bookmarks export creates bookmarks without duplicates. The command line entry point saves a URL and returns the identifier. Do not build yet: No export yet. ### Phase 5 · Export that makes leaving easy Build: Export as JSON with everything, as the standard bookmarks HTML format, and as a directory of Markdown files with the extracted text. Verify that a JSON export imports back into an empty installation exactly. Trap: An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. Done when: A JSON export imports into an empty installation and reproduces every bookmark, tag, note and snapshot, verified by checksum. The Markdown export is readable in any editor with front matter carrying the URL and tags. The bookmarks HTML export imports into a browser. Do not build yet: Hosting and automation come next. ### Phase 6 · Hosting it, refreshing it and keeping it Build: Put it behind authentication and TLS. Periodically re-check saved URLs and mark the dead ones, which is the moment the archive proves its worth. Back up the database and the snapshot directory, and restore both onto a clean machine. Trap: Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months. Done when: An unauthenticated request cannot read or save. A link-check pass marks a deliberately broken URL as dead and leaves its archived text readable, which is the entire point. A restore onto a clean machine reproduces bookmarks and snapshots, and the date is recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - Archiving pages behind paywalls or logins. It is the obvious gap and working around access controls is not something this build does. - Full asset archiving with images, stylesheets and fonts. It multiplies storage many times over for material that is rarely what you came back for. - The hosted service's longevity. A service that has run for years has proved something your own instance has not, and the maintenance is now yours. - Mobile capture without a browser extension, which is genuinely awkward and the bookmarklet only partly solves. ### README must contain - Which tracking parameters are stripped and why unknown parameters are kept - That snapshots are sanitised at capture and make no outbound requests when opened - That a page requiring JavaScript is recorded as a failed snapshot rather than an empty one - That the Markdown export is the format designed to outlive this application - The link check interval and concurrency, and that dead links keep their archived text
AGENTS.md
# Agent instructions · Pinboard indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with a small HTTP server, Readability plus a sanitiser, SQLite with full text search, plus the extracted text on disk, A bookmarklet and a command line entry point, Text and a single-file HTML snapshot, images optional. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in.
BUILD_PLAN.md
# Build plan · Pinboard Save a page from a bookmarklet or the command line, extract and store its readable text so it survives the site going away, tag and annotate it, search the full text instantly, and export the whole archive as plain files you could read without this application. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Save a link and normalise it A URL becomes a bookmark, and the same page saved twice is one entry. ### Steps 1. Build the save endpoint URL, optional title, optional tags. Return the bookmark so a caller knows whether it created or updated. Files: `src/save.js` 2. Normalise carefully Strip known tracking parameters, lowercase the host, remove a default port and a trailing slash. Do not strip unknown parameters, because some of them are the page. Files: `src/normalise.js` 3. Test the collapsing rules A table of URL pairs with expected same or different verdicts. This is where over-eager normalisation gets caught. Files: `test/normalise.test.js` ### Done when - [ ] The same article with and without tracking parameters is one bookmark - [ ] Scheme and trailing slash variations collapse to one - [ ] A content-identifying query string is preserved - [ ] Re-saving updates rather than duplicating ### Watch out - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. ## Phase 2 · Archiving the readable page The article survives the site disappearing. ### Steps 1. Fetch with limits and respect robots Timeout, byte cap enforced during streaming, and a check of robots.txt. You are a well-behaved client saving pages for yourself. Files: `src/fetch.js` 2. Extract and sanitise Readability for the article, then strip scripts, event handlers and remote references so the snapshot is inert and self-contained. Files: `src/extract.js` ```sh npm install @mozilla/readability jsdom dompurify ``` 3. Record failure honestly A page that needs JavaScript, returns an error or is behind a wall is a failed snapshot with a reason. An empty snapshot pretending to be a success is worse than nothing. Files: `src/extract.js` ### Done when - [ ] A saved article is readable offline - [ ] A JavaScript-dependent page is recorded as a failure with a reason - [ ] An opened snapshot makes no outbound requests - [ ] The byte cap is enforced during download, not after ### Watch out - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. ## Phase 3 · Tags, notes and fast search Finding something you saved two years ago takes a few seconds. ### Steps 1. Generate a realistic archive for testing Ten thousand bookmarks with article-length text. Without this fixture every performance claim is a guess. Files: `scripts/generate-archive.js` 2. Index everything searchable Title, description, notes and extracted text in one full text index, updated on save. Files: `src/search.js` 3. Make rebuild a first-class action The index is derived from the stored text, so rebuilding must be possible and must produce identical results. Files: `src/search.js` ### Done when - [ ] Search over ten thousand bookmarks returns in under a hundred milliseconds - [ ] A phrase only in an article body is findable - [ ] Tag filters combine with text search - [ ] Index rebuild produces identical results ### Watch out - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. ## Phase 4 · Capture that takes one click Saving a page is fast enough that you actually do it. ### Steps 1. Write the bookmarklet Title, URL and any selected text. Keep it small enough to paste, and use a save-only token. Files: `public/bookmarklet.js` 2. Handle sites that block it A strict content security policy will stop a bookmarklet. Detect and tell the user rather than failing silently, and offer the command line as the fallback. Files: `public/bookmarklet.js` 3. Import from browsers and services The standard bookmarks HTML export, plus at least one service's JSON. Deduplicate through the same normalisation as saving. Files: `src/import.js` ### Done when - [ ] The bookmarklet saves from three different sites - [ ] A site blocking it produces a visible message - [ ] A browser export imports without duplicates - [ ] The command line entry point saves and returns an identifier ### Watch out - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. ## Phase 5 · Export that makes leaving easy The archive is portable, and its value does not depend on this application existing. ### Steps 1. Export lossless JSON Everything including snapshot paths, with a schema version. Files: `src/export.js` 2. Export a Markdown directory One file per bookmark with front matter and the extracted text. This is the format that outlives the application. Files: `src/export.js` 3. Verify the round trip Import into an empty database and compare a normalised checksum. Files: `test/roundtrip.test.js` ### Done when - [ ] A JSON export round-trips by checksum - [ ] Markdown files are readable in any editor with URL and tags in front matter - [ ] The bookmarks HTML export imports into a browser - [ ] Snapshots are included or their absence is explicit ### Watch out - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. ## Not in this build - Archiving pages behind paywalls or logins. - Full asset archiving with images and stylesheets. - The proven longevity of a service that has run for years. - Convenient mobile capture without a browser extension. ## After v1, if you want it - A weekly digest of pages saved but never opened - Optional image archiving for a subset of tags where it actually matters - Public sharing of a single tag as a static page
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. SQLite database with bookmarks, tags, notes and the search index. DB_PATH=./data/archive.db # Required. Where extracted text and HTML snapshots are written. Back this up alongside the database. SNAPSHOT_DIR=./data/snapshots # Required. Per-page fetch timeout. A slow page should fail rather than hold up a save. FETCH_TIMEOUT_MS=10000 # Required. Streaming byte cap per page, enforced during download. FETCH_MAX_BYTES=5000000 # Required · secret. Used by the bookmarklet. Scoped to saving only, because a bookmarklet cannot keep a secret well. SAVE_TOKEN=replace-with-a-random-token # Required. How often each URL is re-checked. Link rot works on months, so checking often is rude and pointless. LINKCHECK_INTERVAL_DAYS=90 # Required. Simultaneous checks. Low, and per-host limited, so you do not look like a crawler. LINKCHECK_CONCURRENCY=2
PRODUCT.md
# Pinboard · product brief ## Problem The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For Pinboard, run a fast private bookmark archive with tags, notes, and full-text search. The hard boundary is hosted longevity, simple api, archive service, and zero-maintenance reliability, plus capture polish, sync, and content partnerships. ## Product outcome A private archive where the article survives the site, search is instant at ten thousand entries, snapshots are inert, and the whole thing exports to Markdown files that need none of this software. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - local disk - optional browser bookmarklet - user-supplied RSS feeds ## Explicit non-goals for v1 - Archiving pages behind paywalls or logins. - Full asset archiving with images and stylesheets. - The proven longevity of a service that has run for years. - Convenient mobile capture without a browser extension. - hosted longevity, simple API, archive service, and zero-maintenance reliability - publisher bypasses - cross-device mobile capture - high-quality recommendation graph - licensed full-text feeds ## Success criteria - Search latency on the ten thousand entry fixture is measured and has not regressed - A stored snapshot is confirmed to make no outbound requests - The JSON export round-trips by checksum - Markdown export is confirmed readable in a plain editor - A restore of both database and snapshots has been performed and dated
BRIEF.md
# Build brief · Pinboard The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a private bookmark archive that saves the readable page, not just the link to replace Pinboard. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with a small HTTP server · one person's archive, where the work is fetching and indexing rather than serving traffic - Extraction: Readability plus a sanitiser · the article without the navigation, the cookie banner and the newsletter prompt - Storage: SQLite with full text search, plus the extracted text on disk · search wants an index; longevity wants files you can read without the index - Capture: A bookmarklet and a command line entry point · no browser extension to maintain across browsers and their review processes - Snapshots: Text and a single-file HTML snapshot, images optional · a full page archive with every asset is enormous and mostly worthless a year later ### Data model (create this before Phase 1) - bookmarks(id, url, canonical_url, title, description, added_at, read_at, starred, source) - tags(id, name), bookmark_tags(bookmark_id, tag_id) - snapshots(id, bookmark_id, kind, path, bytes, fetched_at, status) · kind is text, html or pdf - notes(id, bookmark_id, body, created_at) - bookmarks_fts · full text over title, description, notes and extracted text - The URL is normalised on save so the same page saved from two places is one bookmark, not two. ### Phase 1 · Save a link and normalise it Build: An endpoint that accepts a URL, title and tags. Normalise the URL: strip tracking parameters, resolve the canonical link where the page declares one, and unify the trivial variations. Saving an existing URL updates rather than duplicates. Trap: Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. Done when: The same article saved with and without tracking parameters produces one bookmark. A URL differing only by a trailing slash or scheme produces one bookmark. A URL whose query string genuinely identifies the content is not collapsed into another page. Do not build yet: No fetching, no extraction, no search. ### Phase 2 · Archiving the readable page Build: Fetch the page with a timeout and a byte cap, extract the readable content, sanitise it, and store both plain text and a self-contained HTML snapshot. Record failures as failures rather than leaving an empty snapshot. Trap: Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. Done when: A saved article's text is readable from the archive with the network disabled. A page requiring JavaScript to render is recorded as a failed extraction with the reason rather than an empty snapshot. Opening a stored snapshot makes no outbound network request, verified in developer tools. Do not build yet: No search and no tagging yet. ### Phase 3 · Tags, notes and fast search Build: Tagging with autocomplete, notes per bookmark, and full text search across titles, descriptions, notes and extracted text. Test at the size an archive reaches after a few years, not the size it is today. Trap: Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. Done when: Search across ten thousand bookmarks with archived text returns in well under a hundred milliseconds. Searching a phrase that appears only in an article body finds it. Tag filtering combines with text search. Rebuilding the index from the stored text produces identical results. Do not build yet: No capture tooling or export yet. ### Phase 4 · Capture that takes one click Build: A bookmarklet that posts the current page with its title and selection. A command line entry point for batch saving. An import for bookmarks exported from a browser or another service. Trap: A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. Done when: The bookmarklet saves the current page from three different sites, including one with an aggressive content security policy, or fails with a visible message on that site. Importing a browser bookmarks export creates bookmarks without duplicates. The command line entry point saves a URL and returns the identifier. Do not build yet: No export yet. ### Phase 5 · Export that makes leaving easy Build: Export as JSON with everything, as the standard bookmarks HTML format, and as a directory of Markdown files with the extracted text. Verify that a JSON export imports back into an empty installation exactly. Trap: An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. Done when: A JSON export imports into an empty installation and reproduces every bookmark, tag, note and snapshot, verified by checksum. The Markdown export is readable in any editor with front matter carrying the URL and tags. The bookmarks HTML export imports into a browser. Do not build yet: Hosting and automation come next. ### Phase 6 · Hosting it, refreshing it and keeping it Build: Put it behind authentication and TLS. Periodically re-check saved URLs and mark the dead ones, which is the moment the archive proves its worth. Back up the database and the snapshot directory, and restore both onto a clean machine. Trap: Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months. Done when: An unauthenticated request cannot read or save. A link-check pass marks a deliberately broken URL as dead and leaves its archived text readable, which is the entire point. A restore onto a clean machine reproduces bookmarks and snapshots, and the date is recorded. Do not build yet: Nothing after this. ### Out of scope (and why) - Archiving pages behind paywalls or logins. It is the obvious gap and working around access controls is not something this build does. - Full asset archiving with images, stylesheets and fonts. It multiplies storage many times over for material that is rarely what you came back for. - The hosted service's longevity. A service that has run for years has proved something your own instance has not, and the maintenance is now yours. - Mobile capture without a browser extension, which is genuinely awkward and the bookmarklet only partly solves. ### README must contain - Which tracking parameters are stripped and why unknown parameters are kept - That snapshots are sanitised at capture and make no outbound requests when opened - That a page requiring JavaScript is recorded as a failed snapshot rather than an empty one - That the Markdown export is the format designed to outlive this application - The link check interval and concurrency, and that dead links keep their archived text
ARCHITECTURE.md
# Architecture · Pinboard ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with a small HTTP server | one person's archive, where the work is fetching and indexing rather than serving traffic | | Extraction | Readability plus a sanitiser | the article without the navigation, the cookie banner and the newsletter prompt | | Storage | SQLite with full text search, plus the extracted text on disk | search wants an index; longevity wants files you can read without the index | | Capture | A bookmarklet and a command line entry point | no browser extension to maintain across browsers and their review processes | | Snapshots | Text and a single-file HTML snapshot, images optional | a full page archive with every asset is enormous and mostly worthless a year later | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Normalise | Turning a URL into a stable identity. | A pure function with a test table. Over-eager stripping is the failure mode, so any change needs the table updated first. | | Fetch and extract | Getting the article and making it inert. | A headless browser would handle JavaScript-dependent pages at a large cost in speed and complexity. Keep it behind the same interface if you add it. | | Store | Bookmarks, snapshots and the search index. | The index is derived and rebuildable; the snapshots are not. That asymmetry decides what gets backed up. | | Capture | Bookmarklet and command line entry points. | An extension would be more capable and more secure, at the cost of maintaining it per browser. | | Export | Portability. | Never remove the Markdown export. It is the promise that the archive is yours. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DB_PATH` · required · SQLite database with bookmarks, tags, notes and the search index. - `SNAPSHOT_DIR` · required · Where extracted text and HTML snapshots are written. Back this up alongside the database. - `FETCH_TIMEOUT_MS` · required · Per-page fetch timeout. A slow page should fail rather than hold up a save. - `FETCH_MAX_BYTES` · required · Streaming byte cap per page, enforced during download. - `SAVE_TOKEN` · required, secret · Used by the bookmarklet. Scoped to saving only, because a bookmarklet cannot keep a secret well. - `LINKCHECK_INTERVAL_DAYS` · required · How often each URL is re-checked. Link rot works on months, so checking often is rude and pointless. - `LINKCHECK_CONCURRENCY` · required · Simultaneous checks. Low, and per-host limited, so you do not look like a crawler. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Pinboard product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with a small HTTP server, Readability plus a sanitiser, SQLite with full text search, plus the extracted text on disk, A bookmarklet and a command line entry point, Text and a single-file HTML snapshot, images optional. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. - Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months.
MILESTONES.md
# Delivery milestones · Pinboard Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Save a link and normalise it A URL becomes a bookmark, and the same page saved twice is one entry. ### Steps 1. Build the save endpoint URL, optional title, optional tags. Return the bookmark so a caller knows whether it created or updated. Files: `src/save.js` 2. Normalise carefully Strip known tracking parameters, lowercase the host, remove a default port and a trailing slash. Do not strip unknown parameters, because some of them are the page. Files: `src/normalise.js` 3. Test the collapsing rules A table of URL pairs with expected same or different verdicts. This is where over-eager normalisation gets caught. Files: `test/normalise.test.js` ### Done when - [ ] The same article with and without tracking parameters is one bookmark - [ ] Scheme and trailing slash variations collapse to one - [ ] A content-identifying query string is preserved - [ ] Re-saving updates rather than duplicating ### Watch out - Tracking parameters make every share of the same article a different URL, and an archive that treats them as different fills with duplicates within a month. Strip the known tracking parameters, but keep parameters that identify content, because for many sites the query string is the page. ## M2 · Archiving the readable page The article survives the site disappearing. ### Steps 1. Fetch with limits and respect robots Timeout, byte cap enforced during streaming, and a check of robots.txt. You are a well-behaved client saving pages for yourself. Files: `src/fetch.js` 2. Extract and sanitise Readability for the article, then strip scripts, event handlers and remote references so the snapshot is inert and self-contained. Files: `src/extract.js` ```sh npm install @mozilla/readability jsdom dompurify ``` 3. Record failure honestly A page that needs JavaScript, returns an error or is behind a wall is a failed snapshot with a reason. An empty snapshot pretending to be a success is worse than nothing. Files: `src/extract.js` ### Done when - [ ] A saved article is readable offline - [ ] A JavaScript-dependent page is recorded as a failure with a reason - [ ] An opened snapshot makes no outbound requests - [ ] The byte cap is enforced during download, not after ### Watch out - Storing raw fetched HTML and rendering it later means executing someone else's scripts and loading their trackers every time you open your own archive. Sanitise at capture: strip scripts, event handlers and remote references, and store the cleaned version. ## M3 · Tags, notes and fast search Finding something you saved two years ago takes a few seconds. ### Steps 1. Generate a realistic archive for testing Ten thousand bookmarks with article-length text. Without this fixture every performance claim is a guess. Files: `scripts/generate-archive.js` 2. Index everything searchable Title, description, notes and extracted text in one full text index, updated on save. Files: `src/search.js` 3. Make rebuild a first-class action The index is derived from the stored text, so rebuilding must be possible and must produce identical results. Files: `src/search.js` ### Done when - [ ] Search over ten thousand bookmarks returns in under a hundred milliseconds - [ ] A phrase only in an article body is findable - [ ] Tag filters combine with text search - [ ] Index rebuild produces identical results ### Watch out - Ten thousand bookmarks with full article text is a large index, and a naive query with wildcards on both sides cannot use it. Use the full text index properly and measure with a realistic corpus, because the design that works at two hundred entries will not be obvious until it fails. ## M4 · Capture that takes one click Saving a page is fast enough that you actually do it. ### Steps 1. Write the bookmarklet Title, URL and any selected text. Keep it small enough to paste, and use a save-only token. Files: `public/bookmarklet.js` 2. Handle sites that block it A strict content security policy will stop a bookmarklet. Detect and tell the user rather than failing silently, and offer the command line as the fallback. Files: `public/bookmarklet.js` 3. Import from browsers and services The standard bookmarks HTML export, plus at least one service's JSON. Deduplicate through the same normalisation as saving. Files: `src/import.js` ### Done when - [ ] The bookmarklet saves from three different sites - [ ] A site blocking it produces a visible message - [ ] A browser export imports without duplicates - [ ] The command line entry point saves and returns an identifier ### Watch out - A bookmarklet runs on every site you visit, and a naive one exposes your archive token to any page that looks for it. Keep the token in the request rather than in a global, use a token scoped only to saving, and accept that a bookmarklet cannot be as secure as an extension. ## M5 · Export that makes leaving easy The archive is portable, and its value does not depend on this application existing. ### Steps 1. Export lossless JSON Everything including snapshot paths, with a schema version. Files: `src/export.js` 2. Export a Markdown directory One file per bookmark with front matter and the extracted text. This is the format that outlives the application. Files: `src/export.js` 3. Verify the round trip Import into an empty database and compare a normalised checksum. Files: `test/roundtrip.test.js` ### Done when - [ ] A JSON export round-trips by checksum - [ ] Markdown files are readable in any editor with URL and tags in front matter - [ ] The bookmarks HTML export imports into a browser - [ ] Snapshots are included or their absence is explicit ### Watch out - An archive whose contents are only readable through its own application is the situation people are escaping when they leave a hosted service. The Markdown directory is not a convenience feature; it is the difference between an archive and a lock-in. ## M6 · Hosting it, refreshing it and keeping it (production only) It runs somewhere reachable, notices dead links, and you have restored it once. ### Steps 1. Authenticate and serve over TLS A private archive of what you read is sensitive. Put a login in front and get a certificate. Files: `src/auth.js`, `Caddyfile` 2. Check links gently Low concurrency, per-host limits, and a long interval. Mark dead rather than deleting, because the snapshot is now the only copy. Files: `src/linkcheck.js` 3. Back up both halves and restore The database and the snapshot directory. Restore onto a clean machine and record the date. Files: `scripts/backup.sh`, `README.md` ### Done when - [ ] Unauthenticated requests cannot read or save - [ ] A dead link is marked and its archived text is still readable - [ ] Link checking stays within polite concurrency limits - [ ] A restore has been performed on a clean machine and dated ### Watch out - Re-checking every URL on a schedule looks like a crawler to the sites you are checking and will get your address blocked. Spread checks over time, limit concurrency per host, and check each URL rarely: link rot works on a scale of months.
OPERATIONS.md
# Operations · Pinboard ## Backup The database and the snapshot directory together. The database alone leaves you with metadata pointing at nothing, which is exactly the situation archiving exists to prevent. ## Restore Restore both, rebuild the index if needed, and open a bookmark whose original site is dead. That is the test that the archive is real. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Failed extraction rate and dead link count. A jump in extraction failures usually means a change in the extraction library or a shift in how sites render, not a change in your reading. ## Incident checklist If your address gets blocked by a site, reduce link check frequency and concurrency first. If snapshots are lost but the database survives, the archive is metadata only and the lesson is that the two halves must be backed up together. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Search latency on the ten thousand entry fixture is measured and has not regressed - [ ] A stored snapshot is confirmed to make no outbound requests - [ ] The JSON export round-trips by checksum - [ ] Markdown export is confirmed readable in a plain editor - [ ] A restore of both database and snapshots has been performed and dated ## Launch constraint Do not market omitted Pinboard capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. SQLite database with bookmarks, tags, notes and the search index. DB_PATH=./data/archive.db # Required. Where extracted text and HTML snapshots are written. Back this up alongside the database. SNAPSHOT_DIR=./data/snapshots # Required. Per-page fetch timeout. A slow page should fail rather than hold up a save. FETCH_TIMEOUT_MS=10000 # Required. Streaming byte cap per page, enforced during download. FETCH_MAX_BYTES=5000000 # Required · secret. Used by the bookmarklet. Scoped to saving only, because a bookmarklet cannot keep a secret well. SAVE_TOKEN=replace-with-a-random-token # Required. How often each URL is re-checked. Link rot works on months, so checking often is rude and pointless. LINKCHECK_INTERVAL_DAYS=90 # Required. Simultaneous checks. Low, and per-host limited, so you do not look like a crawler. LINKCHECK_CONCURRENCY=2
$ choose a build depth, inspect the files, then open the complete pack in your agent
People still pay for Pinboard because people pay when capture works everywhere, reading state syncs instantly, and the archive remains clean without constant parser repairs. The recurring cost buys feed quirks, parsing, paywalls, canonical URLs, deduplication, images, search, sync, browser extensions, and publisher changes, not just the visible interface.
xhosted longevity, simple API, archive service, and zero-maintenance reliability
xpublisher bypasses
xcross-device mobile capture
xhigh-quality recommendation graph
xlicensed full-text feeds
share on X ↗"I just replaced Pinboard ($1.83/mo) with one prompt"
Don't feel like building it? These folks already made it free.
KarakeepBookmarks, notes, full-text search and automatic tags for people with too many tabs.open source↗
LinkwardenA private bookmark archive that also keeps searchable copies of what you saved.open source↗
linkdingPinboard with a nicer face and no annual invoice.open source↗all 4 free alternatives to Pinboard →· no votes, no pay-to-list · just what's real
Pinboard pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| regular account | — | $1.83/user | Private bookmark archive with tags and notes; imports are capped at 40 MB/file. |
| archiving account | — | $3.25/user | Regular features plus saved page copies and full-text search. |
free tierno free tier and no trial
billingannual only; no month-to-month plan
hidden costsmulti-year discounts may be offered at checkout but the current discount schedule is not published
verified 2026-08-13 · source ↗
Vibecode Pinboard
Yes. A competent AI coding agent (Claude Code, Codex, Cursor) can build a usable personal Pinboard replacement in one session with the prompt on this page. It runs on your own machine or server with no subscription.
How much does Pinboard cost?
Pinboard costs about $1.83/month (Regular account, checked 2026-07-31), which is $21.96 per year. That's what you save by replacing it with one prompt.
What do I lose by replacing Pinboard?
Honestly: hosted longevity, simple API, archive service, and zero-maintenance reliability; publisher bypasses; cross-device mobile capture; high-quality recommendation graph; licensed full-text feeds. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Pinboard?
Yes: Karakeep (Bookmarks, notes, full-text search and automatic tags for people with too many tabs.) Linkwarden (A private bookmark archive that also keeps searchable copies of what you saved.) linkding (Pinboard with a nicer face and no annual invoice.) All 4 curated free alternatives are at vibecodeit.com/pinboard/alternatives. The prompt is for when you want it exactly your way.