Vibecode Feedbin
track this build5 phases, 15 steps, beginner friendly0%YES · one-shottable
Feedbin's solo core is compact: build a private RSS reader workspace that imports user-supplied URLs or files, extracts metadata, supports notes, and searches the local corpus. A competent builder can reach a useful personal version in one sitting, while the paid product mainly wins on data, import reliability.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Feedbin. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Feedbin · indie build A feed reader whose real work is being a well-behaved HTTP client at scale: conditional requests so unchanged feeds cost nothing, backoff that respects what servers ask for, parsing feeds that violate their own specifications, and read state that survives a feed rewriting its own entries. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Ruby or TypeScript with PostgreSQL | the work is scheduling, HTTP and text, not computation | | Fetching | Conditional requests with stored validators, per-feed backoff | a reader that ignores caching headers is a reader that gets blocked | | Parsing | A tolerant parser over RSS, Atom and JSON Feed | real feeds break their own specification constantly | | Identity | A stable entry identity derived carefully, with content hashing | an entry whose identifier changes will appear as unread again forever | | Push | WebSub subscriptions where a feed advertises a hub | a feed that pushes does not need polling at all | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or Ruby 3.3** · free - Why: Runs the fetcher and the application. Either is fine, since the work is HTTP and text. - Get it: Your package manager, or nvm and rbenv respectively. - Verify: The runtime version prints and a HTTP client library installs - [ ] **PostgreSQL 16 or newer** · free locally - Why: Entries, revisions and read state with correct unread counts want a relational database with real indexes. - Get it: Docker or your package manager. - Verify: psql connects and full text search returns a result - [ ] **Twenty real feeds including broken ones** · free - Why: Every parsing decision in phase 1 is driven by how real feeds actually misbehave, and no synthetic fixture reproduces that. - Get it: Collect feeds you actually read, plus deliberately a few from old software with undeclared encodings or invalid dates. Save the raw bytes as fixtures. - Verify: Twenty raw feeds are saved and at least three fail strict validation - [ ] **A stub HTTP server you control** · free - Why: Conditional requests, rate limiting and backoff cannot be tested against real servers without being exactly the nuisance this build is trying not to be. - Get it: Any small server that can return validators, not-modified responses, error statuses and explicit wait instructions on demand. - Verify: It can return a not-modified response and a rate-limited response with a wait instruction - [ ] **A feed that advertises a hub** (optional) · free - Why: Phase 5 needs a real hub, since the verification handshake and signature format are where implementations differ from the specification. - Get it: Find a feed advertising a hub, or run an open source hub locally. - Verify: A subscription is established and a push is received ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Newsletter ingestion by email address. - Article extraction for summary-only feeds. - Mobile and offline clients. - Feed fetching as a distributed system. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - resilient polling of malformed and rate-limited feeds - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Feedbin, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Feedbin The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a feed reader that polls politely, never loses an article, and keeps read state straight to replace Feedbin. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Ruby or TypeScript with PostgreSQL · the work is scheduling, HTTP and text, not computation - Fetching: Conditional requests with stored validators, per-feed backoff · a reader that ignores caching headers is a reader that gets blocked - Parsing: A tolerant parser over RSS, Atom and JSON Feed · real feeds break their own specification constantly - Identity: A stable entry identity derived carefully, with content hashing · an entry whose identifier changes will appear as unread again forever - Push: WebSub subscriptions where a feed advertises a hub · a feed that pushes does not need polling at all ### Data model (create this before Phase 1) - feeds(id, url, site_url, title, etag, last_modified, last_fetched_at, next_fetch_at, failure_count, hub_url, disabled_at) - entries(id, feed_id, guid, url, title, content, published_at, content_hash, first_seen_at, updated_at) - revisions(entry_id, ordinal, content_hash, content, at) · because publishers edit articles after posting - subscriptions(user_id, feed_id, folder, created_at) - reads(user_id, entry_id, read_at, source) · source is manual or auto - fetches(feed_id, at, status, bytes, duration_ms, outcome) · outcome is not_modified, updated, error or rate_limited - The fetch log is what tells you whether you are a good citizen, and it is worth keeping even though it is the largest table. ### Phase 1 · Parsing feeds that do not follow the rules Build: A parser handling RSS, Atom and JSON Feed, tolerating undeclared encodings, invalid dates, missing identifiers and malformed markup. Derive a stable entry identity. Collect twenty real feeds as fixtures, including badly broken ones. Trap: Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. Done when: Twenty real feeds parse, including at least three with known specification violations. An entry keeps its identity when its title is corrected, tested with a modified fixture. An undeclared encoding is detected rather than producing mojibake. A feed with no per-entry identifiers still produces stable identities across two fetches. Do not build yet: No fetching schedule, subscriptions or reading. ### Phase 2 · Fetching without being a nuisance Build: Store the validators a server returns and send them on the next request so an unchanged feed answers with a not-modified status and no body. Identify your client honestly in the user agent. Log every fetch and its outcome. Trap: Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. Done when: An unchanged feed returns a not-modified response and no body on the second fetch, verified against a real server. The user agent identifies the client and a contact route. The fetch log records outcomes. Across twenty feeds over a day, the majority of fetches are not-modified, measured from the log. Do not build yet: No scheduling or backoff yet. ### Phase 3 · Scheduling and backing off Build: A per-feed schedule adapting to observed publishing frequency. Honour a server asking you to wait after a rate-limited response, and back off exponentially with jitter on errors. Disable a feed that has failed persistently and tell the user. Trap: Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. Done when: A rate-limited response with an explicit wait instruction is honoured exactly rather than overridden by your own schedule, tested against a stub server. Errors back off exponentially with jitter. A feed failing persistently is disabled and surfaced to the user. Feeds do not all fetch on the same clock, verified from the fetch log. Do not build yet: No reading interface yet. ### Phase 4 · Reading, and read state that stays correct Build: Subscriptions, folders and a reading interface with unread counts. Store revisions when an entry's content changes rather than overwriting. Read state keyed on entry identity so an edit does not resurrect an article. Trap: Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. Done when: An edited article shows as updated rather than unread, tested by modifying a fixture and refetching. Previous revisions are retained and viewable. Unread counts are correct across ten thousand entries and update without a full recount. Marking a folder read is atomic and does not partially apply. Do not build yet: No push or search yet. ### Phase 5 · Push, for feeds that offer it Build: Detect a hub advertised in a feed, subscribe, verify the subscription callback, and handle incoming pushes. Verify payload signatures. Reduce polling for pushed feeds but do not stop entirely, and renew subscriptions before they lapse. Trap: Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. Done when: A hub subscription is established and verified, tested against a real hub. A pushed update appears without polling. An unsigned or wrongly signed push is rejected, tested with a crafted request. Subscriptions renew before lapsing and a lapsed one falls back to polling rather than going silent. Do not build yet: Search and export come last. ### Phase 6 · Search, import and getting out Build: Full text search across entries with feed and date filters. Import and export subscriptions in the standard interchange format, preserving folders. Retention that trims entry bodies while keeping read state. Trap: Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small. Done when: Search returns results across a year of entries quickly. Subscriptions export and re-import with folders preserved, verified in another reader. Retention trims bodies while identities and read state survive, tested by trimming then refetching a feed that still lists the entry. Deleting an account removes its subscriptions and read state. Do not build yet: Nothing after this. ### Out of scope (and why) - Newsletter ingestion by email address, which is a mail receiving and parsing project in its own right. - Article extraction for feeds that publish only summaries, which is a per-site scraping problem that never stops needing maintenance. - Mobile and offline clients, which is where a reader is most used and is a synchronisation problem on top of this. - Running this at a scale where feed fetching is a distributed system, which is the hosted product's actual engineering. ### README must contain - That entry identity is derived in a documented order and why a content hash is the last resort - That validators are stored and resent so unchanged feeds cost nothing - That an explicit wait instruction from a server outranks your own schedule - That edited articles create revisions rather than resurfacing as unread - That retention trims bodies but never identities or read state ===== AGENTS.md ===== # Agent instructions · Feedbin indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Ruby or TypeScript with PostgreSQL, Conditional requests with stored validators, per-feed backoff, A tolerant parser over RSS, Atom and JSON Feed, A stable entry identity derived carefully, with content hashing, WebSub subscriptions where a feed advertises a hub. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. ===== BUILD_PLAN.md ===== # Build plan · Feedbin A feed reader whose real work is being a well-behaved HTTP client at scale: conditional requests so unchanged feeds cost nothing, backoff that respects what servers ask for, parsing feeds that violate their own specifications, and read state that survives a feed rewriting its own entries. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Parsing feeds that do not follow the rules Real feeds from real sites parse into entries. ### Steps 1. Handle all three formats tolerantly Real feeds violate their specifications. Refusing them is refusing the web. Files: `src/parse.ts` 2. Derive identity in a documented order Feed identifier, then link, then content hash. Record which was used. Files: `src/identity.ts`, `src/identity.test.ts` 3. Fixture twenty real feeds Including broken ones. Synthetic feeds are always better formed than real ones. Files: `fixtures/feeds/` ### Done when - [ ] Twenty real feeds parse including broken ones - [ ] A corrected typo does not change entry identity - [ ] An undeclared encoding is detected - [ ] A feed without identifiers produces stable ones ### Watch out - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. ## Phase 2 · Fetching without being a nuisance An unchanged feed costs one small request and no bandwidth. ### Steps 1. Store and resend validators Both kinds a server may give you, sent back on the next request. Files: `src/fetch.ts` 2. Identify the client honestly A user agent with a name and a way to reach you. Anonymous crawlers get blocked. Files: `src/fetch.ts` 3. Log every fetch outcome The not-modified ratio is how you know whether your client behaves. Files: `src/fetches.ts` ### Done when - [ ] An unchanged feed returns not-modified with no body - [ ] The user agent identifies the client and a contact - [ ] Every fetch is logged with its outcome - [ ] Most fetches across a day are not-modified ### Watch out - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. ## Phase 3 · Scheduling and backing off Feeds are polled at a sensible rate and problems reduce it rather than continuing. ### Steps 1. Honour an explicit wait instruction It outranks your own schedule. Ignoring it is how you get blocked permanently. Files: `src/schedule.ts` 2. Back off exponentially with jitter Jitter is what stops every feed retrying at the same instant. Files: `src/backoff.ts`, `src/backoff.test.ts` 3. Disable persistently failing feeds And tell the user. Silent dead feeds are how a reader quietly stops working. Files: `src/health.ts` ### Done when - [ ] An explicit wait instruction is honoured exactly - [ ] Errors back off exponentially with jitter - [ ] A persistently failing feed is disabled and surfaced - [ ] Fetches are spread rather than synchronised ### Watch out - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. ## Phase 4 · Reading, and read state that stays correct What you have read stays read, even when publishers edit their articles. ### Steps 1. Store revisions, never overwrite The version you read should still exist after the publisher edits it. Files: `src/revisions.ts` 2. Key read state on identity So an edit updates rather than resurrecting. Files: `src/reads.ts` 3. Keep unread counts incremental A full recount at ten thousand entries is felt every time. Files: `src/counts.ts` ### Done when - [ ] An edited article shows updated, not unread - [ ] Previous revisions are retained - [ ] Unread counts are correct and incremental at scale - [ ] Marking a folder read is atomic ### Watch out - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. ## Phase 5 · Push, for feeds that offer it Feeds advertising a hub arrive instantly and stop being polled. ### Steps 1. Subscribe and verify the callback The verification handshake is what proves you asked for it. Files: `src/websub.ts` 2. Verify push signatures An unauthenticated push is content injection into a user's reader. Files: `src/websub.ts`, `src/websub.test.ts` 3. Keep a reduced poll and renew early A lapsed subscription must degrade to polling, not to silence. Files: `src/schedule.ts` ### Done when - [ ] A subscription is established and verified - [ ] A pushed update arrives without polling - [ ] A wrongly signed push is rejected - [ ] A lapsed subscription falls back to polling ### Watch out - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. ## Not in this build - Newsletter ingestion by email address. - Article extraction for summary-only feeds. - Mobile and offline clients. - Feed fetching as a distributed system. ## After v1, if you want it - Filters that hide entries matching a rule, which is what makes a high-volume feed usable - Saved searches as feeds of their own - A shared read state API, once the counts are provably correct ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Feeds, entries and read state. DATABASE_URL=postgres://localhost:5432/reader # Required. Identifies your client and gives a route to reach you. Anonymous clients get blocked. USER_AGENT=MyReader/1.0 (+https://example.com/bot) # Required. The floor between fetches of one feed, regardless of what the schedule computes. MIN_FETCH_INTERVAL_MINUTES=15 # Required. After this a feed is disabled and surfaced. Retrying a dead site forever helps nobody. MAX_FAILURES_BEFORE_DISABLE=20 # Optional. Phase 5 only. Where hubs deliver pushes. Must be reachable from outside. CALLBACK_BASE_URL=https://reader.example.com/websub # Required. Bodies are trimmed after this. Identities and read state are kept indefinitely. ENTRY_BODY_RETENTION_DAYS=365
You are building a lean indie version of Feedbin. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Feedbin · indie build A feed reader whose real work is being a well-behaved HTTP client at scale: conditional requests so unchanged feeds cost nothing, backoff that respects what servers ask for, parsing feeds that violate their own specifications, and read state that survives a feed rewriting its own entries. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Ruby or TypeScript with PostgreSQL | the work is scheduling, HTTP and text, not computation | | Fetching | Conditional requests with stored validators, per-feed backoff | a reader that ignores caching headers is a reader that gets blocked | | Parsing | A tolerant parser over RSS, Atom and JSON Feed | real feeds break their own specification constantly | | Identity | A stable entry identity derived carefully, with content hashing | an entry whose identifier changes will appear as unread again forever | | Push | WebSub subscriptions where a feed advertises a hub | a feed that pushes does not need polling at all | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or Ruby 3.3** · free - Why: Runs the fetcher and the application. Either is fine, since the work is HTTP and text. - Get it: Your package manager, or nvm and rbenv respectively. - Verify: The runtime version prints and a HTTP client library installs - [ ] **PostgreSQL 16 or newer** · free locally - Why: Entries, revisions and read state with correct unread counts want a relational database with real indexes. - Get it: Docker or your package manager. - Verify: psql connects and full text search returns a result - [ ] **Twenty real feeds including broken ones** · free - Why: Every parsing decision in phase 1 is driven by how real feeds actually misbehave, and no synthetic fixture reproduces that. - Get it: Collect feeds you actually read, plus deliberately a few from old software with undeclared encodings or invalid dates. Save the raw bytes as fixtures. - Verify: Twenty raw feeds are saved and at least three fail strict validation - [ ] **A stub HTTP server you control** · free - Why: Conditional requests, rate limiting and backoff cannot be tested against real servers without being exactly the nuisance this build is trying not to be. - Get it: Any small server that can return validators, not-modified responses, error statuses and explicit wait instructions on demand. - Verify: It can return a not-modified response and a rate-limited response with a wait instruction - [ ] **A feed that advertises a hub** (optional) · free - Why: Phase 5 needs a real hub, since the verification handshake and signature format are where implementations differ from the specification. - Get it: Find a feed advertising a hub, or run an open source hub locally. - Verify: A subscription is established and a push is received ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Newsletter ingestion by email address. - Article extraction for summary-only feeds. - Mobile and offline clients. - Feed fetching as a distributed system. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - resilient polling of malformed and rate-limited feeds - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Feedbin, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Feedbin The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a feed reader that polls politely, never loses an article, and keeps read state straight to replace Feedbin. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Ruby or TypeScript with PostgreSQL · the work is scheduling, HTTP and text, not computation - Fetching: Conditional requests with stored validators, per-feed backoff · a reader that ignores caching headers is a reader that gets blocked - Parsing: A tolerant parser over RSS, Atom and JSON Feed · real feeds break their own specification constantly - Identity: A stable entry identity derived carefully, with content hashing · an entry whose identifier changes will appear as unread again forever - Push: WebSub subscriptions where a feed advertises a hub · a feed that pushes does not need polling at all ### Data model (create this before Phase 1) - feeds(id, url, site_url, title, etag, last_modified, last_fetched_at, next_fetch_at, failure_count, hub_url, disabled_at) - entries(id, feed_id, guid, url, title, content, published_at, content_hash, first_seen_at, updated_at) - revisions(entry_id, ordinal, content_hash, content, at) · because publishers edit articles after posting - subscriptions(user_id, feed_id, folder, created_at) - reads(user_id, entry_id, read_at, source) · source is manual or auto - fetches(feed_id, at, status, bytes, duration_ms, outcome) · outcome is not_modified, updated, error or rate_limited - The fetch log is what tells you whether you are a good citizen, and it is worth keeping even though it is the largest table. ### Phase 1 · Parsing feeds that do not follow the rules Build: A parser handling RSS, Atom and JSON Feed, tolerating undeclared encodings, invalid dates, missing identifiers and malformed markup. Derive a stable entry identity. Collect twenty real feeds as fixtures, including badly broken ones. Trap: Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. Done when: Twenty real feeds parse, including at least three with known specification violations. An entry keeps its identity when its title is corrected, tested with a modified fixture. An undeclared encoding is detected rather than producing mojibake. A feed with no per-entry identifiers still produces stable identities across two fetches. Do not build yet: No fetching schedule, subscriptions or reading. ### Phase 2 · Fetching without being a nuisance Build: Store the validators a server returns and send them on the next request so an unchanged feed answers with a not-modified status and no body. Identify your client honestly in the user agent. Log every fetch and its outcome. Trap: Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. Done when: An unchanged feed returns a not-modified response and no body on the second fetch, verified against a real server. The user agent identifies the client and a contact route. The fetch log records outcomes. Across twenty feeds over a day, the majority of fetches are not-modified, measured from the log. Do not build yet: No scheduling or backoff yet. ### Phase 3 · Scheduling and backing off Build: A per-feed schedule adapting to observed publishing frequency. Honour a server asking you to wait after a rate-limited response, and back off exponentially with jitter on errors. Disable a feed that has failed persistently and tell the user. Trap: Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. Done when: A rate-limited response with an explicit wait instruction is honoured exactly rather than overridden by your own schedule, tested against a stub server. Errors back off exponentially with jitter. A feed failing persistently is disabled and surfaced to the user. Feeds do not all fetch on the same clock, verified from the fetch log. Do not build yet: No reading interface yet. ### Phase 4 · Reading, and read state that stays correct Build: Subscriptions, folders and a reading interface with unread counts. Store revisions when an entry's content changes rather than overwriting. Read state keyed on entry identity so an edit does not resurrect an article. Trap: Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. Done when: An edited article shows as updated rather than unread, tested by modifying a fixture and refetching. Previous revisions are retained and viewable. Unread counts are correct across ten thousand entries and update without a full recount. Marking a folder read is atomic and does not partially apply. Do not build yet: No push or search yet. ### Phase 5 · Push, for feeds that offer it Build: Detect a hub advertised in a feed, subscribe, verify the subscription callback, and handle incoming pushes. Verify payload signatures. Reduce polling for pushed feeds but do not stop entirely, and renew subscriptions before they lapse. Trap: Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. Done when: A hub subscription is established and verified, tested against a real hub. A pushed update appears without polling. An unsigned or wrongly signed push is rejected, tested with a crafted request. Subscriptions renew before lapsing and a lapsed one falls back to polling rather than going silent. Do not build yet: Search and export come last. ### Phase 6 · Search, import and getting out Build: Full text search across entries with feed and date filters. Import and export subscriptions in the standard interchange format, preserving folders. Retention that trims entry bodies while keeping read state. Trap: Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small. Done when: Search returns results across a year of entries quickly. Subscriptions export and re-import with folders preserved, verified in another reader. Retention trims bodies while identities and read state survive, tested by trimming then refetching a feed that still lists the entry. Deleting an account removes its subscriptions and read state. Do not build yet: Nothing after this. ### Out of scope (and why) - Newsletter ingestion by email address, which is a mail receiving and parsing project in its own right. - Article extraction for feeds that publish only summaries, which is a per-site scraping problem that never stops needing maintenance. - Mobile and offline clients, which is where a reader is most used and is a synchronisation problem on top of this. - Running this at a scale where feed fetching is a distributed system, which is the hosted product's actual engineering. ### README must contain - That entry identity is derived in a documented order and why a content hash is the last resort - That validators are stored and resent so unchanged feeds cost nothing - That an explicit wait instruction from a server outranks your own schedule - That edited articles create revisions rather than resurfacing as unread - That retention trims bodies but never identities or read state ===== AGENTS.md ===== # Agent instructions · Feedbin indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Ruby or TypeScript with PostgreSQL, Conditional requests with stored validators, per-feed backoff, A tolerant parser over RSS, Atom and JSON Feed, A stable entry identity derived carefully, with content hashing, WebSub subscriptions where a feed advertises a hub. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. ===== BUILD_PLAN.md ===== # Build plan · Feedbin A feed reader whose real work is being a well-behaved HTTP client at scale: conditional requests so unchanged feeds cost nothing, backoff that respects what servers ask for, parsing feeds that violate their own specifications, and read state that survives a feed rewriting its own entries. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Parsing feeds that do not follow the rules Real feeds from real sites parse into entries. ### Steps 1. Handle all three formats tolerantly Real feeds violate their specifications. Refusing them is refusing the web. Files: `src/parse.ts` 2. Derive identity in a documented order Feed identifier, then link, then content hash. Record which was used. Files: `src/identity.ts`, `src/identity.test.ts` 3. Fixture twenty real feeds Including broken ones. Synthetic feeds are always better formed than real ones. Files: `fixtures/feeds/` ### Done when - [ ] Twenty real feeds parse including broken ones - [ ] A corrected typo does not change entry identity - [ ] An undeclared encoding is detected - [ ] A feed without identifiers produces stable ones ### Watch out - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. ## Phase 2 · Fetching without being a nuisance An unchanged feed costs one small request and no bandwidth. ### Steps 1. Store and resend validators Both kinds a server may give you, sent back on the next request. Files: `src/fetch.ts` 2. Identify the client honestly A user agent with a name and a way to reach you. Anonymous crawlers get blocked. Files: `src/fetch.ts` 3. Log every fetch outcome The not-modified ratio is how you know whether your client behaves. Files: `src/fetches.ts` ### Done when - [ ] An unchanged feed returns not-modified with no body - [ ] The user agent identifies the client and a contact - [ ] Every fetch is logged with its outcome - [ ] Most fetches across a day are not-modified ### Watch out - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. ## Phase 3 · Scheduling and backing off Feeds are polled at a sensible rate and problems reduce it rather than continuing. ### Steps 1. Honour an explicit wait instruction It outranks your own schedule. Ignoring it is how you get blocked permanently. Files: `src/schedule.ts` 2. Back off exponentially with jitter Jitter is what stops every feed retrying at the same instant. Files: `src/backoff.ts`, `src/backoff.test.ts` 3. Disable persistently failing feeds And tell the user. Silent dead feeds are how a reader quietly stops working. Files: `src/health.ts` ### Done when - [ ] An explicit wait instruction is honoured exactly - [ ] Errors back off exponentially with jitter - [ ] A persistently failing feed is disabled and surfaced - [ ] Fetches are spread rather than synchronised ### Watch out - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. ## Phase 4 · Reading, and read state that stays correct What you have read stays read, even when publishers edit their articles. ### Steps 1. Store revisions, never overwrite The version you read should still exist after the publisher edits it. Files: `src/revisions.ts` 2. Key read state on identity So an edit updates rather than resurrecting. Files: `src/reads.ts` 3. Keep unread counts incremental A full recount at ten thousand entries is felt every time. Files: `src/counts.ts` ### Done when - [ ] An edited article shows updated, not unread - [ ] Previous revisions are retained - [ ] Unread counts are correct and incremental at scale - [ ] Marking a folder read is atomic ### Watch out - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. ## Phase 5 · Push, for feeds that offer it Feeds advertising a hub arrive instantly and stop being polled. ### Steps 1. Subscribe and verify the callback The verification handshake is what proves you asked for it. Files: `src/websub.ts` 2. Verify push signatures An unauthenticated push is content injection into a user's reader. Files: `src/websub.ts`, `src/websub.test.ts` 3. Keep a reduced poll and renew early A lapsed subscription must degrade to polling, not to silence. Files: `src/schedule.ts` ### Done when - [ ] A subscription is established and verified - [ ] A pushed update arrives without polling - [ ] A wrongly signed push is rejected - [ ] A lapsed subscription falls back to polling ### Watch out - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. ## Not in this build - Newsletter ingestion by email address. - Article extraction for summary-only feeds. - Mobile and offline clients. - Feed fetching as a distributed system. ## After v1, if you want it - Filters that hide entries matching a rule, which is what makes a high-volume feed usable - Saved searches as feeds of their own - A shared read state API, once the counts are provably correct ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Feeds, entries and read state. DATABASE_URL=postgres://localhost:5432/reader # Required. Identifies your client and gives a route to reach you. Anonymous clients get blocked. USER_AGENT=MyReader/1.0 (+https://example.com/bot) # Required. The floor between fetches of one feed, regardless of what the schedule computes. MIN_FETCH_INTERVAL_MINUTES=15 # Required. After this a feed is disabled and surfaced. Retrying a dead site forever helps nobody. MAX_FAILURES_BEFORE_DISABLE=20 # Optional. Phase 5 only. Where hubs deliver pushes. Must be reachable from outside. CALLBACK_BASE_URL=https://reader.example.com/websub # Required. Bodies are trimmed after this. Identities and read state are kept indefinitely. ENTRY_BODY_RETENTION_DAYS=365
You are building a production product version of Feedbin. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Feedbin · product brief ## Problem Feedbin's solo core is compact: build a private RSS reader workspace that imports user-supplied URLs or files, extracts metadata, supports notes, and searches the local corpus. A competent builder can reach a useful personal version in one sitting, while the paid product mainly wins on data, import reliability. ## Product outcome A reader that costs the sites it reads almost nothing, parses feeds that break their own rules, and never shows you an article twice because someone fixed a typo. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - browser automation for user-authorized imports - optional OpenAI API key stored in .env ## Explicit non-goals for v1 - Newsletter ingestion by email address. - Article extraction for summary-only feeds. - Mobile and offline clients. - Feed fetching as a distributed system. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - resilient polling of malformed and rate-limited feeds - licensed scholarly metadata ## Success criteria - Twenty real feeds including broken ones are confirmed to parse - An unchanged feed is confirmed to return not-modified with no body - An explicit wait instruction is confirmed honoured exactly - An edited article is confirmed to show updated rather than unread - Trimming is confirmed to keep identities and read state ===== BRIEF.md ===== # Build brief · Feedbin The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a feed reader that polls politely, never loses an article, and keeps read state straight to replace Feedbin. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Ruby or TypeScript with PostgreSQL · the work is scheduling, HTTP and text, not computation - Fetching: Conditional requests with stored validators, per-feed backoff · a reader that ignores caching headers is a reader that gets blocked - Parsing: A tolerant parser over RSS, Atom and JSON Feed · real feeds break their own specification constantly - Identity: A stable entry identity derived carefully, with content hashing · an entry whose identifier changes will appear as unread again forever - Push: WebSub subscriptions where a feed advertises a hub · a feed that pushes does not need polling at all ### Data model (create this before Phase 1) - feeds(id, url, site_url, title, etag, last_modified, last_fetched_at, next_fetch_at, failure_count, hub_url, disabled_at) - entries(id, feed_id, guid, url, title, content, published_at, content_hash, first_seen_at, updated_at) - revisions(entry_id, ordinal, content_hash, content, at) · because publishers edit articles after posting - subscriptions(user_id, feed_id, folder, created_at) - reads(user_id, entry_id, read_at, source) · source is manual or auto - fetches(feed_id, at, status, bytes, duration_ms, outcome) · outcome is not_modified, updated, error or rate_limited - The fetch log is what tells you whether you are a good citizen, and it is worth keeping even though it is the largest table. ### Phase 1 · Parsing feeds that do not follow the rules Build: A parser handling RSS, Atom and JSON Feed, tolerating undeclared encodings, invalid dates, missing identifiers and malformed markup. Derive a stable entry identity. Collect twenty real feeds as fixtures, including badly broken ones. Trap: Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. Done when: Twenty real feeds parse, including at least three with known specification violations. An entry keeps its identity when its title is corrected, tested with a modified fixture. An undeclared encoding is detected rather than producing mojibake. A feed with no per-entry identifiers still produces stable identities across two fetches. Do not build yet: No fetching schedule, subscriptions or reading. ### Phase 2 · Fetching without being a nuisance Build: Store the validators a server returns and send them on the next request so an unchanged feed answers with a not-modified status and no body. Identify your client honestly in the user agent. Log every fetch and its outcome. Trap: Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. Done when: An unchanged feed returns a not-modified response and no body on the second fetch, verified against a real server. The user agent identifies the client and a contact route. The fetch log records outcomes. Across twenty feeds over a day, the majority of fetches are not-modified, measured from the log. Do not build yet: No scheduling or backoff yet. ### Phase 3 · Scheduling and backing off Build: A per-feed schedule adapting to observed publishing frequency. Honour a server asking you to wait after a rate-limited response, and back off exponentially with jitter on errors. Disable a feed that has failed persistently and tell the user. Trap: Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. Done when: A rate-limited response with an explicit wait instruction is honoured exactly rather than overridden by your own schedule, tested against a stub server. Errors back off exponentially with jitter. A feed failing persistently is disabled and surfaced to the user. Feeds do not all fetch on the same clock, verified from the fetch log. Do not build yet: No reading interface yet. ### Phase 4 · Reading, and read state that stays correct Build: Subscriptions, folders and a reading interface with unread counts. Store revisions when an entry's content changes rather than overwriting. Read state keyed on entry identity so an edit does not resurrect an article. Trap: Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. Done when: An edited article shows as updated rather than unread, tested by modifying a fixture and refetching. Previous revisions are retained and viewable. Unread counts are correct across ten thousand entries and update without a full recount. Marking a folder read is atomic and does not partially apply. Do not build yet: No push or search yet. ### Phase 5 · Push, for feeds that offer it Build: Detect a hub advertised in a feed, subscribe, verify the subscription callback, and handle incoming pushes. Verify payload signatures. Reduce polling for pushed feeds but do not stop entirely, and renew subscriptions before they lapse. Trap: Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. Done when: A hub subscription is established and verified, tested against a real hub. A pushed update appears without polling. An unsigned or wrongly signed push is rejected, tested with a crafted request. Subscriptions renew before lapsing and a lapsed one falls back to polling rather than going silent. Do not build yet: Search and export come last. ### Phase 6 · Search, import and getting out Build: Full text search across entries with feed and date filters. Import and export subscriptions in the standard interchange format, preserving folders. Retention that trims entry bodies while keeping read state. Trap: Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small. Done when: Search returns results across a year of entries quickly. Subscriptions export and re-import with folders preserved, verified in another reader. Retention trims bodies while identities and read state survive, tested by trimming then refetching a feed that still lists the entry. Deleting an account removes its subscriptions and read state. Do not build yet: Nothing after this. ### Out of scope (and why) - Newsletter ingestion by email address, which is a mail receiving and parsing project in its own right. - Article extraction for feeds that publish only summaries, which is a per-site scraping problem that never stops needing maintenance. - Mobile and offline clients, which is where a reader is most used and is a synchronisation problem on top of this. - Running this at a scale where feed fetching is a distributed system, which is the hosted product's actual engineering. ### README must contain - That entry identity is derived in a documented order and why a content hash is the last resort - That validators are stored and resent so unchanged feeds cost nothing - That an explicit wait instruction from a server outranks your own schedule - That edited articles create revisions rather than resurfacing as unread - That retention trims bodies but never identities or read state ===== ARCHITECTURE.md ===== # Architecture · Feedbin ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Ruby or TypeScript with PostgreSQL | the work is scheduling, HTTP and text, not computation | | Fetching | Conditional requests with stored validators, per-feed backoff | a reader that ignores caching headers is a reader that gets blocked | | Parsing | A tolerant parser over RSS, Atom and JSON Feed | real feeds break their own specification constantly | | Identity | A stable entry identity derived carefully, with content hashing | an entry whose identifier changes will appear as unread again forever | | Push | WebSub subscriptions where a feed advertises a hub | a feed that pushes does not need polling at all | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Parser | Tolerant handling of three formats. | Any parser that survives real feeds. Strict validation is the wrong posture here. | | Identity | Deriving a stable entry identity. | Any order, provided it is documented and content hashing is the last resort. | | Fetcher | Conditional requests and honest identification. | Any client, provided validators are stored and resent. | | Scheduler | Adaptive intervals, backoff and disabling. | Any scheduler that honours an explicit wait instruction over its own plan. | | Retention | Trimming bodies without losing read state. | Any policy, provided identities survive trimming. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Feeds, entries and read state. - `USER_AGENT` · required · Identifies your client and gives a route to reach you. Anonymous clients get blocked. - `MIN_FETCH_INTERVAL_MINUTES` · required · The floor between fetches of one feed, regardless of what the schedule computes. - `MAX_FAILURES_BEFORE_DISABLE` · required · After this a feed is disabled and surfaced. Retrying a dead site forever helps nobody. - `CALLBACK_BASE_URL` · optional · Phase 5 only. Where hubs deliver pushes. Must be reachable from outside. - `ENTRY_BODY_RETENTION_DAYS` · required · Bodies are trimmed after this. Identities and read state are kept indefinitely. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Feedbin product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Ruby or TypeScript with PostgreSQL, Conditional requests with stored validators, per-feed backoff, A tolerant parser over RSS, Atom and JSON Feed, A stable entry identity derived carefully, with content hashing, WebSub subscriptions where a feed advertises a hub. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. - Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small. ===== MILESTONES.md ===== # Delivery milestones · Feedbin Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Parsing feeds that do not follow the rules Real feeds from real sites parse into entries. ### Steps 1. Handle all three formats tolerantly Real feeds violate their specifications. Refusing them is refusing the web. Files: `src/parse.ts` 2. Derive identity in a documented order Feed identifier, then link, then content hash. Record which was used. Files: `src/identity.ts`, `src/identity.test.ts` 3. Fixture twenty real feeds Including broken ones. Synthetic feeds are always better formed than real ones. Files: `fixtures/feeds/` ### Done when - [ ] Twenty real feeds parse including broken ones - [ ] A corrected typo does not change entry identity - [ ] An undeclared encoding is detected - [ ] A feed without identifiers produces stable ones ### Watch out - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. ## M2 · Fetching without being a nuisance An unchanged feed costs one small request and no bandwidth. ### Steps 1. Store and resend validators Both kinds a server may give you, sent back on the next request. Files: `src/fetch.ts` 2. Identify the client honestly A user agent with a name and a way to reach you. Anonymous crawlers get blocked. Files: `src/fetch.ts` 3. Log every fetch outcome The not-modified ratio is how you know whether your client behaves. Files: `src/fetches.ts` ### Done when - [ ] An unchanged feed returns not-modified with no body - [ ] The user agent identifies the client and a contact - [ ] Every fetch is logged with its outcome - [ ] Most fetches across a day are not-modified ### Watch out - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. ## M3 · Scheduling and backing off Feeds are polled at a sensible rate and problems reduce it rather than continuing. ### Steps 1. Honour an explicit wait instruction It outranks your own schedule. Ignoring it is how you get blocked permanently. Files: `src/schedule.ts` 2. Back off exponentially with jitter Jitter is what stops every feed retrying at the same instant. Files: `src/backoff.ts`, `src/backoff.test.ts` 3. Disable persistently failing feeds And tell the user. Silent dead feeds are how a reader quietly stops working. Files: `src/health.ts` ### Done when - [ ] An explicit wait instruction is honoured exactly - [ ] Errors back off exponentially with jitter - [ ] A persistently failing feed is disabled and surfaced - [ ] Fetches are spread rather than synchronised ### Watch out - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. ## M4 · Reading, and read state that stays correct What you have read stays read, even when publishers edit their articles. ### Steps 1. Store revisions, never overwrite The version you read should still exist after the publisher edits it. Files: `src/revisions.ts` 2. Key read state on identity So an edit updates rather than resurrecting. Files: `src/reads.ts` 3. Keep unread counts incremental A full recount at ten thousand entries is felt every time. Files: `src/counts.ts` ### Done when - [ ] An edited article shows updated, not unread - [ ] Previous revisions are retained - [ ] Unread counts are correct and incremental at scale - [ ] Marking a folder read is atomic ### Watch out - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. ## M5 · Push, for feeds that offer it Feeds advertising a hub arrive instantly and stop being polled. ### Steps 1. Subscribe and verify the callback The verification handshake is what proves you asked for it. Files: `src/websub.ts` 2. Verify push signatures An unauthenticated push is content injection into a user's reader. Files: `src/websub.ts`, `src/websub.test.ts` 3. Keep a reduced poll and renew early A lapsed subscription must degrade to polling, not to silence. Files: `src/schedule.ts` ### Done when - [ ] A subscription is established and verified - [ ] A pushed update arrives without polling - [ ] A wrongly signed push is rejected - [ ] A lapsed subscription falls back to polling ### Watch out - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. ## M6 · Search, import and getting out (production only) You can find an article from a year ago and take your subscriptions elsewhere. ### Steps 1. Index entry text with filters Feed and date, combined with text. A year of entries should search instantly. Files: `src/search.ts` 2. Support the standard interchange format Both directions, with folders. A reader you cannot leave is one to avoid. Files: `src/opml.ts` 3. Trim bodies, keep identities Read state is small and its loss resurrects old articles. Files: `src/retention.ts` ### Done when - [ ] Search across a year returns quickly - [ ] Export re-imports elsewhere with folders intact - [ ] Trimming keeps identities and read state - [ ] Account deletion removes subscriptions and reads ### Watch out - Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small. ===== OPERATIONS.md ===== # Operations · Feedbin ## Backup Subscriptions and read state are the irreplaceable data and are small. Entry bodies are large and re-fetchable for recent items only, so back up the former carefully and the latter as convenient. ## Restore Restore and confirm read state and unread counts are consistent, since a restore that loses recent reads presents a user with hundreds of already-read articles. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring The ratio of not-modified to full responses is the health metric for your behaviour as a client. Disabled feed count and oldest successful fetch tell you whether the reader is quietly rotting. ## Incident checklist Being blocked by a site is the incident that matters, and the cause is nearly always ignoring caching headers or an explicit wait instruction. The fetch log is what lets you demonstrate what your client actually did. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Twenty real feeds including broken ones are confirmed to parse - [ ] An unchanged feed is confirmed to return not-modified with no body - [ ] An explicit wait instruction is confirmed honoured exactly - [ ] An edited article is confirmed to show updated rather than unread - [ ] Trimming is confirmed to keep identities and read state ## Launch constraint Do not market omitted Feedbin capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Feeds, entries and read state. DATABASE_URL=postgres://localhost:5432/reader # Required. Identifies your client and gives a route to reach you. Anonymous clients get blocked. USER_AGENT=MyReader/1.0 (+https://example.com/bot) # Required. The floor between fetches of one feed, regardless of what the schedule computes. MIN_FETCH_INTERVAL_MINUTES=15 # Required. After this a feed is disabled and surfaced. Retrying a dead site forever helps nobody. MAX_FAILURES_BEFORE_DISABLE=20 # Optional. Phase 5 only. Where hubs deliver pushes. Must be reachable from outside. CALLBACK_BASE_URL=https://reader.example.com/websub # Required. Bodies are trimmed after this. Identities and read state are kept indefinitely. ENTRY_BODY_RETENTION_DAYS=365
README.md
# Feedbin · indie build A feed reader whose real work is being a well-behaved HTTP client at scale: conditional requests so unchanged feeds cost nothing, backoff that respects what servers ask for, parsing feeds that violate their own specifications, and read state that survives a feed rewriting its own entries. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Ruby or TypeScript with PostgreSQL | the work is scheduling, HTTP and text, not computation | | Fetching | Conditional requests with stored validators, per-feed backoff | a reader that ignores caching headers is a reader that gets blocked | | Parsing | A tolerant parser over RSS, Atom and JSON Feed | real feeds break their own specification constantly | | Identity | A stable entry identity derived carefully, with content hashing | an entry whose identifier changes will appear as unread again forever | | Push | WebSub subscriptions where a feed advertises a hub | a feed that pushes does not need polling at all | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or Ruby 3.3** · free - Why: Runs the fetcher and the application. Either is fine, since the work is HTTP and text. - Get it: Your package manager, or nvm and rbenv respectively. - Verify: The runtime version prints and a HTTP client library installs - [ ] **PostgreSQL 16 or newer** · free locally - Why: Entries, revisions and read state with correct unread counts want a relational database with real indexes. - Get it: Docker or your package manager. - Verify: psql connects and full text search returns a result - [ ] **Twenty real feeds including broken ones** · free - Why: Every parsing decision in phase 1 is driven by how real feeds actually misbehave, and no synthetic fixture reproduces that. - Get it: Collect feeds you actually read, plus deliberately a few from old software with undeclared encodings or invalid dates. Save the raw bytes as fixtures. - Verify: Twenty raw feeds are saved and at least three fail strict validation - [ ] **A stub HTTP server you control** · free - Why: Conditional requests, rate limiting and backoff cannot be tested against real servers without being exactly the nuisance this build is trying not to be. - Get it: Any small server that can return validators, not-modified responses, error statuses and explicit wait instructions on demand. - Verify: It can return a not-modified response and a rate-limited response with a wait instruction - [ ] **A feed that advertises a hub** (optional) · free - Why: Phase 5 needs a real hub, since the verification handshake and signature format are where implementations differ from the specification. - Get it: Find a feed advertising a hub, or run an open source hub locally. - Verify: A subscription is established and a push is received ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Newsletter ingestion by email address. - Article extraction for summary-only feeds. - Mobile and offline clients. - Feed fetching as a distributed system. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - resilient polling of malformed and rate-limited feeds - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Feedbin, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Feedbin The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a feed reader that polls politely, never loses an article, and keeps read state straight to replace Feedbin. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Ruby or TypeScript with PostgreSQL · the work is scheduling, HTTP and text, not computation - Fetching: Conditional requests with stored validators, per-feed backoff · a reader that ignores caching headers is a reader that gets blocked - Parsing: A tolerant parser over RSS, Atom and JSON Feed · real feeds break their own specification constantly - Identity: A stable entry identity derived carefully, with content hashing · an entry whose identifier changes will appear as unread again forever - Push: WebSub subscriptions where a feed advertises a hub · a feed that pushes does not need polling at all ### Data model (create this before Phase 1) - feeds(id, url, site_url, title, etag, last_modified, last_fetched_at, next_fetch_at, failure_count, hub_url, disabled_at) - entries(id, feed_id, guid, url, title, content, published_at, content_hash, first_seen_at, updated_at) - revisions(entry_id, ordinal, content_hash, content, at) · because publishers edit articles after posting - subscriptions(user_id, feed_id, folder, created_at) - reads(user_id, entry_id, read_at, source) · source is manual or auto - fetches(feed_id, at, status, bytes, duration_ms, outcome) · outcome is not_modified, updated, error or rate_limited - The fetch log is what tells you whether you are a good citizen, and it is worth keeping even though it is the largest table. ### Phase 1 · Parsing feeds that do not follow the rules Build: A parser handling RSS, Atom and JSON Feed, tolerating undeclared encodings, invalid dates, missing identifiers and malformed markup. Derive a stable entry identity. Collect twenty real feeds as fixtures, including badly broken ones. Trap: Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. Done when: Twenty real feeds parse, including at least three with known specification violations. An entry keeps its identity when its title is corrected, tested with a modified fixture. An undeclared encoding is detected rather than producing mojibake. A feed with no per-entry identifiers still produces stable identities across two fetches. Do not build yet: No fetching schedule, subscriptions or reading. ### Phase 2 · Fetching without being a nuisance Build: Store the validators a server returns and send them on the next request so an unchanged feed answers with a not-modified status and no body. Identify your client honestly in the user agent. Log every fetch and its outcome. Trap: Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. Done when: An unchanged feed returns a not-modified response and no body on the second fetch, verified against a real server. The user agent identifies the client and a contact route. The fetch log records outcomes. Across twenty feeds over a day, the majority of fetches are not-modified, measured from the log. Do not build yet: No scheduling or backoff yet. ### Phase 3 · Scheduling and backing off Build: A per-feed schedule adapting to observed publishing frequency. Honour a server asking you to wait after a rate-limited response, and back off exponentially with jitter on errors. Disable a feed that has failed persistently and tell the user. Trap: Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. Done when: A rate-limited response with an explicit wait instruction is honoured exactly rather than overridden by your own schedule, tested against a stub server. Errors back off exponentially with jitter. A feed failing persistently is disabled and surfaced to the user. Feeds do not all fetch on the same clock, verified from the fetch log. Do not build yet: No reading interface yet. ### Phase 4 · Reading, and read state that stays correct Build: Subscriptions, folders and a reading interface with unread counts. Store revisions when an entry's content changes rather than overwriting. Read state keyed on entry identity so an edit does not resurrect an article. Trap: Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. Done when: An edited article shows as updated rather than unread, tested by modifying a fixture and refetching. Previous revisions are retained and viewable. Unread counts are correct across ten thousand entries and update without a full recount. Marking a folder read is atomic and does not partially apply. Do not build yet: No push or search yet. ### Phase 5 · Push, for feeds that offer it Build: Detect a hub advertised in a feed, subscribe, verify the subscription callback, and handle incoming pushes. Verify payload signatures. Reduce polling for pushed feeds but do not stop entirely, and renew subscriptions before they lapse. Trap: Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. Done when: A hub subscription is established and verified, tested against a real hub. A pushed update appears without polling. An unsigned or wrongly signed push is rejected, tested with a crafted request. Subscriptions renew before lapsing and a lapsed one falls back to polling rather than going silent. Do not build yet: Search and export come last. ### Phase 6 · Search, import and getting out Build: Full text search across entries with feed and date filters. Import and export subscriptions in the standard interchange format, preserving folders. Retention that trims entry bodies while keeping read state. Trap: Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small. Done when: Search returns results across a year of entries quickly. Subscriptions export and re-import with folders preserved, verified in another reader. Retention trims bodies while identities and read state survive, tested by trimming then refetching a feed that still lists the entry. Deleting an account removes its subscriptions and read state. Do not build yet: Nothing after this. ### Out of scope (and why) - Newsletter ingestion by email address, which is a mail receiving and parsing project in its own right. - Article extraction for feeds that publish only summaries, which is a per-site scraping problem that never stops needing maintenance. - Mobile and offline clients, which is where a reader is most used and is a synchronisation problem on top of this. - Running this at a scale where feed fetching is a distributed system, which is the hosted product's actual engineering. ### README must contain - That entry identity is derived in a documented order and why a content hash is the last resort - That validators are stored and resent so unchanged feeds cost nothing - That an explicit wait instruction from a server outranks your own schedule - That edited articles create revisions rather than resurfacing as unread - That retention trims bodies but never identities or read state
AGENTS.md
# Agent instructions · Feedbin indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Ruby or TypeScript with PostgreSQL, Conditional requests with stored validators, per-feed backoff, A tolerant parser over RSS, Atom and JSON Feed, A stable entry identity derived carefully, with content hashing, WebSub subscriptions where a feed advertises a hub. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it.
BUILD_PLAN.md
# Build plan · Feedbin A feed reader whose real work is being a well-behaved HTTP client at scale: conditional requests so unchanged feeds cost nothing, backoff that respects what servers ask for, parsing feeds that violate their own specifications, and read state that survives a feed rewriting its own entries. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Parsing feeds that do not follow the rules Real feeds from real sites parse into entries. ### Steps 1. Handle all three formats tolerantly Real feeds violate their specifications. Refusing them is refusing the web. Files: `src/parse.ts` 2. Derive identity in a documented order Feed identifier, then link, then content hash. Record which was used. Files: `src/identity.ts`, `src/identity.test.ts` 3. Fixture twenty real feeds Including broken ones. Synthetic feeds are always better formed than real ones. Files: `fixtures/feeds/` ### Done when - [ ] Twenty real feeds parse including broken ones - [ ] A corrected typo does not change entry identity - [ ] An undeclared encoding is detected - [ ] A feed without identifiers produces stable ones ### Watch out - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. ## Phase 2 · Fetching without being a nuisance An unchanged feed costs one small request and no bandwidth. ### Steps 1. Store and resend validators Both kinds a server may give you, sent back on the next request. Files: `src/fetch.ts` 2. Identify the client honestly A user agent with a name and a way to reach you. Anonymous crawlers get blocked. Files: `src/fetch.ts` 3. Log every fetch outcome The not-modified ratio is how you know whether your client behaves. Files: `src/fetches.ts` ### Done when - [ ] An unchanged feed returns not-modified with no body - [ ] The user agent identifies the client and a contact - [ ] Every fetch is logged with its outcome - [ ] Most fetches across a day are not-modified ### Watch out - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. ## Phase 3 · Scheduling and backing off Feeds are polled at a sensible rate and problems reduce it rather than continuing. ### Steps 1. Honour an explicit wait instruction It outranks your own schedule. Ignoring it is how you get blocked permanently. Files: `src/schedule.ts` 2. Back off exponentially with jitter Jitter is what stops every feed retrying at the same instant. Files: `src/backoff.ts`, `src/backoff.test.ts` 3. Disable persistently failing feeds And tell the user. Silent dead feeds are how a reader quietly stops working. Files: `src/health.ts` ### Done when - [ ] An explicit wait instruction is honoured exactly - [ ] Errors back off exponentially with jitter - [ ] A persistently failing feed is disabled and surfaced - [ ] Fetches are spread rather than synchronised ### Watch out - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. ## Phase 4 · Reading, and read state that stays correct What you have read stays read, even when publishers edit their articles. ### Steps 1. Store revisions, never overwrite The version you read should still exist after the publisher edits it. Files: `src/revisions.ts` 2. Key read state on identity So an edit updates rather than resurrecting. Files: `src/reads.ts` 3. Keep unread counts incremental A full recount at ten thousand entries is felt every time. Files: `src/counts.ts` ### Done when - [ ] An edited article shows updated, not unread - [ ] Previous revisions are retained - [ ] Unread counts are correct and incremental at scale - [ ] Marking a folder read is atomic ### Watch out - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. ## Phase 5 · Push, for feeds that offer it Feeds advertising a hub arrive instantly and stop being polled. ### Steps 1. Subscribe and verify the callback The verification handshake is what proves you asked for it. Files: `src/websub.ts` 2. Verify push signatures An unauthenticated push is content injection into a user's reader. Files: `src/websub.ts`, `src/websub.test.ts` 3. Keep a reduced poll and renew early A lapsed subscription must degrade to polling, not to silence. Files: `src/schedule.ts` ### Done when - [ ] A subscription is established and verified - [ ] A pushed update arrives without polling - [ ] A wrongly signed push is rejected - [ ] A lapsed subscription falls back to polling ### Watch out - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. ## Not in this build - Newsletter ingestion by email address. - Article extraction for summary-only feeds. - Mobile and offline clients. - Feed fetching as a distributed system. ## After v1, if you want it - Filters that hide entries matching a rule, which is what makes a high-volume feed usable - Saved searches as feeds of their own - A shared read state API, once the counts are provably correct
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Feeds, entries and read state. DATABASE_URL=postgres://localhost:5432/reader # Required. Identifies your client and gives a route to reach you. Anonymous clients get blocked. USER_AGENT=MyReader/1.0 (+https://example.com/bot) # Required. The floor between fetches of one feed, regardless of what the schedule computes. MIN_FETCH_INTERVAL_MINUTES=15 # Required. After this a feed is disabled and surfaced. Retrying a dead site forever helps nobody. MAX_FAILURES_BEFORE_DISABLE=20 # Optional. Phase 5 only. Where hubs deliver pushes. Must be reachable from outside. CALLBACK_BASE_URL=https://reader.example.com/websub # Required. Bodies are trimmed after this. Identities and read state are kept indefinitely. ENTRY_BODY_RETENTION_DAYS=365
PRODUCT.md
# Feedbin · product brief ## Problem Feedbin's solo core is compact: build a private RSS reader workspace that imports user-supplied URLs or files, extracts metadata, supports notes, and searches the local corpus. A competent builder can reach a useful personal version in one sitting, while the paid product mainly wins on data, import reliability. ## Product outcome A reader that costs the sites it reads almost nothing, parses feeds that break their own rules, and never shows you an article twice because someone fixed a typo. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - browser automation for user-authorized imports - optional OpenAI API key stored in .env ## Explicit non-goals for v1 - Newsletter ingestion by email address. - Article extraction for summary-only feeds. - Mobile and offline clients. - Feed fetching as a distributed system. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - resilient polling of malformed and rate-limited feeds - licensed scholarly metadata ## Success criteria - Twenty real feeds including broken ones are confirmed to parse - An unchanged feed is confirmed to return not-modified with no body - An explicit wait instruction is confirmed honoured exactly - An edited article is confirmed to show updated rather than unread - Trimming is confirmed to keep identities and read state
BRIEF.md
# Build brief · Feedbin The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a feed reader that polls politely, never loses an article, and keeps read state straight to replace Feedbin. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Ruby or TypeScript with PostgreSQL · the work is scheduling, HTTP and text, not computation - Fetching: Conditional requests with stored validators, per-feed backoff · a reader that ignores caching headers is a reader that gets blocked - Parsing: A tolerant parser over RSS, Atom and JSON Feed · real feeds break their own specification constantly - Identity: A stable entry identity derived carefully, with content hashing · an entry whose identifier changes will appear as unread again forever - Push: WebSub subscriptions where a feed advertises a hub · a feed that pushes does not need polling at all ### Data model (create this before Phase 1) - feeds(id, url, site_url, title, etag, last_modified, last_fetched_at, next_fetch_at, failure_count, hub_url, disabled_at) - entries(id, feed_id, guid, url, title, content, published_at, content_hash, first_seen_at, updated_at) - revisions(entry_id, ordinal, content_hash, content, at) · because publishers edit articles after posting - subscriptions(user_id, feed_id, folder, created_at) - reads(user_id, entry_id, read_at, source) · source is manual or auto - fetches(feed_id, at, status, bytes, duration_ms, outcome) · outcome is not_modified, updated, error or rate_limited - The fetch log is what tells you whether you are a good citizen, and it is worth keeping even though it is the largest table. ### Phase 1 · Parsing feeds that do not follow the rules Build: A parser handling RSS, Atom and JSON Feed, tolerating undeclared encodings, invalid dates, missing identifiers and malformed markup. Derive a stable entry identity. Collect twenty real feeds as fixtures, including badly broken ones. Trap: Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. Done when: Twenty real feeds parse, including at least three with known specification violations. An entry keeps its identity when its title is corrected, tested with a modified fixture. An undeclared encoding is detected rather than producing mojibake. A feed with no per-entry identifiers still produces stable identities across two fetches. Do not build yet: No fetching schedule, subscriptions or reading. ### Phase 2 · Fetching without being a nuisance Build: Store the validators a server returns and send them on the next request so an unchanged feed answers with a not-modified status and no body. Identify your client honestly in the user agent. Log every fetch and its outcome. Trap: Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. Done when: An unchanged feed returns a not-modified response and no body on the second fetch, verified against a real server. The user agent identifies the client and a contact route. The fetch log records outcomes. Across twenty feeds over a day, the majority of fetches are not-modified, measured from the log. Do not build yet: No scheduling or backoff yet. ### Phase 3 · Scheduling and backing off Build: A per-feed schedule adapting to observed publishing frequency. Honour a server asking you to wait after a rate-limited response, and back off exponentially with jitter on errors. Disable a feed that has failed persistently and tell the user. Trap: Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. Done when: A rate-limited response with an explicit wait instruction is honoured exactly rather than overridden by your own schedule, tested against a stub server. Errors back off exponentially with jitter. A feed failing persistently is disabled and surfaced to the user. Feeds do not all fetch on the same clock, verified from the fetch log. Do not build yet: No reading interface yet. ### Phase 4 · Reading, and read state that stays correct Build: Subscriptions, folders and a reading interface with unread counts. Store revisions when an entry's content changes rather than overwriting. Read state keyed on entry identity so an edit does not resurrect an article. Trap: Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. Done when: An edited article shows as updated rather than unread, tested by modifying a fixture and refetching. Previous revisions are retained and viewable. Unread counts are correct across ten thousand entries and update without a full recount. Marking a folder read is atomic and does not partially apply. Do not build yet: No push or search yet. ### Phase 5 · Push, for feeds that offer it Build: Detect a hub advertised in a feed, subscribe, verify the subscription callback, and handle incoming pushes. Verify payload signatures. Reduce polling for pushed feeds but do not stop entirely, and renew subscriptions before they lapse. Trap: Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. Done when: A hub subscription is established and verified, tested against a real hub. A pushed update appears without polling. An unsigned or wrongly signed push is rejected, tested with a crafted request. Subscriptions renew before lapsing and a lapsed one falls back to polling rather than going silent. Do not build yet: Search and export come last. ### Phase 6 · Search, import and getting out Build: Full text search across entries with feed and date filters. Import and export subscriptions in the standard interchange format, preserving folders. Retention that trims entry bodies while keeping read state. Trap: Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small. Done when: Search returns results across a year of entries quickly. Subscriptions export and re-import with folders preserved, verified in another reader. Retention trims bodies while identities and read state survive, tested by trimming then refetching a feed that still lists the entry. Deleting an account removes its subscriptions and read state. Do not build yet: Nothing after this. ### Out of scope (and why) - Newsletter ingestion by email address, which is a mail receiving and parsing project in its own right. - Article extraction for feeds that publish only summaries, which is a per-site scraping problem that never stops needing maintenance. - Mobile and offline clients, which is where a reader is most used and is a synchronisation problem on top of this. - Running this at a scale where feed fetching is a distributed system, which is the hosted product's actual engineering. ### README must contain - That entry identity is derived in a documented order and why a content hash is the last resort - That validators are stored and resent so unchanged feeds cost nothing - That an explicit wait instruction from a server outranks your own schedule - That edited articles create revisions rather than resurfacing as unread - That retention trims bodies but never identities or read state
ARCHITECTURE.md
# Architecture · Feedbin ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Ruby or TypeScript with PostgreSQL | the work is scheduling, HTTP and text, not computation | | Fetching | Conditional requests with stored validators, per-feed backoff | a reader that ignores caching headers is a reader that gets blocked | | Parsing | A tolerant parser over RSS, Atom and JSON Feed | real feeds break their own specification constantly | | Identity | A stable entry identity derived carefully, with content hashing | an entry whose identifier changes will appear as unread again forever | | Push | WebSub subscriptions where a feed advertises a hub | a feed that pushes does not need polling at all | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Parser | Tolerant handling of three formats. | Any parser that survives real feeds. Strict validation is the wrong posture here. | | Identity | Deriving a stable entry identity. | Any order, provided it is documented and content hashing is the last resort. | | Fetcher | Conditional requests and honest identification. | Any client, provided validators are stored and resent. | | Scheduler | Adaptive intervals, backoff and disabling. | Any scheduler that honours an explicit wait instruction over its own plan. | | Retention | Trimming bodies without losing read state. | Any policy, provided identities survive trimming. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Feeds, entries and read state. - `USER_AGENT` · required · Identifies your client and gives a route to reach you. Anonymous clients get blocked. - `MIN_FETCH_INTERVAL_MINUTES` · required · The floor between fetches of one feed, regardless of what the schedule computes. - `MAX_FAILURES_BEFORE_DISABLE` · required · After this a feed is disabled and surfaced. Retrying a dead site forever helps nobody. - `CALLBACK_BASE_URL` · optional · Phase 5 only. Where hubs deliver pushes. Must be reachable from outside. - `ENTRY_BODY_RETENTION_DAYS` · required · Bodies are trimmed after this. Identities and read state are kept indefinitely. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Feedbin product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Ruby or TypeScript with PostgreSQL, Conditional requests with stored validators, per-feed backoff, A tolerant parser over RSS, Atom and JSON Feed, A stable entry identity derived carefully, with content hashing, WebSub subscriptions where a feed advertises a hub. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. - Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small.
MILESTONES.md
# Delivery milestones · Feedbin Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Parsing feeds that do not follow the rules Real feeds from real sites parse into entries. ### Steps 1. Handle all three formats tolerantly Real feeds violate their specifications. Refusing them is refusing the web. Files: `src/parse.ts` 2. Derive identity in a documented order Feed identifier, then link, then content hash. Record which was used. Files: `src/identity.ts`, `src/identity.test.ts` 3. Fixture twenty real feeds Including broken ones. Synthetic feeds are always better formed than real ones. Files: `fixtures/feeds/` ### Done when - [ ] Twenty real feeds parse including broken ones - [ ] A corrected typo does not change entry identity - [ ] An undeclared encoding is detected - [ ] A feed without identifiers produces stable ones ### Watch out - Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable. ## M2 · Fetching without being a nuisance An unchanged feed costs one small request and no bandwidth. ### Steps 1. Store and resend validators Both kinds a server may give you, sent back on the next request. Files: `src/fetch.ts` 2. Identify the client honestly A user agent with a name and a way to reach you. Anonymous crawlers get blocked. Files: `src/fetch.ts` 3. Log every fetch outcome The not-modified ratio is how you know whether your client behaves. Files: `src/fetches.ts` ### Done when - [ ] An unchanged feed returns not-modified with no body - [ ] The user agent identifies the client and a contact - [ ] Every fetch is logged with its outcome - [ ] Most fetches across a day are not-modified ### Watch out - Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds. ## M3 · Scheduling and backing off Feeds are polled at a sensible rate and problems reduce it rather than continuing. ### Steps 1. Honour an explicit wait instruction It outranks your own schedule. Ignoring it is how you get blocked permanently. Files: `src/schedule.ts` 2. Back off exponentially with jitter Jitter is what stops every feed retrying at the same instant. Files: `src/backoff.ts`, `src/backoff.test.ts` 3. Disable persistently failing feeds And tell the user. Silent dead feeds are how a reader quietly stops working. Files: `src/health.ts` ### Done when - [ ] An explicit wait instruction is honoured exactly - [ ] Errors back off exponentially with jitter - [ ] A persistently failing feed is disabled and surfaced - [ ] Fetches are spread rather than synchronised ### Watch out - Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure. ## M4 · Reading, and read state that stays correct What you have read stays read, even when publishers edit their articles. ### Steps 1. Store revisions, never overwrite The version you read should still exist after the publisher edits it. Files: `src/revisions.ts` 2. Key read state on identity So an edit updates rather than resurrecting. Files: `src/reads.ts` 3. Keep unread counts incremental A full recount at ten thousand entries is felt every time. Files: `src/counts.ts` ### Done when - [ ] An edited article shows updated, not unread - [ ] Previous revisions are retained - [ ] Unread counts are correct and incremental at scale - [ ] Marking a folder read is atomic ### Watch out - Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new. ## M5 · Push, for feeds that offer it Feeds advertising a hub arrive instantly and stop being polled. ### Steps 1. Subscribe and verify the callback The verification handshake is what proves you asked for it. Files: `src/websub.ts` 2. Verify push signatures An unauthenticated push is content injection into a user's reader. Files: `src/websub.ts`, `src/websub.test.ts` 3. Keep a reduced poll and renew early A lapsed subscription must degrade to polling, not to silence. Files: `src/schedule.ts` ### Done when - [ ] A subscription is established and verified - [ ] A pushed update arrives without polling - [ ] A wrongly signed push is rejected - [ ] A lapsed subscription falls back to polling ### Watch out - Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it. ## M6 · Search, import and getting out (production only) You can find an article from a year ago and take your subscriptions elsewhere. ### Steps 1. Index entry text with filters Feed and date, combined with text. A year of entries should search instantly. Files: `src/search.ts` 2. Support the standard interchange format Both directions, with folders. A reader you cannot leave is one to avoid. Files: `src/opml.ts` 3. Trim bodies, keep identities Read state is small and its loss resurrects old articles. Files: `src/retention.ts` ### Done when - [ ] Search across a year returns quickly - [ ] Export re-imports elsewhere with folders intact - [ ] Trimming keeps identities and read state - [ ] Account deletion removes subscriptions and reads ### Watch out - Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small.
OPERATIONS.md
# Operations · Feedbin ## Backup Subscriptions and read state are the irreplaceable data and are small. Entry bodies are large and re-fetchable for recent items only, so back up the former carefully and the latter as convenient. ## Restore Restore and confirm read state and unread counts are consistent, since a restore that loses recent reads presents a user with hundreds of already-read articles. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring The ratio of not-modified to full responses is the health metric for your behaviour as a client. Disabled feed count and oldest successful fetch tell you whether the reader is quietly rotting. ## Incident checklist Being blocked by a site is the incident that matters, and the cause is nearly always ignoring caching headers or an explicit wait instruction. The fetch log is what lets you demonstrate what your client actually did. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Twenty real feeds including broken ones are confirmed to parse - [ ] An unchanged feed is confirmed to return not-modified with no body - [ ] An explicit wait instruction is confirmed honoured exactly - [ ] An edited article is confirmed to show updated rather than unread - [ ] Trimming is confirmed to keep identities and read state ## Launch constraint Do not market omitted Feedbin capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Feeds, entries and read state. DATABASE_URL=postgres://localhost:5432/reader # Required. Identifies your client and gives a route to reach you. Anonymous clients get blocked. USER_AGENT=MyReader/1.0 (+https://example.com/bot) # Required. The floor between fetches of one feed, regardless of what the schedule computes. MIN_FETCH_INTERVAL_MINUTES=15 # Required. After this a feed is disabled and surfaced. Retrying a dead site forever helps nobody. MAX_FAILURES_BEFORE_DISABLE=20 # Optional. Phase 5 only. Where hubs deliver pushes. Must be reachable from outside. CALLBACK_BASE_URL=https://reader.example.com/websub # Required. Bodies are trimmed after this. Identities and read state are kept indefinitely. ENTRY_BODY_RETENTION_DAYS=365
$ choose a build depth, inspect the files, then open the complete pack in your agent
Feedbin: Researchers pay for correct metadata, resilient importers, citation coverage, and workflows that survive publisher and browser changes.
xpublisher-specific import reliability
xcitation graph scale
xteam libraries and institutional access
xresilient polling of malformed and rate-limited feeds
xlicensed scholarly metadata
share on X ↗"I just replaced Feedbin ($7/mo) with one prompt"
Don't feel like building it? These folks already made it free.
FoloRSS and one newsletter inbox for free; the 151st feed starts a pricing conversation.open source↗
NewsBlurRSS, email newsletters, search and filtering, provided you operate the whole contraption.open source↗no votes, no pay-to-list · just what's real
Feedbin pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| standard | $7/user | $5.83/user | All Feedbin RSS, newsletter, search and filtering features; no separate lower paid tier. |
free tierno free tier; 30-day trial with no card
billingmonthly + annual
verified 2026-08-13 · source ↗
Vibecode Feedbin
Yes. A competent AI coding agent (Claude Code, Codex, Cursor) can build a usable personal Feedbin replacement in one session with the prompt on this page. It runs on your own machine or server with no subscription.
How much does Feedbin cost?
Feedbin costs about $7/month (Standard, checked 2026-07-31), which is $84 per year. That's what you save by replacing it with one prompt.
What do I lose by replacing Feedbin?
Honestly: publisher-specific import reliability; citation graph scale; team libraries and institutional access; resilient polling of malformed and rate-limited feeds; licensed scholarly metadata. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Feedbin?
Yes: Folo (RSS and one newsletter inbox for free; the 151st feed starts a pricing conversation.) NewsBlur (RSS, email newsletters, search and filtering, provided you operate the whole contraption.) The prompt is for when you want it exactly your way.