Build Feedbin

YESreplaces $7/mosaves $84/yrback to the verdict

0%0 of 20 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A feed reader whose real work is being a well-behaved HTTP client at scale: conditional requests so unchanged feeds cost nothing, backoff that respects what servers ask for, parsing feeds that violate their own specifications, and read state that survives a feed rewriting its own entries.

estimated effort multi-daythe files for this build are in the project pack

RuntimeRuby or TypeScript with PostgreSQLFetchingConditional requests with stored validators, per-feed backoffParsingA tolerant parser over RSS, Atom and JSON FeedIdentityA stable entry identity derived carefully, with content hashingPushWebSub subscriptions where a feed advertises a hub

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the fetcher and the application. Either is fine, since the work is HTTP and text.
Get it Your package manager, or nvm and rbenv respectively.
Verify The runtime version prints and a HTTP client library installs
installfree locally
Why Entries, revisions and read state with correct unread counts want a relational database with real indexes.
Get it Docker or your package manager. open ↗
Verify psql connects and full text search returns a result
have readyfree
Why Every parsing decision in phase 1 is driven by how real feeds actually misbehave, and no synthetic fixture reproduces that.
Get it Collect feeds you actually read, plus deliberately a few from old software with undeclared encodings or invalid dates. Save the raw bytes as fixtures.
Verify Twenty raw feeds are saved and at least three fail strict validation
installfree
Why Conditional requests, rate limiting and backoff cannot be tested against real servers without being exactly the nuisance this build is trying not to be.
Get it Any small server that can return validators, not-modified responses, error statuses and explicit wait instructions on demand.
Verify It can return a not-modified response and a rate-limited response with a wait instruction
have readyfree
Why Phase 5 needs a real hub, since the verification handshake and signature format are where implementations differ from the specification.
Get it Find a feed advertising a hub, or run an open source hub locally.
Verify A subscription is established and a push is received

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- feeds(id, url, site_url, title, etag, last_modified, last_fetched_at, next_fetch_at, failure_count, hub_url, disabled_at)
- entries(id, feed_id, guid, url, title, content, published_at, content_hash, first_seen_at, updated_at)
- revisions(entry_id, ordinal, content_hash, content, at) · because publishers edit articles after posting
- subscriptions(user_id, feed_id, folder, created_at)
- reads(user_id, entry_id, read_at, source) · source is manual or auto
- fetches(feed_id, at, status, bytes, duration_ms, outcome) · outcome is not_modified, updated, error or rate_limited
- The fetch log is what tells you whether you are a good citizen, and it is worth keeping even though it is the largest table.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_URL`secret	required	`postgres://localhost:5432/reader`	Feeds, entries and read state.
`USER_AGENT`	required	`MyReader/1.0 (+https://example.com/bot)`	Identifies your client and gives a route to reach you. Anonymous clients get blocked.
`MIN_FETCH_INTERVAL_MINUTES`	required	`15`	The floor between fetches of one feed, regardless of what the schedule computes.
`MAX_FAILURES_BEFORE_DISABLE`	required	`20`	After this a feed is disabled and surfaced. Retrying a dead site forever helps nobody.
`CALLBACK_BASE_URL`	optional	`https://reader.example.com/websub`	Phase 5 only. Where hubs deliver pushes. Must be reachable from outside.
`ENTRY_BODY_RETENTION_DAYS`	required	`365`	Bodies are trimmed after this. Identities and read state are kept indefinitely.

The build, in order

1
Parsing feeds that do not follow the rules
Real feeds from real sites parse into entries.
1. Real feeds violate their specifications. Refusing them is refusing the web.
  Files src/parse.ts
2. Feed identifier, then link, then content hash. Record which was used.
  Files src/identity.tssrc/identity.test.ts
3. Including broken ones. Synthetic feeds are always better formed than real ones.
  Files fixtures/feeds/
done when · tick each as it passes
watch out
- Deriving an entry's identity from its title or its content means a publisher fixing a typo creates what looks like a new entry, and the user sees the same article again. Prefer the feed's own identifier, fall back to the link, and only then to a content hash, recording which was used so the behaviour is explicable.
next phase
2
Fetching without being a nuisance
An unchanged feed costs one small request and no bandwidth.
1. Both kinds a server may give you, sent back on the next request.
  Files src/fetch.ts
2. A user agent with a name and a way to reach you. Anonymous crawlers get blocked.
  Files src/fetch.ts
3. The not-modified ratio is how you know whether your client behaves.
  Files src/fetches.ts
done when · tick each as it passes
watch out
- Downloading the full feed every poll is how a reader gets blocked, and at any scale it is a meaningful cost imposed on someone else's server for nothing. Store the validators the server gives you and send them back, and treat a high ratio of full downloads to not-modified responses as a defect in your client rather than in the feeds.
next phase
3
Scheduling and backing off
Feeds are polled at a sensible rate and problems reduce it rather than continuing.
1. It outranks your own schedule. Ignoring it is how you get blocked permanently.
  Files src/schedule.ts
2. Jitter is what stops every feed retrying at the same instant.
  Files src/backoff.tssrc/backoff.test.ts
3. And tell the user. Silent dead feeds are how a reader quietly stops working.
  Files src/health.ts
done when · tick each as it passes
watch out
- Retrying a failing feed on the same schedule forever turns one dead site into thousands of pointless requests, and retrying every feed on the same clock produces a thundering burst at the top of every hour. Back off with jitter, honour an explicit wait instruction over your own schedule, and stop entirely after persistent failure.
next phase
4
Reading, and read state that stays correct
What you have read stays read, even when publishers edit their articles.
1. The version you read should still exist after the publisher edits it.
  Files src/revisions.ts
2. So an edit updates rather than resurrecting.
  Files src/reads.ts
3. A full recount at ten thousand entries is felt every time.
  Files src/counts.ts
done when · tick each as it passes
watch out
- Overwriting an entry when its content changes loses the version you read and, if the identity is content-derived, marks it unread again. Store a revision and keep the read state attached to the identity, so an edited article shows as updated rather than as new.
next phase
5
Push, for feeds that offer it
Feeds advertising a hub arrive instantly and stop being polled.
1. The verification handshake is what proves you asked for it.
  Files src/websub.ts
2. An unauthenticated push is content injection into a user's reader.
  Files src/websub.tssrc/websub.test.ts
3. A lapsed subscription must degrade to polling, not to silence.
  Files src/schedule.ts
done when · tick each as it passes
watch out
- Trusting an unauthenticated push means anyone who learns your callback address can inject entries into a user's reader, which is a content injection vector with your name on it. Verify the signature where the hub provides one, and confirm pushed content against the feed before trusting it.
6
Search, import and getting outproduct builder
You can find an article from a year ago and take your subscriptions elsewhere.
1. Feed and date, combined with text. A year of entries should search instantly.
  Files src/search.ts
2. Both directions, with folders. A reader you cannot leave is one to avoid.
  Files src/opml.ts
3. Read state is small and its loss resurrects old articles.
  Files src/retention.ts
done when · tick each as it passes
watch out
- Deleting old entries to control database size also deletes the read state attached to them, so a feed that still lists a two-year-old article will present it as unread the next time it is fetched. Trim the body and keep the identity and read state, which are small.

what this build does not replace

Newsletter ingestion by email address.
Article extraction for summary-only feeds.
Mobile and offline clients.
Feed fetching as a distributed system.
publisher-specific import reliability
citation graph scale
team libraries and institutional access
resilient polling of malformed and rate-limited feeds
licensed scholarly metadata

after v1, if you want it

Filters that hide entries matching a rule, which is what makes a high-volume feed usable
Saved searches as feeds of their own
A shared read state API, once the counts are provably correct

Need the files? The project pack on the verdict page hands your agent the whole brief · more rss & research.