Vibecode Dailygram
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The personal digest pipeline is a realistic weekend build if you bring feeds or a data provider. Reliable ingestion from Instagram, X, TikTok, and LinkedIn is not a one-shot project: access restrictions, changing page formats, rate limits, and connector maintenance are the product's durable edge.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Dailygram. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Dailygram · indie build A personal digest pipeline whose hard problems are deduplicating items across feeds that rewrite their own identifiers, summarising without inventing anything, and sending exactly one digest a day even when the mail server times out. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with Express and one scheduler in the same process | a digest that only runs when the laptop is open is a bookmark folder | | Storage | SQLite through better-sqlite3, with the digest state machine in the schema | sending twice is a database problem, so the database prevents it | | Ingestion | RSS, Atom and JSON feeds that you supply, fetched with conditional requests | no logging in, no scraping, no cookie jar, and therefore nothing to get blocked | | Summarising | One model call per digest over new items only, with a daily cost ledger | a chatty feed is a bill, and you want to see it before the invoice | | Delivery | SMTP through a provider you already have, one message per digest | delivery reporting you would otherwise have to build | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the poller, the scheduler and the localhost dashboard in one process. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A machine that stays on** · a few pounds a month, or free if you own the box - Why: The digest fires on a schedule, and a laptop that sleeps turns a daily digest into a weekly surprise. - Get it: A small server, a home box or a single-board computer. - Verify: A digest scheduled for seven in the morning has arrived by breakfast - [ ] **Working feed URLs for the profiles you care about** · free, or whatever a feed service charges - Why: This is the actual constraint on the whole product, and discovering that a network publishes none is better done before you build than after. - Get it: Check each profile for an RSS, Atom or JSON feed, or a service that publishes one for it within that network's terms. - Verify: Each URL parses and returns recent entries - [ ] **An API key for a language model** · pennies a day at a few dozen items - Why: Summarising is the only external cost and it needs a ceiling from day one. - Get it: Create a key with its own spend limit rather than reusing one you use elsewhere. - Verify: A one item summary returns and the ledger records the tokens - [ ] **SMTP credentials for a provider you already use** · free tier is ample for one message a day - Why: Sending your own mail from a home address is a separate project in deliverability. - Get it: Any transactional provider with a verified sending domain. - Verify: A test digest arrives and the provider reports it accepted ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any login, cookie or page scraping. - Feed discovery from a handle. - Maintained connectors for closed networks. - Multiple recipients and subscription management. - maintained ingestion across four social networks - resilience to rate limits, blocks, and page changes - automatic profile discovery and source setup - managed email delivery and monitoring If one of those is essential to you, that is the reason to keep paying for Dailygram, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Dailygram The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a once-a-day email digest built from feeds you supply for the public profiles you actually follow to replace Dailygram. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with Express and one scheduler in the same process · a digest that only runs when the laptop is open is a bookmark folder - Storage: SQLite through better-sqlite3, with the digest state machine in the schema · sending twice is a database problem, so the database prevents it - Ingestion: RSS, Atom and JSON feeds that you supply, fetched with conditional requests · no logging in, no scraping, no cookie jar, and therefore nothing to get blocked - Summarising: One model call per digest over new items only, with a daily cost ledger · a chatty feed is a bill, and you want to see it before the invoice - Delivery: SMTP through a provider you already have, one message per digest · delivery reporting you would otherwise have to build ### Data model (create this before Phase 1) - sources(id, network, handle, profile_url, feed_url, feed_format, etag, last_modified, enabled, consecutive_failures, last_ok_at, paused_reason) - items(id, source_id, remote_id, canonical_url, url_hash, content_hash, title, body_text, posted_at, first_seen_at) - summaries(item_id, model, prompt_version, text, tokens_in, tokens_out, created_at) · keyed by item, so a rebuilt digest never pays twice - digests(id, window_start, window_end, state, item_count, built_at, sent_at, message_id, last_error) · state is building, ready, sending, sent, needs_check or failed - digest_items(digest_id, item_id, ordinal) - cost_ledger(day, tokens_in, tokens_out, calls) · one row per day, checked against a ceiling before every call - Identity is a ladder: remote identifier first, then canonical URL, then a hash of the text, because feeds rewrite their own identifiers and republish old posts more often than you would believe. ### Phase 1 · Sources are feeds, and feeds are checked when you add them Build: A localhost page to add, edit, pause and delete sources. Each source stores its network, handle, profile URL and feed URL, and saving fetches and parses the feed before accepting it. Poll with conditional requests, storing the validators the server returns, and back off per source after failures instead of hammering a feed that is gone. Trap: Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. Done when: Saving a source fetches and parses the feed and refuses the save with the parser's error when it fails. Polling sends the validators the server previously returned and a response saying nothing changed costs no parsing work, verified in the log. A source that fails repeatedly backs off and records the reason rather than retrying at full rate. Every network listed in the interface states whether a feed exists for it or not. Do not build yet: No deduplication, summaries, digests or email. ### Phase 2 · The same post, twice, from three angles Build: Normalise every entry to one schema. Deduplicate on the remote identifier, then the canonical URL with tracking parameters stripped, then a hash of the text. On the very first poll of a new source, record everything as seen and send none of it. Trap: The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. Done when: The first poll of a source with a large archive produces zero digest items and a populated items table, tested with a recorded feed. An item whose remote identifier changes but whose URL is unchanged is not reinserted, tested. Tracking parameters do not make two URLs distinct, tested with a link carrying campaign parameters. A post edited in place updates rather than duplicating. Do not build yet: No summaries, digests or email yet. ### Phase 3 · Summaries that cannot invent a post Build: Summarise only items new since the last digest, in one call, giving the model the item ids and requiring each output line to carry the id it came from. Drop any line whose id is not in the set you sent. Store the summary against the item so a rebuilt digest never pays again. Check the day's ledger against a ceiling before the call and refuse rather than exceeding it. Trap: Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. Done when: A line whose item id was not in the request is dropped before rendering, tested by feeding a fabricated response. Every link in the digest is built from the stored canonical URL, never from model output, verified by review. Summaries are stored per item and rebuilding a digest makes no new calls, tested. A day already at its token ceiling refuses the call and records the refusal instead of spending. Do not build yet: No sending or scheduling yet. ### Phase 4 · One digest a day, once Build: A digest covers a named window and there is one row per window, claimed by moving it to sending in its own transaction before the mail server is touched. Record the message identifier the server returns. A send whose outcome is unknown becomes needs_check, never a retry, and is settled by looking rather than by sending again. Trap: Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. Done when: Killing the process between the mail server accepting and the database recording leaves one delivered message and a row in needs_check, tested with a fake SMTP server. Two scheduler ticks in the same window produce one digest, tested by firing them concurrently. A window with no new items sends nothing and records why, rather than sending an empty message. The stored message identifier matches the one the server returned. Do not build yet: No source health alerts or archive export yet. ### Phase 5 · Knowing when a feed quietly died Build: A dashboard showing each source's last success, consecutive failures and last error, plus the next digest and a preview of exactly what would be sent. Pause a source after a stated run of failures and alert once. Export the whole archive as JSON plus a folder of Markdown. Bind to localhost and keep every secret in the environment. Trap: A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors. Done when: The dashboard shows last success, failure count and last error for every source, and a preview of the pending digest identical to what would be sent. A source failing for the stated run pauses and alerts once, tested by pointing one at a dead URL. A source that silently stops producing items is flagged by rate, tested with a feed that goes empty. The archive exports and the export is readable without this application. Do not build yet: Nothing after this. ### Out of scope (and why) - Logging into any social network, storing session cookies, or fetching pages a network's terms say you may not. A digest tool that gets your account restricted has cost more than it saved. - Discovering a feed from a handle. Most networks do not publish one, and the honest interface says so rather than pretending the source is broken. - Instagram, X, TikTok and LinkedIn as first-class integrations, which is exactly the maintained ingestion the paid product sells and the part that breaks monthly. - Multiple recipients and subscription management, which turns a personal digest into a mailing list with its own compliance work. ### README must contain - That sources are feed URLs you supply, validated on save, and that some networks publish none - That nothing here logs into a network, stores a cookie, or scrapes a page - That the first poll of a new source sends nothing, because a feed's archive is not news - That every digest line carries an item id and lines the model invented are dropped - That a digest window is claimed before the mail server is touched, so an unknown outcome is checked rather than resent ===== AGENTS.md ===== # Agent instructions · Dailygram indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with Express and one scheduler in the same process, SQLite through better-sqlite3, with the digest state machine in the schema, RSS, Atom and JSON feeds that you supply, fetched with conditional requests, One model call per digest over new items only, with a daily cost ledger, SMTP through a provider you already have, one message per digest. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. ===== BUILD_PLAN.md ===== # Build plan · Dailygram A personal digest pipeline whose hard problems are deduplicating items across feeds that rewrite their own identifiers, summarising without inventing anything, and sending exactly one digest a day even when the mail server times out. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sources are feeds, and feeds are checked when you add them Adding a source means giving a feed URL that is fetched and parsed there and then. ### Steps 1. Validate the feed at save time The moment to discover a dead URL is while the person is looking at it. Files: `src/sources.js`, `src/feeds/parse.js` 2. Send conditional request validators Store what the server gave you and hand it back next time. Files: `src/feeds/fetch.js`, `tests/conditional.test.js` 3. Back off per source and record why A feed that is gone should go quiet, not louder. Files: `src/poller.js` ### Done when - [ ] A bad feed URL is refused at save with the parse error - [ ] Unchanged responses skip parsing - [ ] Repeated failures back off and record a reason - [ ] The interface says which networks have no feed ### Watch out - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. ## Phase 2 · The same post, twice, from three angles An item already seen never appears in a second digest, whatever the feed does to its identifiers. ### Steps 1. Watermark a new source on first poll Seen, not sent. This is the difference between a tool and a spam incident. Files: `src/ingest.js`, `tests/first-poll.test.js` 2. Deduplicate down a ladder, not on one key Identifier, then canonical URL, then content hash. Files: `src/dedupe.js`, `tests/dedupe.test.js` 3. Strip tracking parameters before hashing the URL Two links to the same post differ only in the campaign tag. Files: `src/canonical.js` ### Done when - [ ] A new source's first poll sends nothing - [ ] A rewritten identifier does not duplicate an item - [ ] Campaign parameters do not create a second item - [ ] An edited post updates in place ### Watch out - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. ## Phase 3 · Summaries that cannot invent a post Every line in the digest points at an item that exists in your database. ### Steps 1. Require an item id on every line An unattributable sentence is a hallucination with good grammar. Files: `src/summarise.js`, `tests/citation.test.js` 2. Render links from the database only Model output never becomes a URL. Files: `src/render/digest.js` 3. Check the daily ledger before each call A ceiling you enforce beats a bill you discover. Files: `src/costs.js`, `tests/ceiling.test.js` ### Done when - [ ] Uncited or unknown-id lines are dropped - [ ] Links come only from stored URLs - [ ] A rebuilt digest makes no new model calls - [ ] The daily ceiling refuses rather than overspends ### Watch out - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. ## Phase 4 · One digest a day, once A crash or a mail server timeout never produces two copies of the same digest. ### Steps 1. One row per window, claimed in a transaction The claim happens before any network call. That is the whole protection. Files: `src/digest.js`, `tests/exactly-once.test.js` 2. Record the message identifier the server returned It is the only proof that accepted actually happened. Files: `src/mailer.js` 3. Treat an unknown outcome as needs_check Never as a reason to send again. Files: `src/reconcile.js` ### Done when - [ ] A kill between accept and record yields one email - [ ] Concurrent ticks produce one digest - [ ] An empty window sends nothing and says why - [ ] The stored message identifier matches the server's ### Watch out - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. ## Not in this build - Any login, cookie or page scraping. - Feed discovery from a handle. - Maintained connectors for closed networks. - Multiple recipients and subscription management. ## After v1, if you want it - A weekly digest alongside the daily one, reusing the same window machinery - Per-source importance so a quiet account is never crowded out by a busy one - Full text search over the archive, which you already store item by item ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Sources, items, summaries and digests. The archive. DATABASE_PATH=./data/digest.db # Required. Per source, subject to per-source backoff after failures. POLL_INTERVAL_MINUTES=360 # Required. One window per day, named by date in this timezone. DIGEST_HOUR_LOCAL=07 # Required · secret. Summarising only. Give it its own spend limit at the provider. LLM_API_KEY=sk-...redacted # Required. Checked before every call. Refusing beats overspending. DAILY_TOKEN_CEILING=120000 # Required · secret. Delivery. The returned message identifier is stored. SMTP_URL=smtps://user:redacted@smtp.example:465 # Required. One recipient. This is a personal digest, not a list. DIGEST_TO=you@example.com # Required. Consecutive failures before a source pauses and alerts once. SOURCE_FAIL_PAUSE=5
You are building a lean indie version of Dailygram. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Dailygram · indie build A personal digest pipeline whose hard problems are deduplicating items across feeds that rewrite their own identifiers, summarising without inventing anything, and sending exactly one digest a day even when the mail server times out. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with Express and one scheduler in the same process | a digest that only runs when the laptop is open is a bookmark folder | | Storage | SQLite through better-sqlite3, with the digest state machine in the schema | sending twice is a database problem, so the database prevents it | | Ingestion | RSS, Atom and JSON feeds that you supply, fetched with conditional requests | no logging in, no scraping, no cookie jar, and therefore nothing to get blocked | | Summarising | One model call per digest over new items only, with a daily cost ledger | a chatty feed is a bill, and you want to see it before the invoice | | Delivery | SMTP through a provider you already have, one message per digest | delivery reporting you would otherwise have to build | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the poller, the scheduler and the localhost dashboard in one process. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A machine that stays on** · a few pounds a month, or free if you own the box - Why: The digest fires on a schedule, and a laptop that sleeps turns a daily digest into a weekly surprise. - Get it: A small server, a home box or a single-board computer. - Verify: A digest scheduled for seven in the morning has arrived by breakfast - [ ] **Working feed URLs for the profiles you care about** · free, or whatever a feed service charges - Why: This is the actual constraint on the whole product, and discovering that a network publishes none is better done before you build than after. - Get it: Check each profile for an RSS, Atom or JSON feed, or a service that publishes one for it within that network's terms. - Verify: Each URL parses and returns recent entries - [ ] **An API key for a language model** · pennies a day at a few dozen items - Why: Summarising is the only external cost and it needs a ceiling from day one. - Get it: Create a key with its own spend limit rather than reusing one you use elsewhere. - Verify: A one item summary returns and the ledger records the tokens - [ ] **SMTP credentials for a provider you already use** · free tier is ample for one message a day - Why: Sending your own mail from a home address is a separate project in deliverability. - Get it: Any transactional provider with a verified sending domain. - Verify: A test digest arrives and the provider reports it accepted ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any login, cookie or page scraping. - Feed discovery from a handle. - Maintained connectors for closed networks. - Multiple recipients and subscription management. - maintained ingestion across four social networks - resilience to rate limits, blocks, and page changes - automatic profile discovery and source setup - managed email delivery and monitoring If one of those is essential to you, that is the reason to keep paying for Dailygram, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Dailygram The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a once-a-day email digest built from feeds you supply for the public profiles you actually follow to replace Dailygram. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with Express and one scheduler in the same process · a digest that only runs when the laptop is open is a bookmark folder - Storage: SQLite through better-sqlite3, with the digest state machine in the schema · sending twice is a database problem, so the database prevents it - Ingestion: RSS, Atom and JSON feeds that you supply, fetched with conditional requests · no logging in, no scraping, no cookie jar, and therefore nothing to get blocked - Summarising: One model call per digest over new items only, with a daily cost ledger · a chatty feed is a bill, and you want to see it before the invoice - Delivery: SMTP through a provider you already have, one message per digest · delivery reporting you would otherwise have to build ### Data model (create this before Phase 1) - sources(id, network, handle, profile_url, feed_url, feed_format, etag, last_modified, enabled, consecutive_failures, last_ok_at, paused_reason) - items(id, source_id, remote_id, canonical_url, url_hash, content_hash, title, body_text, posted_at, first_seen_at) - summaries(item_id, model, prompt_version, text, tokens_in, tokens_out, created_at) · keyed by item, so a rebuilt digest never pays twice - digests(id, window_start, window_end, state, item_count, built_at, sent_at, message_id, last_error) · state is building, ready, sending, sent, needs_check or failed - digest_items(digest_id, item_id, ordinal) - cost_ledger(day, tokens_in, tokens_out, calls) · one row per day, checked against a ceiling before every call - Identity is a ladder: remote identifier first, then canonical URL, then a hash of the text, because feeds rewrite their own identifiers and republish old posts more often than you would believe. ### Phase 1 · Sources are feeds, and feeds are checked when you add them Build: A localhost page to add, edit, pause and delete sources. Each source stores its network, handle, profile URL and feed URL, and saving fetches and parses the feed before accepting it. Poll with conditional requests, storing the validators the server returns, and back off per source after failures instead of hammering a feed that is gone. Trap: Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. Done when: Saving a source fetches and parses the feed and refuses the save with the parser's error when it fails. Polling sends the validators the server previously returned and a response saying nothing changed costs no parsing work, verified in the log. A source that fails repeatedly backs off and records the reason rather than retrying at full rate. Every network listed in the interface states whether a feed exists for it or not. Do not build yet: No deduplication, summaries, digests or email. ### Phase 2 · The same post, twice, from three angles Build: Normalise every entry to one schema. Deduplicate on the remote identifier, then the canonical URL with tracking parameters stripped, then a hash of the text. On the very first poll of a new source, record everything as seen and send none of it. Trap: The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. Done when: The first poll of a source with a large archive produces zero digest items and a populated items table, tested with a recorded feed. An item whose remote identifier changes but whose URL is unchanged is not reinserted, tested. Tracking parameters do not make two URLs distinct, tested with a link carrying campaign parameters. A post edited in place updates rather than duplicating. Do not build yet: No summaries, digests or email yet. ### Phase 3 · Summaries that cannot invent a post Build: Summarise only items new since the last digest, in one call, giving the model the item ids and requiring each output line to carry the id it came from. Drop any line whose id is not in the set you sent. Store the summary against the item so a rebuilt digest never pays again. Check the day's ledger against a ceiling before the call and refuse rather than exceeding it. Trap: Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. Done when: A line whose item id was not in the request is dropped before rendering, tested by feeding a fabricated response. Every link in the digest is built from the stored canonical URL, never from model output, verified by review. Summaries are stored per item and rebuilding a digest makes no new calls, tested. A day already at its token ceiling refuses the call and records the refusal instead of spending. Do not build yet: No sending or scheduling yet. ### Phase 4 · One digest a day, once Build: A digest covers a named window and there is one row per window, claimed by moving it to sending in its own transaction before the mail server is touched. Record the message identifier the server returns. A send whose outcome is unknown becomes needs_check, never a retry, and is settled by looking rather than by sending again. Trap: Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. Done when: Killing the process between the mail server accepting and the database recording leaves one delivered message and a row in needs_check, tested with a fake SMTP server. Two scheduler ticks in the same window produce one digest, tested by firing them concurrently. A window with no new items sends nothing and records why, rather than sending an empty message. The stored message identifier matches the one the server returned. Do not build yet: No source health alerts or archive export yet. ### Phase 5 · Knowing when a feed quietly died Build: A dashboard showing each source's last success, consecutive failures and last error, plus the next digest and a preview of exactly what would be sent. Pause a source after a stated run of failures and alert once. Export the whole archive as JSON plus a folder of Markdown. Bind to localhost and keep every secret in the environment. Trap: A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors. Done when: The dashboard shows last success, failure count and last error for every source, and a preview of the pending digest identical to what would be sent. A source failing for the stated run pauses and alerts once, tested by pointing one at a dead URL. A source that silently stops producing items is flagged by rate, tested with a feed that goes empty. The archive exports and the export is readable without this application. Do not build yet: Nothing after this. ### Out of scope (and why) - Logging into any social network, storing session cookies, or fetching pages a network's terms say you may not. A digest tool that gets your account restricted has cost more than it saved. - Discovering a feed from a handle. Most networks do not publish one, and the honest interface says so rather than pretending the source is broken. - Instagram, X, TikTok and LinkedIn as first-class integrations, which is exactly the maintained ingestion the paid product sells and the part that breaks monthly. - Multiple recipients and subscription management, which turns a personal digest into a mailing list with its own compliance work. ### README must contain - That sources are feed URLs you supply, validated on save, and that some networks publish none - That nothing here logs into a network, stores a cookie, or scrapes a page - That the first poll of a new source sends nothing, because a feed's archive is not news - That every digest line carries an item id and lines the model invented are dropped - That a digest window is claimed before the mail server is touched, so an unknown outcome is checked rather than resent ===== AGENTS.md ===== # Agent instructions · Dailygram indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with Express and one scheduler in the same process, SQLite through better-sqlite3, with the digest state machine in the schema, RSS, Atom and JSON feeds that you supply, fetched with conditional requests, One model call per digest over new items only, with a daily cost ledger, SMTP through a provider you already have, one message per digest. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. ===== BUILD_PLAN.md ===== # Build plan · Dailygram A personal digest pipeline whose hard problems are deduplicating items across feeds that rewrite their own identifiers, summarising without inventing anything, and sending exactly one digest a day even when the mail server times out. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sources are feeds, and feeds are checked when you add them Adding a source means giving a feed URL that is fetched and parsed there and then. ### Steps 1. Validate the feed at save time The moment to discover a dead URL is while the person is looking at it. Files: `src/sources.js`, `src/feeds/parse.js` 2. Send conditional request validators Store what the server gave you and hand it back next time. Files: `src/feeds/fetch.js`, `tests/conditional.test.js` 3. Back off per source and record why A feed that is gone should go quiet, not louder. Files: `src/poller.js` ### Done when - [ ] A bad feed URL is refused at save with the parse error - [ ] Unchanged responses skip parsing - [ ] Repeated failures back off and record a reason - [ ] The interface says which networks have no feed ### Watch out - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. ## Phase 2 · The same post, twice, from three angles An item already seen never appears in a second digest, whatever the feed does to its identifiers. ### Steps 1. Watermark a new source on first poll Seen, not sent. This is the difference between a tool and a spam incident. Files: `src/ingest.js`, `tests/first-poll.test.js` 2. Deduplicate down a ladder, not on one key Identifier, then canonical URL, then content hash. Files: `src/dedupe.js`, `tests/dedupe.test.js` 3. Strip tracking parameters before hashing the URL Two links to the same post differ only in the campaign tag. Files: `src/canonical.js` ### Done when - [ ] A new source's first poll sends nothing - [ ] A rewritten identifier does not duplicate an item - [ ] Campaign parameters do not create a second item - [ ] An edited post updates in place ### Watch out - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. ## Phase 3 · Summaries that cannot invent a post Every line in the digest points at an item that exists in your database. ### Steps 1. Require an item id on every line An unattributable sentence is a hallucination with good grammar. Files: `src/summarise.js`, `tests/citation.test.js` 2. Render links from the database only Model output never becomes a URL. Files: `src/render/digest.js` 3. Check the daily ledger before each call A ceiling you enforce beats a bill you discover. Files: `src/costs.js`, `tests/ceiling.test.js` ### Done when - [ ] Uncited or unknown-id lines are dropped - [ ] Links come only from stored URLs - [ ] A rebuilt digest makes no new model calls - [ ] The daily ceiling refuses rather than overspends ### Watch out - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. ## Phase 4 · One digest a day, once A crash or a mail server timeout never produces two copies of the same digest. ### Steps 1. One row per window, claimed in a transaction The claim happens before any network call. That is the whole protection. Files: `src/digest.js`, `tests/exactly-once.test.js` 2. Record the message identifier the server returned It is the only proof that accepted actually happened. Files: `src/mailer.js` 3. Treat an unknown outcome as needs_check Never as a reason to send again. Files: `src/reconcile.js` ### Done when - [ ] A kill between accept and record yields one email - [ ] Concurrent ticks produce one digest - [ ] An empty window sends nothing and says why - [ ] The stored message identifier matches the server's ### Watch out - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. ## Not in this build - Any login, cookie or page scraping. - Feed discovery from a handle. - Maintained connectors for closed networks. - Multiple recipients and subscription management. ## After v1, if you want it - A weekly digest alongside the daily one, reusing the same window machinery - Per-source importance so a quiet account is never crowded out by a busy one - Full text search over the archive, which you already store item by item ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Sources, items, summaries and digests. The archive. DATABASE_PATH=./data/digest.db # Required. Per source, subject to per-source backoff after failures. POLL_INTERVAL_MINUTES=360 # Required. One window per day, named by date in this timezone. DIGEST_HOUR_LOCAL=07 # Required · secret. Summarising only. Give it its own spend limit at the provider. LLM_API_KEY=sk-...redacted # Required. Checked before every call. Refusing beats overspending. DAILY_TOKEN_CEILING=120000 # Required · secret. Delivery. The returned message identifier is stored. SMTP_URL=smtps://user:redacted@smtp.example:465 # Required. One recipient. This is a personal digest, not a list. DIGEST_TO=you@example.com # Required. Consecutive failures before a source pauses and alerts once. SOURCE_FAIL_PAUSE=5
You are building a production product version of Dailygram. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Dailygram · product brief ## Problem The personal digest pipeline is a realistic weekend build if you bring feeds or a data provider. Reliable ingestion from Instagram, X, TikTok, and LinkedIn is not a one-shot project: access restrictions, changing page formats, rate limits, and connector maintenance are the product's durable edge. ## Product outcome A digest that arrives once a day from feeds you chose, never repeats a post, never contains a line about something that does not exist, and tells you when a source has quietly stopped producing anything. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - RSS, Atom, or JSON feeds for selected profiles - OpenAI API key - SMTP credentials - always-on machine or small server ## Explicit non-goals for v1 - Any login, cookie or page scraping. - Feed discovery from a handle. - Maintained connectors for closed networks. - Multiple recipients and subscription management. - maintained ingestion across four social networks - resilience to rate limits, blocks, and page changes - automatic profile discovery and source setup - managed email delivery and monitoring ## Success criteria - A new source's first poll is confirmed to send nothing - A rewritten feed identifier is confirmed not to duplicate an item - Fabricated summary lines are confirmed dropped - A kill between accept and record is confirmed to yield one email - A silently empty source is confirmed flagged by rate ===== BRIEF.md ===== # Build brief · Dailygram The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a once-a-day email digest built from feeds you supply for the public profiles you actually follow to replace Dailygram. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with Express and one scheduler in the same process · a digest that only runs when the laptop is open is a bookmark folder - Storage: SQLite through better-sqlite3, with the digest state machine in the schema · sending twice is a database problem, so the database prevents it - Ingestion: RSS, Atom and JSON feeds that you supply, fetched with conditional requests · no logging in, no scraping, no cookie jar, and therefore nothing to get blocked - Summarising: One model call per digest over new items only, with a daily cost ledger · a chatty feed is a bill, and you want to see it before the invoice - Delivery: SMTP through a provider you already have, one message per digest · delivery reporting you would otherwise have to build ### Data model (create this before Phase 1) - sources(id, network, handle, profile_url, feed_url, feed_format, etag, last_modified, enabled, consecutive_failures, last_ok_at, paused_reason) - items(id, source_id, remote_id, canonical_url, url_hash, content_hash, title, body_text, posted_at, first_seen_at) - summaries(item_id, model, prompt_version, text, tokens_in, tokens_out, created_at) · keyed by item, so a rebuilt digest never pays twice - digests(id, window_start, window_end, state, item_count, built_at, sent_at, message_id, last_error) · state is building, ready, sending, sent, needs_check or failed - digest_items(digest_id, item_id, ordinal) - cost_ledger(day, tokens_in, tokens_out, calls) · one row per day, checked against a ceiling before every call - Identity is a ladder: remote identifier first, then canonical URL, then a hash of the text, because feeds rewrite their own identifiers and republish old posts more often than you would believe. ### Phase 1 · Sources are feeds, and feeds are checked when you add them Build: A localhost page to add, edit, pause and delete sources. Each source stores its network, handle, profile URL and feed URL, and saving fetches and parses the feed before accepting it. Poll with conditional requests, storing the validators the server returns, and back off per source after failures instead of hammering a feed that is gone. Trap: Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. Done when: Saving a source fetches and parses the feed and refuses the save with the parser's error when it fails. Polling sends the validators the server previously returned and a response saying nothing changed costs no parsing work, verified in the log. A source that fails repeatedly backs off and records the reason rather than retrying at full rate. Every network listed in the interface states whether a feed exists for it or not. Do not build yet: No deduplication, summaries, digests or email. ### Phase 2 · The same post, twice, from three angles Build: Normalise every entry to one schema. Deduplicate on the remote identifier, then the canonical URL with tracking parameters stripped, then a hash of the text. On the very first poll of a new source, record everything as seen and send none of it. Trap: The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. Done when: The first poll of a source with a large archive produces zero digest items and a populated items table, tested with a recorded feed. An item whose remote identifier changes but whose URL is unchanged is not reinserted, tested. Tracking parameters do not make two URLs distinct, tested with a link carrying campaign parameters. A post edited in place updates rather than duplicating. Do not build yet: No summaries, digests or email yet. ### Phase 3 · Summaries that cannot invent a post Build: Summarise only items new since the last digest, in one call, giving the model the item ids and requiring each output line to carry the id it came from. Drop any line whose id is not in the set you sent. Store the summary against the item so a rebuilt digest never pays again. Check the day's ledger against a ceiling before the call and refuse rather than exceeding it. Trap: Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. Done when: A line whose item id was not in the request is dropped before rendering, tested by feeding a fabricated response. Every link in the digest is built from the stored canonical URL, never from model output, verified by review. Summaries are stored per item and rebuilding a digest makes no new calls, tested. A day already at its token ceiling refuses the call and records the refusal instead of spending. Do not build yet: No sending or scheduling yet. ### Phase 4 · One digest a day, once Build: A digest covers a named window and there is one row per window, claimed by moving it to sending in its own transaction before the mail server is touched. Record the message identifier the server returns. A send whose outcome is unknown becomes needs_check, never a retry, and is settled by looking rather than by sending again. Trap: Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. Done when: Killing the process between the mail server accepting and the database recording leaves one delivered message and a row in needs_check, tested with a fake SMTP server. Two scheduler ticks in the same window produce one digest, tested by firing them concurrently. A window with no new items sends nothing and records why, rather than sending an empty message. The stored message identifier matches the one the server returned. Do not build yet: No source health alerts or archive export yet. ### Phase 5 · Knowing when a feed quietly died Build: A dashboard showing each source's last success, consecutive failures and last error, plus the next digest and a preview of exactly what would be sent. Pause a source after a stated run of failures and alert once. Export the whole archive as JSON plus a folder of Markdown. Bind to localhost and keep every secret in the environment. Trap: A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors. Done when: The dashboard shows last success, failure count and last error for every source, and a preview of the pending digest identical to what would be sent. A source failing for the stated run pauses and alerts once, tested by pointing one at a dead URL. A source that silently stops producing items is flagged by rate, tested with a feed that goes empty. The archive exports and the export is readable without this application. Do not build yet: Nothing after this. ### Out of scope (and why) - Logging into any social network, storing session cookies, or fetching pages a network's terms say you may not. A digest tool that gets your account restricted has cost more than it saved. - Discovering a feed from a handle. Most networks do not publish one, and the honest interface says so rather than pretending the source is broken. - Instagram, X, TikTok and LinkedIn as first-class integrations, which is exactly the maintained ingestion the paid product sells and the part that breaks monthly. - Multiple recipients and subscription management, which turns a personal digest into a mailing list with its own compliance work. ### README must contain - That sources are feed URLs you supply, validated on save, and that some networks publish none - That nothing here logs into a network, stores a cookie, or scrapes a page - That the first poll of a new source sends nothing, because a feed's archive is not news - That every digest line carries an item id and lines the model invented are dropped - That a digest window is claimed before the mail server is touched, so an unknown outcome is checked rather than resent ===== ARCHITECTURE.md ===== # Architecture · Dailygram ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with Express and one scheduler in the same process | a digest that only runs when the laptop is open is a bookmark folder | | Storage | SQLite through better-sqlite3, with the digest state machine in the schema | sending twice is a database problem, so the database prevents it | | Ingestion | RSS, Atom and JSON feeds that you supply, fetched with conditional requests | no logging in, no scraping, no cookie jar, and therefore nothing to get blocked | | Summarising | One model call per digest over new items only, with a daily cost ledger | a chatty feed is a bill, and you want to see it before the invoice | | Delivery | SMTP through a provider you already have, one message per digest | delivery reporting you would otherwise have to build | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Sources | Feed validation, conditional fetching and backoff. | Any feed format, added as a parser without touching the poller. | | Ingest | Normalisation and the deduplication ladder. | Any store, provided all three fallbacks stay. | | Summarise | One call per digest with required citations. | Any model, provided uncited lines are still dropped. | | Digest | The window state machine. | Not swappable for retry-on-timeout, which sends twice. | | Health | Failure counts and silent-source detection. | Any presentation, provided rate collapse is flagged as well as errors. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Sources, items, summaries and digests. The archive. - `POLL_INTERVAL_MINUTES` · required · Per source, subject to per-source backoff after failures. - `DIGEST_HOUR_LOCAL` · required · One window per day, named by date in this timezone. - `LLM_API_KEY` · required, secret · Summarising only. Give it its own spend limit at the provider. - `DAILY_TOKEN_CEILING` · required · Checked before every call. Refusing beats overspending. - `SMTP_URL` · required, secret · Delivery. The returned message identifier is stored. - `DIGEST_TO` · required · One recipient. This is a personal digest, not a list. - `SOURCE_FAIL_PAUSE` · required · Consecutive failures before a source pauses and alerts once. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Dailygram product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with Express and one scheduler in the same process, SQLite through better-sqlite3, with the digest state machine in the schema, RSS, Atom and JSON feeds that you supply, fetched with conditional requests, One model call per digest over new items only, with a daily cost ledger, SMTP through a provider you already have, one message per digest. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. - A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors. ===== MILESTONES.md ===== # Delivery milestones · Dailygram Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Sources are feeds, and feeds are checked when you add them Adding a source means giving a feed URL that is fetched and parsed there and then. ### Steps 1. Validate the feed at save time The moment to discover a dead URL is while the person is looking at it. Files: `src/sources.js`, `src/feeds/parse.js` 2. Send conditional request validators Store what the server gave you and hand it back next time. Files: `src/feeds/fetch.js`, `tests/conditional.test.js` 3. Back off per source and record why A feed that is gone should go quiet, not louder. Files: `src/poller.js` ### Done when - [ ] A bad feed URL is refused at save with the parse error - [ ] Unchanged responses skip parsing - [ ] Repeated failures back off and record a reason - [ ] The interface says which networks have no feed ### Watch out - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. ## M2 · The same post, twice, from three angles An item already seen never appears in a second digest, whatever the feed does to its identifiers. ### Steps 1. Watermark a new source on first poll Seen, not sent. This is the difference between a tool and a spam incident. Files: `src/ingest.js`, `tests/first-poll.test.js` 2. Deduplicate down a ladder, not on one key Identifier, then canonical URL, then content hash. Files: `src/dedupe.js`, `tests/dedupe.test.js` 3. Strip tracking parameters before hashing the URL Two links to the same post differ only in the campaign tag. Files: `src/canonical.js` ### Done when - [ ] A new source's first poll sends nothing - [ ] A rewritten identifier does not duplicate an item - [ ] Campaign parameters do not create a second item - [ ] An edited post updates in place ### Watch out - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. ## M3 · Summaries that cannot invent a post Every line in the digest points at an item that exists in your database. ### Steps 1. Require an item id on every line An unattributable sentence is a hallucination with good grammar. Files: `src/summarise.js`, `tests/citation.test.js` 2. Render links from the database only Model output never becomes a URL. Files: `src/render/digest.js` 3. Check the daily ledger before each call A ceiling you enforce beats a bill you discover. Files: `src/costs.js`, `tests/ceiling.test.js` ### Done when - [ ] Uncited or unknown-id lines are dropped - [ ] Links come only from stored URLs - [ ] A rebuilt digest makes no new model calls - [ ] The daily ceiling refuses rather than overspends ### Watch out - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. ## M4 · One digest a day, once A crash or a mail server timeout never produces two copies of the same digest. ### Steps 1. One row per window, claimed in a transaction The claim happens before any network call. That is the whole protection. Files: `src/digest.js`, `tests/exactly-once.test.js` 2. Record the message identifier the server returned It is the only proof that accepted actually happened. Files: `src/mailer.js` 3. Treat an unknown outcome as needs_check Never as a reason to send again. Files: `src/reconcile.js` ### Done when - [ ] A kill between accept and record yields one email - [ ] Concurrent ticks produce one digest - [ ] An empty window sends nothing and says why - [ ] The stored message identifier matches the server's ### Watch out - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. ## M5 · Knowing when a feed quietly died (production only) A broken source is visible within a day rather than showing up as a slowly emptying digest. ### Steps 1. Show every source's health in one place Last success, failure count, last error, items this week. Files: `src/views/dashboard.ejs` 2. Alert once on pause, not every tick A repeated alert is an alert nobody reads. Files: `src/health.js`, `tests/pause.test.js` 3. Flag sources whose item rate falls to zero Silence is the failure mode that does not raise an error. Files: `src/health.js` ### Done when - [ ] Every source shows last success, failures and last error - [ ] The preview matches what would be sent - [ ] A dead source pauses and alerts once - [ ] A silently empty source is flagged by rate ### Watch out - A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors. ===== OPERATIONS.md ===== # Operations · Dailygram ## Backup The database alone, since it holds sources, items, summaries and the digest history. Summaries are the expensive part and restoring without them means paying for them again. ## Restore Restore, then move any digest row in sending to needs_check before the scheduler runs, so nothing is resent on the assumption it failed. Confirm the first digest after restore covers the right window. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Paused sources, sources with zero items this week, digests in needs_check, and the day's token ledger against its ceiling. The zero-items count is the one that catches a feed returning an error page with a success status. ## Incident checklist A duplicate digest means the window row was not claimed before the send, and the digest table shows both attempts. A thin digest means a source died quietly, and the health view names it. Neither needs log archaeology. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A new source's first poll is confirmed to send nothing - [ ] A rewritten feed identifier is confirmed not to duplicate an item - [ ] Fabricated summary lines are confirmed dropped - [ ] A kill between accept and record is confirmed to yield one email - [ ] A silently empty source is confirmed flagged by rate ## Launch constraint Do not market omitted Dailygram capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Sources, items, summaries and digests. The archive. DATABASE_PATH=./data/digest.db # Required. Per source, subject to per-source backoff after failures. POLL_INTERVAL_MINUTES=360 # Required. One window per day, named by date in this timezone. DIGEST_HOUR_LOCAL=07 # Required · secret. Summarising only. Give it its own spend limit at the provider. LLM_API_KEY=sk-...redacted # Required. Checked before every call. Refusing beats overspending. DAILY_TOKEN_CEILING=120000 # Required · secret. Delivery. The returned message identifier is stored. SMTP_URL=smtps://user:redacted@smtp.example:465 # Required. One recipient. This is a personal digest, not a list. DIGEST_TO=you@example.com # Required. Consecutive failures before a source pauses and alerts once. SOURCE_FAIL_PAUSE=5
README.md
# Dailygram · indie build A personal digest pipeline whose hard problems are deduplicating items across feeds that rewrite their own identifiers, summarising without inventing anything, and sending exactly one digest a day even when the mail server times out. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with Express and one scheduler in the same process | a digest that only runs when the laptop is open is a bookmark folder | | Storage | SQLite through better-sqlite3, with the digest state machine in the schema | sending twice is a database problem, so the database prevents it | | Ingestion | RSS, Atom and JSON feeds that you supply, fetched with conditional requests | no logging in, no scraping, no cookie jar, and therefore nothing to get blocked | | Summarising | One model call per digest over new items only, with a daily cost ledger | a chatty feed is a bill, and you want to see it before the invoice | | Delivery | SMTP through a provider you already have, one message per digest | delivery reporting you would otherwise have to build | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the poller, the scheduler and the localhost dashboard in one process. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A machine that stays on** · a few pounds a month, or free if you own the box - Why: The digest fires on a schedule, and a laptop that sleeps turns a daily digest into a weekly surprise. - Get it: A small server, a home box or a single-board computer. - Verify: A digest scheduled for seven in the morning has arrived by breakfast - [ ] **Working feed URLs for the profiles you care about** · free, or whatever a feed service charges - Why: This is the actual constraint on the whole product, and discovering that a network publishes none is better done before you build than after. - Get it: Check each profile for an RSS, Atom or JSON feed, or a service that publishes one for it within that network's terms. - Verify: Each URL parses and returns recent entries - [ ] **An API key for a language model** · pennies a day at a few dozen items - Why: Summarising is the only external cost and it needs a ceiling from day one. - Get it: Create a key with its own spend limit rather than reusing one you use elsewhere. - Verify: A one item summary returns and the ledger records the tokens - [ ] **SMTP credentials for a provider you already use** · free tier is ample for one message a day - Why: Sending your own mail from a home address is a separate project in deliverability. - Get it: Any transactional provider with a verified sending domain. - Verify: A test digest arrives and the provider reports it accepted ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Any login, cookie or page scraping. - Feed discovery from a handle. - Maintained connectors for closed networks. - Multiple recipients and subscription management. - maintained ingestion across four social networks - resilience to rate limits, blocks, and page changes - automatic profile discovery and source setup - managed email delivery and monitoring If one of those is essential to you, that is the reason to keep paying for Dailygram, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Dailygram The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a once-a-day email digest built from feeds you supply for the public profiles you actually follow to replace Dailygram. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with Express and one scheduler in the same process · a digest that only runs when the laptop is open is a bookmark folder - Storage: SQLite through better-sqlite3, with the digest state machine in the schema · sending twice is a database problem, so the database prevents it - Ingestion: RSS, Atom and JSON feeds that you supply, fetched with conditional requests · no logging in, no scraping, no cookie jar, and therefore nothing to get blocked - Summarising: One model call per digest over new items only, with a daily cost ledger · a chatty feed is a bill, and you want to see it before the invoice - Delivery: SMTP through a provider you already have, one message per digest · delivery reporting you would otherwise have to build ### Data model (create this before Phase 1) - sources(id, network, handle, profile_url, feed_url, feed_format, etag, last_modified, enabled, consecutive_failures, last_ok_at, paused_reason) - items(id, source_id, remote_id, canonical_url, url_hash, content_hash, title, body_text, posted_at, first_seen_at) - summaries(item_id, model, prompt_version, text, tokens_in, tokens_out, created_at) · keyed by item, so a rebuilt digest never pays twice - digests(id, window_start, window_end, state, item_count, built_at, sent_at, message_id, last_error) · state is building, ready, sending, sent, needs_check or failed - digest_items(digest_id, item_id, ordinal) - cost_ledger(day, tokens_in, tokens_out, calls) · one row per day, checked against a ceiling before every call - Identity is a ladder: remote identifier first, then canonical URL, then a hash of the text, because feeds rewrite their own identifiers and republish old posts more often than you would believe. ### Phase 1 · Sources are feeds, and feeds are checked when you add them Build: A localhost page to add, edit, pause and delete sources. Each source stores its network, handle, profile URL and feed URL, and saving fetches and parses the feed before accepting it. Poll with conditional requests, storing the validators the server returns, and back off per source after failures instead of hammering a feed that is gone. Trap: Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. Done when: Saving a source fetches and parses the feed and refuses the save with the parser's error when it fails. Polling sends the validators the server previously returned and a response saying nothing changed costs no parsing work, verified in the log. A source that fails repeatedly backs off and records the reason rather than retrying at full rate. Every network listed in the interface states whether a feed exists for it or not. Do not build yet: No deduplication, summaries, digests or email. ### Phase 2 · The same post, twice, from three angles Build: Normalise every entry to one schema. Deduplicate on the remote identifier, then the canonical URL with tracking parameters stripped, then a hash of the text. On the very first poll of a new source, record everything as seen and send none of it. Trap: The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. Done when: The first poll of a source with a large archive produces zero digest items and a populated items table, tested with a recorded feed. An item whose remote identifier changes but whose URL is unchanged is not reinserted, tested. Tracking parameters do not make two URLs distinct, tested with a link carrying campaign parameters. A post edited in place updates rather than duplicating. Do not build yet: No summaries, digests or email yet. ### Phase 3 · Summaries that cannot invent a post Build: Summarise only items new since the last digest, in one call, giving the model the item ids and requiring each output line to carry the id it came from. Drop any line whose id is not in the set you sent. Store the summary against the item so a rebuilt digest never pays again. Check the day's ledger against a ceiling before the call and refuse rather than exceeding it. Trap: Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. Done when: A line whose item id was not in the request is dropped before rendering, tested by feeding a fabricated response. Every link in the digest is built from the stored canonical URL, never from model output, verified by review. Summaries are stored per item and rebuilding a digest makes no new calls, tested. A day already at its token ceiling refuses the call and records the refusal instead of spending. Do not build yet: No sending or scheduling yet. ### Phase 4 · One digest a day, once Build: A digest covers a named window and there is one row per window, claimed by moving it to sending in its own transaction before the mail server is touched. Record the message identifier the server returns. A send whose outcome is unknown becomes needs_check, never a retry, and is settled by looking rather than by sending again. Trap: Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. Done when: Killing the process between the mail server accepting and the database recording leaves one delivered message and a row in needs_check, tested with a fake SMTP server. Two scheduler ticks in the same window produce one digest, tested by firing them concurrently. A window with no new items sends nothing and records why, rather than sending an empty message. The stored message identifier matches the one the server returned. Do not build yet: No source health alerts or archive export yet. ### Phase 5 · Knowing when a feed quietly died Build: A dashboard showing each source's last success, consecutive failures and last error, plus the next digest and a preview of exactly what would be sent. Pause a source after a stated run of failures and alert once. Export the whole archive as JSON plus a folder of Markdown. Bind to localhost and keep every secret in the environment. Trap: A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors. Done when: The dashboard shows last success, failure count and last error for every source, and a preview of the pending digest identical to what would be sent. A source failing for the stated run pauses and alerts once, tested by pointing one at a dead URL. A source that silently stops producing items is flagged by rate, tested with a feed that goes empty. The archive exports and the export is readable without this application. Do not build yet: Nothing after this. ### Out of scope (and why) - Logging into any social network, storing session cookies, or fetching pages a network's terms say you may not. A digest tool that gets your account restricted has cost more than it saved. - Discovering a feed from a handle. Most networks do not publish one, and the honest interface says so rather than pretending the source is broken. - Instagram, X, TikTok and LinkedIn as first-class integrations, which is exactly the maintained ingestion the paid product sells and the part that breaks monthly. - Multiple recipients and subscription management, which turns a personal digest into a mailing list with its own compliance work. ### README must contain - That sources are feed URLs you supply, validated on save, and that some networks publish none - That nothing here logs into a network, stores a cookie, or scrapes a page - That the first poll of a new source sends nothing, because a feed's archive is not news - That every digest line carries an item id and lines the model invented are dropped - That a digest window is claimed before the mail server is touched, so an unknown outcome is checked rather than resent
AGENTS.md
# Agent instructions · Dailygram indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with Express and one scheduler in the same process, SQLite through better-sqlite3, with the digest state machine in the schema, RSS, Atom and JSON feeds that you supply, fetched with conditional requests, One model call per digest over new items only, with a daily cost ledger, SMTP through a provider you already have, one message per digest. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted.
BUILD_PLAN.md
# Build plan · Dailygram A personal digest pipeline whose hard problems are deduplicating items across feeds that rewrite their own identifiers, summarising without inventing anything, and sending exactly one digest a day even when the mail server times out. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sources are feeds, and feeds are checked when you add them Adding a source means giving a feed URL that is fetched and parsed there and then. ### Steps 1. Validate the feed at save time The moment to discover a dead URL is while the person is looking at it. Files: `src/sources.js`, `src/feeds/parse.js` 2. Send conditional request validators Store what the server gave you and hand it back next time. Files: `src/feeds/fetch.js`, `tests/conditional.test.js` 3. Back off per source and record why A feed that is gone should go quiet, not louder. Files: `src/poller.js` ### Done when - [ ] A bad feed URL is refused at save with the parse error - [ ] Unchanged responses skip parsing - [ ] Repeated failures back off and record a reason - [ ] The interface says which networks have no feed ### Watch out - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. ## Phase 2 · The same post, twice, from three angles An item already seen never appears in a second digest, whatever the feed does to its identifiers. ### Steps 1. Watermark a new source on first poll Seen, not sent. This is the difference between a tool and a spam incident. Files: `src/ingest.js`, `tests/first-poll.test.js` 2. Deduplicate down a ladder, not on one key Identifier, then canonical URL, then content hash. Files: `src/dedupe.js`, `tests/dedupe.test.js` 3. Strip tracking parameters before hashing the URL Two links to the same post differ only in the campaign tag. Files: `src/canonical.js` ### Done when - [ ] A new source's first poll sends nothing - [ ] A rewritten identifier does not duplicate an item - [ ] Campaign parameters do not create a second item - [ ] An edited post updates in place ### Watch out - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. ## Phase 3 · Summaries that cannot invent a post Every line in the digest points at an item that exists in your database. ### Steps 1. Require an item id on every line An unattributable sentence is a hallucination with good grammar. Files: `src/summarise.js`, `tests/citation.test.js` 2. Render links from the database only Model output never becomes a URL. Files: `src/render/digest.js` 3. Check the daily ledger before each call A ceiling you enforce beats a bill you discover. Files: `src/costs.js`, `tests/ceiling.test.js` ### Done when - [ ] Uncited or unknown-id lines are dropped - [ ] Links come only from stored URLs - [ ] A rebuilt digest makes no new model calls - [ ] The daily ceiling refuses rather than overspends ### Watch out - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. ## Phase 4 · One digest a day, once A crash or a mail server timeout never produces two copies of the same digest. ### Steps 1. One row per window, claimed in a transaction The claim happens before any network call. That is the whole protection. Files: `src/digest.js`, `tests/exactly-once.test.js` 2. Record the message identifier the server returned It is the only proof that accepted actually happened. Files: `src/mailer.js` 3. Treat an unknown outcome as needs_check Never as a reason to send again. Files: `src/reconcile.js` ### Done when - [ ] A kill between accept and record yields one email - [ ] Concurrent ticks produce one digest - [ ] An empty window sends nothing and says why - [ ] The stored message identifier matches the server's ### Watch out - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. ## Not in this build - Any login, cookie or page scraping. - Feed discovery from a handle. - Maintained connectors for closed networks. - Multiple recipients and subscription management. ## After v1, if you want it - A weekly digest alongside the daily one, reusing the same window machinery - Per-source importance so a quiet account is never crowded out by a busy one - Full text search over the archive, which you already store item by item
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Sources, items, summaries and digests. The archive. DATABASE_PATH=./data/digest.db # Required. Per source, subject to per-source backoff after failures. POLL_INTERVAL_MINUTES=360 # Required. One window per day, named by date in this timezone. DIGEST_HOUR_LOCAL=07 # Required · secret. Summarising only. Give it its own spend limit at the provider. LLM_API_KEY=sk-...redacted # Required. Checked before every call. Refusing beats overspending. DAILY_TOKEN_CEILING=120000 # Required · secret. Delivery. The returned message identifier is stored. SMTP_URL=smtps://user:redacted@smtp.example:465 # Required. One recipient. This is a personal digest, not a list. DIGEST_TO=you@example.com # Required. Consecutive failures before a source pauses and alerts once. SOURCE_FAIL_PAUSE=5
PRODUCT.md
# Dailygram · product brief ## Problem The personal digest pipeline is a realistic weekend build if you bring feeds or a data provider. Reliable ingestion from Instagram, X, TikTok, and LinkedIn is not a one-shot project: access restrictions, changing page formats, rate limits, and connector maintenance are the product's durable edge. ## Product outcome A digest that arrives once a day from feeds you chose, never repeats a post, never contains a line about something that does not exist, and tells you when a source has quietly stopped producing anything. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - RSS, Atom, or JSON feeds for selected profiles - OpenAI API key - SMTP credentials - always-on machine or small server ## Explicit non-goals for v1 - Any login, cookie or page scraping. - Feed discovery from a handle. - Maintained connectors for closed networks. - Multiple recipients and subscription management. - maintained ingestion across four social networks - resilience to rate limits, blocks, and page changes - automatic profile discovery and source setup - managed email delivery and monitoring ## Success criteria - A new source's first poll is confirmed to send nothing - A rewritten feed identifier is confirmed not to duplicate an item - Fabricated summary lines are confirmed dropped - A kill between accept and record is confirmed to yield one email - A silently empty source is confirmed flagged by rate
BRIEF.md
# Build brief · Dailygram The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a once-a-day email digest built from feeds you supply for the public profiles you actually follow to replace Dailygram. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22 with Express and one scheduler in the same process · a digest that only runs when the laptop is open is a bookmark folder - Storage: SQLite through better-sqlite3, with the digest state machine in the schema · sending twice is a database problem, so the database prevents it - Ingestion: RSS, Atom and JSON feeds that you supply, fetched with conditional requests · no logging in, no scraping, no cookie jar, and therefore nothing to get blocked - Summarising: One model call per digest over new items only, with a daily cost ledger · a chatty feed is a bill, and you want to see it before the invoice - Delivery: SMTP through a provider you already have, one message per digest · delivery reporting you would otherwise have to build ### Data model (create this before Phase 1) - sources(id, network, handle, profile_url, feed_url, feed_format, etag, last_modified, enabled, consecutive_failures, last_ok_at, paused_reason) - items(id, source_id, remote_id, canonical_url, url_hash, content_hash, title, body_text, posted_at, first_seen_at) - summaries(item_id, model, prompt_version, text, tokens_in, tokens_out, created_at) · keyed by item, so a rebuilt digest never pays twice - digests(id, window_start, window_end, state, item_count, built_at, sent_at, message_id, last_error) · state is building, ready, sending, sent, needs_check or failed - digest_items(digest_id, item_id, ordinal) - cost_ledger(day, tokens_in, tokens_out, calls) · one row per day, checked against a ceiling before every call - Identity is a ladder: remote identifier first, then canonical URL, then a hash of the text, because feeds rewrite their own identifiers and republish old posts more often than you would believe. ### Phase 1 · Sources are feeds, and feeds are checked when you add them Build: A localhost page to add, edit, pause and delete sources. Each source stores its network, handle, profile URL and feed URL, and saving fetches and parses the feed before accepting it. Poll with conditional requests, storing the validators the server returns, and back off per source after failures instead of hammering a feed that is gone. Trap: Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. Done when: Saving a source fetches and parses the feed and refuses the save with the parser's error when it fails. Polling sends the validators the server previously returned and a response saying nothing changed costs no parsing work, verified in the log. A source that fails repeatedly backs off and records the reason rather than retrying at full rate. Every network listed in the interface states whether a feed exists for it or not. Do not build yet: No deduplication, summaries, digests or email. ### Phase 2 · The same post, twice, from three angles Build: Normalise every entry to one schema. Deduplicate on the remote identifier, then the canonical URL with tracking parameters stripped, then a hash of the text. On the very first poll of a new source, record everything as seen and send none of it. Trap: The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. Done when: The first poll of a source with a large archive produces zero digest items and a populated items table, tested with a recorded feed. An item whose remote identifier changes but whose URL is unchanged is not reinserted, tested. Tracking parameters do not make two URLs distinct, tested with a link carrying campaign parameters. A post edited in place updates rather than duplicating. Do not build yet: No summaries, digests or email yet. ### Phase 3 · Summaries that cannot invent a post Build: Summarise only items new since the last digest, in one call, giving the model the item ids and requiring each output line to carry the id it came from. Drop any line whose id is not in the set you sent. Store the summary against the item so a rebuilt digest never pays again. Check the day's ledger against a ceiling before the call and refuse rather than exceeding it. Trap: Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. Done when: A line whose item id was not in the request is dropped before rendering, tested by feeding a fabricated response. Every link in the digest is built from the stored canonical URL, never from model output, verified by review. Summaries are stored per item and rebuilding a digest makes no new calls, tested. A day already at its token ceiling refuses the call and records the refusal instead of spending. Do not build yet: No sending or scheduling yet. ### Phase 4 · One digest a day, once Build: A digest covers a named window and there is one row per window, claimed by moving it to sending in its own transaction before the mail server is touched. Record the message identifier the server returns. A send whose outcome is unknown becomes needs_check, never a retry, and is settled by looking rather than by sending again. Trap: Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. Done when: Killing the process between the mail server accepting and the database recording leaves one delivered message and a row in needs_check, tested with a fake SMTP server. Two scheduler ticks in the same window produce one digest, tested by firing them concurrently. A window with no new items sends nothing and records why, rather than sending an empty message. The stored message identifier matches the one the server returned. Do not build yet: No source health alerts or archive export yet. ### Phase 5 · Knowing when a feed quietly died Build: A dashboard showing each source's last success, consecutive failures and last error, plus the next digest and a preview of exactly what would be sent. Pause a source after a stated run of failures and alert once. Export the whole archive as JSON plus a folder of Markdown. Bind to localhost and keep every secret in the environment. Trap: A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors. Done when: The dashboard shows last success, failure count and last error for every source, and a preview of the pending digest identical to what would be sent. A source failing for the stated run pauses and alerts once, tested by pointing one at a dead URL. A source that silently stops producing items is flagged by rate, tested with a feed that goes empty. The archive exports and the export is readable without this application. Do not build yet: Nothing after this. ### Out of scope (and why) - Logging into any social network, storing session cookies, or fetching pages a network's terms say you may not. A digest tool that gets your account restricted has cost more than it saved. - Discovering a feed from a handle. Most networks do not publish one, and the honest interface says so rather than pretending the source is broken. - Instagram, X, TikTok and LinkedIn as first-class integrations, which is exactly the maintained ingestion the paid product sells and the part that breaks monthly. - Multiple recipients and subscription management, which turns a personal digest into a mailing list with its own compliance work. ### README must contain - That sources are feed URLs you supply, validated on save, and that some networks publish none - That nothing here logs into a network, stores a cookie, or scrapes a page - That the first poll of a new source sends nothing, because a feed's archive is not news - That every digest line carries an item id and lines the model invented are dropped - That a digest window is claimed before the mail server is touched, so an unknown outcome is checked rather than resent
ARCHITECTURE.md
# Architecture · Dailygram ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with Express and one scheduler in the same process | a digest that only runs when the laptop is open is a bookmark folder | | Storage | SQLite through better-sqlite3, with the digest state machine in the schema | sending twice is a database problem, so the database prevents it | | Ingestion | RSS, Atom and JSON feeds that you supply, fetched with conditional requests | no logging in, no scraping, no cookie jar, and therefore nothing to get blocked | | Summarising | One model call per digest over new items only, with a daily cost ledger | a chatty feed is a bill, and you want to see it before the invoice | | Delivery | SMTP through a provider you already have, one message per digest | delivery reporting you would otherwise have to build | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Sources | Feed validation, conditional fetching and backoff. | Any feed format, added as a parser without touching the poller. | | Ingest | Normalisation and the deduplication ladder. | Any store, provided all three fallbacks stay. | | Summarise | One call per digest with required citations. | Any model, provided uncited lines are still dropped. | | Digest | The window state machine. | Not swappable for retry-on-timeout, which sends twice. | | Health | Failure counts and silent-source detection. | Any presentation, provided rate collapse is flagged as well as errors. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Sources, items, summaries and digests. The archive. - `POLL_INTERVAL_MINUTES` · required · Per source, subject to per-source backoff after failures. - `DIGEST_HOUR_LOCAL` · required · One window per day, named by date in this timezone. - `LLM_API_KEY` · required, secret · Summarising only. Give it its own spend limit at the provider. - `DAILY_TOKEN_CEILING` · required · Checked before every call. Refusing beats overspending. - `SMTP_URL` · required, secret · Delivery. The returned message identifier is stored. - `DIGEST_TO` · required · One recipient. This is a personal digest, not a list. - `SOURCE_FAIL_PAUSE` · required · Consecutive failures before a source pauses and alerts once. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Dailygram product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with Express and one scheduler in the same process, SQLite through better-sqlite3, with the digest state machine in the schema, RSS, Atom and JSON feeds that you supply, fetched with conditional requests, One model call per digest over new items only, with a daily cost ledger, SMTP through a provider you already have, one message per digest. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. - A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors.
MILESTONES.md
# Delivery milestones · Dailygram Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Sources are feeds, and feeds are checked when you add them Adding a source means giving a feed URL that is fetched and parsed there and then. ### Steps 1. Validate the feed at save time The moment to discover a dead URL is while the person is looking at it. Files: `src/sources.js`, `src/feeds/parse.js` 2. Send conditional request validators Store what the server gave you and hand it back next time. Files: `src/feeds/fetch.js`, `tests/conditional.test.js` 3. Back off per source and record why A feed that is gone should go quiet, not louder. Files: `src/poller.js` ### Done when - [ ] A bad feed URL is refused at save with the parse error - [ ] Unchanged responses skip parsing - [ ] Repeated failures back off and record a reason - [ ] The interface says which networks have no feed ### Watch out - Designing the add form around a handle, on the assumption that a handle can be turned into a feed, produces a product that works for nothing. There is no general API for reading a person's public posts on most networks: Meta deprecated the Instagram Basic Display API on 4 December 2024 and its replacements cover professional accounts the app owner controls, not arbitrary public profiles. Ask for a feed URL, validate it on save, and say plainly when a network has none. ## M2 · The same post, twice, from three angles An item already seen never appears in a second digest, whatever the feed does to its identifiers. ### Steps 1. Watermark a new source on first poll Seen, not sent. This is the difference between a tool and a spam incident. Files: `src/ingest.js`, `tests/first-poll.test.js` 2. Deduplicate down a ladder, not on one key Identifier, then canonical URL, then content hash. Files: `src/dedupe.js`, `tests/dedupe.test.js` 3. Strip tracking parameters before hashing the URL Two links to the same post differ only in the campaign tag. Files: `src/canonical.js` ### Done when - [ ] A new source's first poll sends nothing - [ ] A rewritten identifier does not duplicate an item - [ ] Campaign parameters do not create a second item - [ ] An edited post updates in place ### Watch out - The first poll of a feed that carries a year of archive turns into a digest of three hundred posts, and the reaction to that email is to delete the tool. Watermark on the first poll: insert every item as already seen, and start the digest from the next one. The mirror of this trap is a feed that regenerates its identifiers on republish, which without the URL and content fallbacks resends last week as new. ## M3 · Summaries that cannot invent a post Every line in the digest points at an item that exists in your database. ### Steps 1. Require an item id on every line An unattributable sentence is a hallucination with good grammar. Files: `src/summarise.js`, `tests/citation.test.js` 2. Render links from the database only Model output never becomes a URL. Files: `src/render/digest.js` 3. Check the daily ledger before each call A ceiling you enforce beats a bill you discover. Files: `src/costs.js`, `tests/ceiling.test.js` ### Done when - [ ] Uncited or unknown-id lines are dropped - [ ] Links come only from stored URLs - [ ] A rebuilt digest makes no new model calls - [ ] The daily ceiling refuses rather than overspends ### Watch out - Asking for a nice five bullet overview and pasting the result into the email produces bullets about posts that do not exist, phrased confidently, and you only notice when you click one and there is nothing there. Require an item id per line, discard lines whose id you did not send, and render the link from your own database rather than from anything the model wrote. ## M4 · One digest a day, once A crash or a mail server timeout never produces two copies of the same digest. ### Steps 1. One row per window, claimed in a transaction The claim happens before any network call. That is the whole protection. Files: `src/digest.js`, `tests/exactly-once.test.js` 2. Record the message identifier the server returned It is the only proof that accepted actually happened. Files: `src/mailer.js` 3. Treat an unknown outcome as needs_check Never as a reason to send again. Files: `src/reconcile.js` ### Done when - [ ] A kill between accept and record yields one email - [ ] Concurrent ticks produce one digest - [ ] An empty window sends nothing and says why - [ ] The stored message identifier matches the server's ### Watch out - Retrying because the SMTP call timed out sends the digest twice whenever the server accepted the first one and the acknowledgement was lost, and the second copy arrives at four in the morning. Claim the window row first, treat an unknown outcome as a question, and only resend a digest whose row says it was never accepted. ## M5 · Knowing when a feed quietly died (production only) A broken source is visible within a day rather than showing up as a slowly emptying digest. ### Steps 1. Show every source's health in one place Last success, failure count, last error, items this week. Files: `src/views/dashboard.ejs` 2. Alert once on pause, not every tick A repeated alert is an alert nobody reads. Files: `src/health.js`, `tests/pause.test.js` 3. Flag sources whose item rate falls to zero Silence is the failure mode that does not raise an error. Files: `src/health.js` ### Done when - [ ] Every source shows last success, failures and last error - [ ] The preview matches what would be sent - [ ] A dead source pauses and alerts once - [ ] A silently empty source is flagged by rate ### Watch out - A feed that starts returning an error page with a 200 status parses as zero entries, so the source never fails, the digest just gets shorter, and three weeks later you realise you stopped hearing from the one person you cared about. Track items per source per week and flag a source whose rate falls to nothing, not only one that errors.
OPERATIONS.md
# Operations · Dailygram ## Backup The database alone, since it holds sources, items, summaries and the digest history. Summaries are the expensive part and restoring without them means paying for them again. ## Restore Restore, then move any digest row in sending to needs_check before the scheduler runs, so nothing is resent on the assumption it failed. Confirm the first digest after restore covers the right window. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Paused sources, sources with zero items this week, digests in needs_check, and the day's token ledger against its ceiling. The zero-items count is the one that catches a feed returning an error page with a success status. ## Incident checklist A duplicate digest means the window row was not claimed before the send, and the digest table shows both attempts. A thin digest means a source died quietly, and the health view names it. Neither needs log archaeology. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A new source's first poll is confirmed to send nothing - [ ] A rewritten feed identifier is confirmed not to duplicate an item - [ ] Fabricated summary lines are confirmed dropped - [ ] A kill between accept and record is confirmed to yield one email - [ ] A silently empty source is confirmed flagged by rate ## Launch constraint Do not market omitted Dailygram capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Sources, items, summaries and digests. The archive. DATABASE_PATH=./data/digest.db # Required. Per source, subject to per-source backoff after failures. POLL_INTERVAL_MINUTES=360 # Required. One window per day, named by date in this timezone. DIGEST_HOUR_LOCAL=07 # Required · secret. Summarising only. Give it its own spend limit at the provider. LLM_API_KEY=sk-...redacted # Required. Checked before every call. Refusing beats overspending. DAILY_TOKEN_CEILING=120000 # Required · secret. Delivery. The returned message identifier is stored. SMTP_URL=smtps://user:redacted@smtp.example:465 # Required. One recipient. This is a personal digest, not a list. DIGEST_TO=you@example.com # Required. Consecutive failures before a source pauses and alerts once. SOURCE_FAIL_PAUSE=5
$ choose a build depth, inspect the files, then open the complete pack in your agent
People pay for Dailygram because the digest UI is the easy part. The subscription keeps public-profile ingestion working across hostile and frequently changing platforms, then runs summarization and email delivery without asking the user to maintain feeds, workers, or retries.
xmaintained ingestion across four social networks
xresilience to rate limits, blocks, and page changes
xautomatic profile discovery and source setup
xmanaged email delivery and monitoring
share on X ↗"I just replaced Dailygram ($5/mo) with one prompt"
Dailygram pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| 10 profiles | $5/workspace | — | Up to 10 monitored public social profiles. |
| unlimited | $50/workspace | — | Unlimited monitored public social profiles. |
free tierno perpetual free tier; free evaluation sends 7 total digests for up to 2 profiles
billingmonthly only; no annual plan published
hidden costsThe public 50%-off prices do not state whether the discount is permanent or when accounts return to the $10/$100 list prices.
verified 2026-08-14 · source ↗
Vibecode Dailygram
Kinda. The core of Dailygram is buildable in a weekend with the prompt on this page, but there are real gaps: maintained ingestion across four social networks, resilience to rate limits, blocks, and page changes. Read the honest list above before committing.
How much does Dailygram cost?
Dailygram costs about $5/month (10 Profiles, checked 2026-08-14), which is $60 per year.
What do I lose by replacing Dailygram?
Honestly: maintained ingestion across four social networks; resilience to rate limits, blocks, and page changes; automatic profile discovery and source setup; managed email delivery and monitoring. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Dailygram?
Yes: RSSHub (Open-source RSS generator with routes for many sites and services.), Huginn (Self-hosted agents that watch sources and build automated digests.). Using prior art is also vibecoding; the prompt is for when you want it exactly your way.