Vibecode GetXAPI
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
For personal, read-only scale this is genuinely kinda: twscrape plus a throwaway account gets you tweets from a few handles in a weekend. The gaps are the product: X actively fights automated access, so the DIY version breaks whenever X changes something, burns the accounts it runs on, cannot safely post or DM, has no real-time push, and does not scale. You pay for requests that keep working next week.
the project pack5 files · written for this build, step by step
You are building a lean indie version of GetXAPI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # GetXAPI · indie build A private X archive whose hard problems are staying inside a redistribution and update policy that makes deletion reconciliation mandatory, keeping a metered per-resource API from quietly becoming a bill, and backfilling without paying twice for the same page. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python command line plus a small local read-only web view | this is a private archive, not a service, and nothing about it wants to be online | | Access | The official X API with your own developer credentials, no scraping and no session cookies | the terms prohibit scraping and the policy is what makes keeping the data defensible | | Storage | SQLite with tombstones for removals and a metrics time series | counts move and posts disappear, so neither can be one mutable column | | Metering | Every request priced before it is sent and reconciled after it returns | a metered API with no ceiling is how a hobby becomes an invoice | | Reconciliation | A budget-bounded sweep that re-checks the least recently verified posts | keeping the archive current is an obligation you accepted, not a nice extra | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the archiver and the local read-only view. - Get it: python.org, or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **An X developer account with its own API credentials** · metered per resource returned; check the current published prices and the monthly cap before you start - Why: This is the only sanctioned way to read the data, and registering is also how you end up reading the policy that shapes the schema. - Get it: Register in the X developer portal and create a project with credentials scoped to reading. - Verify: One authenticated read returns a post you can also see in a browser - [ ] **A written note of what you may keep and what you may share** · free - Why: The update obligation and the distribution limit decide the tombstone design and the export shapes, and deciding them afterwards means rewriting both. - Get it: Read the developer policy, write down the distribution limit, the update obligation and the date you read them. - Verify: The note is committed in the repository and the export code refers to it - [ ] **A monthly read ceiling in real money** · free to decide - Why: A metered API behind an unbounded loop is the only way this project can hurt you. - Get it: Pick a figure you would not mind losing, divide by the current per-resource price, and size the watchlist to fit. - Verify: The ceiling is set in the environment and a test proves reads stop at it ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Posting, replying and direct messages. - Any access outside the official API. - Real-time push or streaming. - Redistribution of anything beyond identifiers. - reliability: every X change is your outage until you patch it - account safety: automation flags and burns the login it runs on - write actions: posting and DMs from a scraper risk real bans - real-time push: no webhooks, you poll and you lag - scale: a few handles works, thousands is a different sport If one of those is essential to you, that is the reason to keep paying for GetXAPI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · GetXAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local X archive that reads through the official API, prices every call before it makes it, and reconciles deletions to replace GetXAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not scrape X, and do not build this on a throwaway account. X's terms prohibit crawling or scraping the service without prior written consent, and the developer policy is the contract that governs API access. Read that policy before the first line of code, because it decides the schema, not just a paragraph in the README. Three clauses matter here. If you provide X content to third parties you may only distribute post ids, direct message ids and user ids, and no more than 1,500,000 post ids to any entity in any 30 day period without written permission. If you store X content offline you must keep it up to date with the current state of that content on X, deleting or modifying anything that was deleted or modified there, as soon as reasonably possible or within 24 hours of a request. So an archive is not a one-way write. The reconciliation sweep in Phase 3 is the product, and a schema with no tombstone cannot comply with the clause you agreed to when you took the key. The second constraint is money. Access is metered per resource returned, so a fetch loop without a ceiling is an invoice. Check the current published prices and caps yourself before choosing a poll interval: when this was checked in September 2026, reads were priced per resource, reads of your own account were cheaper than reads of anyone else's, and pay-per-usage carried a monthly read cap. Price every call before you send it, record what it actually cost, and stop at a ceiling you chose. ### Stack (fixed, do not substitute) - Runtime: A Python command line plus a small local read-only web view · this is a private archive, not a service, and nothing about it wants to be online - Access: The official X API with your own developer credentials, no scraping and no session cookies · the terms prohibit scraping and the policy is what makes keeping the data defensible - Storage: SQLite with tombstones for removals and a metrics time series · counts move and posts disappear, so neither can be one mutable column - Metering: Every request priced before it is sent and reconciled after it returns · a metered API with no ceiling is how a hobby becomes an invoice - Reconciliation: A budget-bounded sweep that re-checks the least recently verified posts · keeping the archive current is an obligation you accepted, not a nice extra ### Data model (create this before Phase 1) - accounts(id, handle, user_id, added_at, backfill_done, oldest_id_seen, newest_id_seen, active) - posts(id, author_id, created_at, text, lang, in_reply_to_id, quoted_id, first_fetched_at, last_verified_at, tombstoned_at, revision) · text is nulled on tombstone, the id and timestamps stay - post_metrics(post_id, at, like_count, repost_count, reply_count, quote_count) · counts change, so this is a time series and never a column on posts - api_calls(id, at, endpoint, params_hash, resources_returned, unit_price_minor, cost_minor, status, retry_of) · written before the request and completed after it - budgets(period_start, period_end, ceiling_minor, spent_minor, blocked_at) - reconcile_runs(id, started_at, finished_at, posts_checked, tombstoned, revised, cost_minor, budget_exhausted) - exports(id, at, kind, row_count, ids_only) · ids_only marks the only export shape you may hand to anybody else - There is no boolean called deleted. A tombstone is a timestamp, because you need to know when the archive learned the post was gone. ### Phase 1 · A client that knows what a call costs Build: Authenticate with your own developer credentials. Wrap every endpoint in one function that writes an api_calls row with the quoted unit price before the request, sends it, then completes the row with resources returned and actual cost. Load prices from a config file carrying the date you checked them. Refuse the call when the period ceiling is reached and say so plainly. Trap: Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. Done when: No module other than the client wrapper imports the HTTP library, asserted by a test that reads the source tree. Every request has an api_calls row written before it is sent, verified by killing the process mid-request and finding an incomplete row. Reaching the period ceiling refuses further reads with a clear message while the archive stays readable, tested at the boundary. The price table carries the date it was checked and the application warns when that date is older than a configured age. Do not build yet: No watchlist, no backfill, no reconciliation, no search. ### Phase 2 · Backfill and catch-up without paying twice Build: A watchlist of handles. Backfill oldest to newest with a cursor persisted after every page, then switch to incremental fetches from the newest stored id. Upsert on post id. Append a metrics row each time rather than overwriting, so engagement history survives. Trap: A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. Done when: Killing the process mid-backfill and restarting resumes from the stored cursor rather than the top, tested. Every run logs resources read and rows newly stored, and a run that reads without storing is flagged. A repeated page updates metrics with a new row and leaves the post row unchanged apart from last_verified_at, checked in the database. Adding a handle whose backfill would exceed the remaining budget is refused before any request is made. Do not build yet: No reconciliation, no search, no export. ### Phase 3 · Reconciliation, because a stale archive is the liability Build: A sweep that selects the least recently verified posts, in batches, within a read budget of its own. A post the API no longer returns gets a tombstoned_at timestamp and its text nulled. A post that comes back changed gets its revision incremented and the new text stored. Publish the oldest last_verified_at in the archive as the health number. Trap: Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. Done when: A post the fixture API stops returning is tombstoned within one sweep, tested against a fake. A tombstone nulls the text and retains the id, author and timestamps, checked directly in the database. The sweep stops at its own budget and records that it stopped rather than silently skipping the rest. The oldest last_verified_at is displayed and the test suite fails when it exceeds the configured window. Do not build yet: No search, no dashboard, no export. ### Phase 4 · Reading your own archive, offline Build: A local page listing recent posts across accounts, a per-account view with a daily count chart, and full text search over stored text. Tombstoned posts render as a stub showing the id and when it vanished. No view calls the API: if something is not stored, the page says so. Trap: A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. Done when: Every page renders with the network disconnected, tested by running the suite with outbound requests blocked. A missing post shows an explanatory stub rather than triggering a request. Search returns stored text only and never matches the text of a tombstoned post, tested. Any action that would spend money shows its estimated cost and requires a confirmation. Do not build yet: No export, no credential rotation, no operational documentation. ### Phase 5 · Two exports, and which one you may share Build: A full export of every table as JSONL for your own backup, and a separate identifier-only export containing post ids and user ids for anything handed to anyone else. Rotate the credential without losing cursors. Document the spend, the verification age and a restore drill you have actually run. Trap: One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution. Done when: Two export commands exist and the full export writes a leading line stating it is not for redistribution, checked in the output. The identifier-only export contains no post text, no display names and no profile text, asserted by a test over its output. Rotating the credential leaves cursors, budgets and the archive intact, tested by changing the token and running a catch-up. A restore from the full export into an empty database reproduces row counts per table and the same verification age. Do not build yet: Nothing after this. ### Out of scope (and why) - Posting, replying and direct messages, which are writes against your real account and a different class of risk from reading. - Any access through a logged-in session, a browser automation or a sacrificial account, which the terms prohibit and which the rest of this build exists to avoid. - Real-time push, because streaming access is a separate product and polling fast enough to feel live is priced accordingly. - Anything commercial or redistributed beyond identifiers, which is where the published distribution limits start applying to you. ### README must contain - That access is through the official API under the developer terms, with the date those terms were read - That stored content must be kept in step with X, and that the reconciliation sweep is how this build does it - That only identifiers may be handed to third parties, with the published limit and the date it was checked - That every call is priced and metered, where the ceiling lives, and what happens when it is reached - What the oldest verification age means, where to see it, and what to do when it starts growing ===== AGENTS.md ===== # Agent instructions · GetXAPI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python command line plus a small local read-only web view, The official X API with your own developer credentials, no scraping and no session cookies, SQLite with tombstones for removals and a metrics time series, Every request priced before it is sent and reconciled after it returns, A budget-bounded sweep that re-checks the least recently verified posts. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. ===== BUILD_PLAN.md ===== # Build plan · GetXAPI A private X archive whose hard problems are staying inside a redistribution and update policy that makes deletion reconciliation mandatory, keeping a metered per-resource API from quietly becoming a bill, and backfilling without paying twice for the same page. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A client that knows what a call costs No request leaves the machine without a priced row written first and a budget that can refuse it. ### Steps 1. Put the priced wrapper in front of the network It is the only thing standing between a loop and an invoice. Files: `src/x/client.py`, `tests/test_no_direct_http.py` 2. Write the call row before the call, complete it after A crash mid-request should still leave evidence of what it spent. Files: `src/x/client.py`, `src/db/schema.sql` 3. Load prices from config with the date you checked them Published prices change and a hard-coded figure ages in silence. Files: `config/pricing.toml`, `src/x/pricing.py` ### Done when - [ ] Only the client wrapper touches the network - [ ] A call row exists before every request - [ ] The ceiling refuses reads and keeps the archive readable - [ ] A stale price-table date raises a warning ### Watch out - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. ## Phase 2 · Backfill and catch-up without paying twice A short watchlist is archived once and then kept current with the fewest reads that work. ### Steps 1. Persist the cursor after every page The page you already paid for is the one you must not read again. Files: `src/archive/backfill.py` 2. Append metrics, never overwrite them Like counts are a time series and the history is the interesting part. Files: `src/archive/metrics.py`, `tests/test_metrics_series.py` 3. Estimate a backfill before starting it Refusing a handle up front is cheaper than discovering the cost halfway. Files: `src/archive/estimate.py` ### Done when - [ ] A killed backfill resumes from its cursor - [ ] Reads and new rows are logged per run - [ ] Metrics append rather than overwrite - [ ] An unaffordable backfill is refused before any request ### Watch out - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. ## Phase 3 · Reconciliation, because a stale archive is the liability Stored posts are re-checked on a schedule and anything gone from X is tombstoned here. ### Steps 1. Sweep oldest-verified first, inside its own budget Age order is what makes a bounded sweep eventually cover everything. Files: `src/archive/reconcile.py` 2. Tombstone by nulling text and keeping the id You still need to know the post existed and when it went. Files: `src/archive/tombstone.py`, `tests/test_tombstone.py` 3. Make the oldest verification age a failing check A health number nobody can fail is a health number nobody reads. Files: `src/archive/health.py`, `tests/test_verification_age.py` ### Done when - [ ] A removed post is tombstoned within one sweep - [ ] Tombstones keep ids and timestamps, drop text - [ ] The sweep records budget exhaustion - [ ] Verification age beyond the window fails the suite ### Watch out - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. ## Phase 4 · Reading your own archive, offline Search and browsing work with the network unplugged and never trigger a paid call. ### Steps 1. Render every view from SQLite alone Offline is the test that proves reading costs nothing. Files: `src/web/views.py`, `tests/test_offline_render.py` 2. Show tombstones as stubs, not as gaps A missing row looks like a bug; a stub looks like the truth. Files: `src/web/templates/post.html` 3. Price any action that reaches the network Confirmation with a number attached is the whole guardrail. Files: `src/web/actions.py` ### Done when - [ ] All views render with the network blocked - [ ] Missing posts show a stub, not a fetch - [ ] Search never surfaces tombstoned text - [ ] Paid actions show a price and ask first ### Watch out - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. ## Not in this build - Posting, replying and direct messages. - Any access outside the official API. - Real-time push or streaming. - Redistribution of anything beyond identifiers. ## After v1, if you want it - A thread reconstructor that stitches replies you already hold, which costs nothing because it reads no new resources - A monthly spend report by endpoint, which is usually the fastest way to find the loop you forgot - A second archive profile for your own account, where owned reads are cheaper and the backfill is worth doing properly ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. The only credential the archiver uses. Rotate it from the developer portal if it leaks. X_BEARER_TOKEN=AAAA...redacted # Required. Posts, metrics, call log and budgets. This file is the whole archive. DATABASE_PATH=./data/archive.db # Required. Reads stop at this figure for the period and the archive stays readable. MONTHLY_READ_CEILING=5.00 # Required. Per-resource prices and the date you checked them. A stale date raises a warning. PRICE_TABLE_PATH=./config/pricing.toml # Required. Every stored post must be re-checked inside this window or the health check fails. VERIFY_WITHIN_DAYS=30 # Required. Handles to archive. Keep it short, because every handle is money. WATCHLIST_PATH=./config/watchlist.txt
You are building a lean indie version of GetXAPI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # GetXAPI · indie build A private X archive whose hard problems are staying inside a redistribution and update policy that makes deletion reconciliation mandatory, keeping a metered per-resource API from quietly becoming a bill, and backfilling without paying twice for the same page. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python command line plus a small local read-only web view | this is a private archive, not a service, and nothing about it wants to be online | | Access | The official X API with your own developer credentials, no scraping and no session cookies | the terms prohibit scraping and the policy is what makes keeping the data defensible | | Storage | SQLite with tombstones for removals and a metrics time series | counts move and posts disappear, so neither can be one mutable column | | Metering | Every request priced before it is sent and reconciled after it returns | a metered API with no ceiling is how a hobby becomes an invoice | | Reconciliation | A budget-bounded sweep that re-checks the least recently verified posts | keeping the archive current is an obligation you accepted, not a nice extra | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the archiver and the local read-only view. - Get it: python.org, or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **An X developer account with its own API credentials** · metered per resource returned; check the current published prices and the monthly cap before you start - Why: This is the only sanctioned way to read the data, and registering is also how you end up reading the policy that shapes the schema. - Get it: Register in the X developer portal and create a project with credentials scoped to reading. - Verify: One authenticated read returns a post you can also see in a browser - [ ] **A written note of what you may keep and what you may share** · free - Why: The update obligation and the distribution limit decide the tombstone design and the export shapes, and deciding them afterwards means rewriting both. - Get it: Read the developer policy, write down the distribution limit, the update obligation and the date you read them. - Verify: The note is committed in the repository and the export code refers to it - [ ] **A monthly read ceiling in real money** · free to decide - Why: A metered API behind an unbounded loop is the only way this project can hurt you. - Get it: Pick a figure you would not mind losing, divide by the current per-resource price, and size the watchlist to fit. - Verify: The ceiling is set in the environment and a test proves reads stop at it ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Posting, replying and direct messages. - Any access outside the official API. - Real-time push or streaming. - Redistribution of anything beyond identifiers. - reliability: every X change is your outage until you patch it - account safety: automation flags and burns the login it runs on - write actions: posting and DMs from a scraper risk real bans - real-time push: no webhooks, you poll and you lag - scale: a few handles works, thousands is a different sport If one of those is essential to you, that is the reason to keep paying for GetXAPI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · GetXAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local X archive that reads through the official API, prices every call before it makes it, and reconciles deletions to replace GetXAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not scrape X, and do not build this on a throwaway account. X's terms prohibit crawling or scraping the service without prior written consent, and the developer policy is the contract that governs API access. Read that policy before the first line of code, because it decides the schema, not just a paragraph in the README. Three clauses matter here. If you provide X content to third parties you may only distribute post ids, direct message ids and user ids, and no more than 1,500,000 post ids to any entity in any 30 day period without written permission. If you store X content offline you must keep it up to date with the current state of that content on X, deleting or modifying anything that was deleted or modified there, as soon as reasonably possible or within 24 hours of a request. So an archive is not a one-way write. The reconciliation sweep in Phase 3 is the product, and a schema with no tombstone cannot comply with the clause you agreed to when you took the key. The second constraint is money. Access is metered per resource returned, so a fetch loop without a ceiling is an invoice. Check the current published prices and caps yourself before choosing a poll interval: when this was checked in September 2026, reads were priced per resource, reads of your own account were cheaper than reads of anyone else's, and pay-per-usage carried a monthly read cap. Price every call before you send it, record what it actually cost, and stop at a ceiling you chose. ### Stack (fixed, do not substitute) - Runtime: A Python command line plus a small local read-only web view · this is a private archive, not a service, and nothing about it wants to be online - Access: The official X API with your own developer credentials, no scraping and no session cookies · the terms prohibit scraping and the policy is what makes keeping the data defensible - Storage: SQLite with tombstones for removals and a metrics time series · counts move and posts disappear, so neither can be one mutable column - Metering: Every request priced before it is sent and reconciled after it returns · a metered API with no ceiling is how a hobby becomes an invoice - Reconciliation: A budget-bounded sweep that re-checks the least recently verified posts · keeping the archive current is an obligation you accepted, not a nice extra ### Data model (create this before Phase 1) - accounts(id, handle, user_id, added_at, backfill_done, oldest_id_seen, newest_id_seen, active) - posts(id, author_id, created_at, text, lang, in_reply_to_id, quoted_id, first_fetched_at, last_verified_at, tombstoned_at, revision) · text is nulled on tombstone, the id and timestamps stay - post_metrics(post_id, at, like_count, repost_count, reply_count, quote_count) · counts change, so this is a time series and never a column on posts - api_calls(id, at, endpoint, params_hash, resources_returned, unit_price_minor, cost_minor, status, retry_of) · written before the request and completed after it - budgets(period_start, period_end, ceiling_minor, spent_minor, blocked_at) - reconcile_runs(id, started_at, finished_at, posts_checked, tombstoned, revised, cost_minor, budget_exhausted) - exports(id, at, kind, row_count, ids_only) · ids_only marks the only export shape you may hand to anybody else - There is no boolean called deleted. A tombstone is a timestamp, because you need to know when the archive learned the post was gone. ### Phase 1 · A client that knows what a call costs Build: Authenticate with your own developer credentials. Wrap every endpoint in one function that writes an api_calls row with the quoted unit price before the request, sends it, then completes the row with resources returned and actual cost. Load prices from a config file carrying the date you checked them. Refuse the call when the period ceiling is reached and say so plainly. Trap: Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. Done when: No module other than the client wrapper imports the HTTP library, asserted by a test that reads the source tree. Every request has an api_calls row written before it is sent, verified by killing the process mid-request and finding an incomplete row. Reaching the period ceiling refuses further reads with a clear message while the archive stays readable, tested at the boundary. The price table carries the date it was checked and the application warns when that date is older than a configured age. Do not build yet: No watchlist, no backfill, no reconciliation, no search. ### Phase 2 · Backfill and catch-up without paying twice Build: A watchlist of handles. Backfill oldest to newest with a cursor persisted after every page, then switch to incremental fetches from the newest stored id. Upsert on post id. Append a metrics row each time rather than overwriting, so engagement history survives. Trap: A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. Done when: Killing the process mid-backfill and restarting resumes from the stored cursor rather than the top, tested. Every run logs resources read and rows newly stored, and a run that reads without storing is flagged. A repeated page updates metrics with a new row and leaves the post row unchanged apart from last_verified_at, checked in the database. Adding a handle whose backfill would exceed the remaining budget is refused before any request is made. Do not build yet: No reconciliation, no search, no export. ### Phase 3 · Reconciliation, because a stale archive is the liability Build: A sweep that selects the least recently verified posts, in batches, within a read budget of its own. A post the API no longer returns gets a tombstoned_at timestamp and its text nulled. A post that comes back changed gets its revision incremented and the new text stored. Publish the oldest last_verified_at in the archive as the health number. Trap: Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. Done when: A post the fixture API stops returning is tombstoned within one sweep, tested against a fake. A tombstone nulls the text and retains the id, author and timestamps, checked directly in the database. The sweep stops at its own budget and records that it stopped rather than silently skipping the rest. The oldest last_verified_at is displayed and the test suite fails when it exceeds the configured window. Do not build yet: No search, no dashboard, no export. ### Phase 4 · Reading your own archive, offline Build: A local page listing recent posts across accounts, a per-account view with a daily count chart, and full text search over stored text. Tombstoned posts render as a stub showing the id and when it vanished. No view calls the API: if something is not stored, the page says so. Trap: A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. Done when: Every page renders with the network disconnected, tested by running the suite with outbound requests blocked. A missing post shows an explanatory stub rather than triggering a request. Search returns stored text only and never matches the text of a tombstoned post, tested. Any action that would spend money shows its estimated cost and requires a confirmation. Do not build yet: No export, no credential rotation, no operational documentation. ### Phase 5 · Two exports, and which one you may share Build: A full export of every table as JSONL for your own backup, and a separate identifier-only export containing post ids and user ids for anything handed to anyone else. Rotate the credential without losing cursors. Document the spend, the verification age and a restore drill you have actually run. Trap: One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution. Done when: Two export commands exist and the full export writes a leading line stating it is not for redistribution, checked in the output. The identifier-only export contains no post text, no display names and no profile text, asserted by a test over its output. Rotating the credential leaves cursors, budgets and the archive intact, tested by changing the token and running a catch-up. A restore from the full export into an empty database reproduces row counts per table and the same verification age. Do not build yet: Nothing after this. ### Out of scope (and why) - Posting, replying and direct messages, which are writes against your real account and a different class of risk from reading. - Any access through a logged-in session, a browser automation or a sacrificial account, which the terms prohibit and which the rest of this build exists to avoid. - Real-time push, because streaming access is a separate product and polling fast enough to feel live is priced accordingly. - Anything commercial or redistributed beyond identifiers, which is where the published distribution limits start applying to you. ### README must contain - That access is through the official API under the developer terms, with the date those terms were read - That stored content must be kept in step with X, and that the reconciliation sweep is how this build does it - That only identifiers may be handed to third parties, with the published limit and the date it was checked - That every call is priced and metered, where the ceiling lives, and what happens when it is reached - What the oldest verification age means, where to see it, and what to do when it starts growing ===== AGENTS.md ===== # Agent instructions · GetXAPI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python command line plus a small local read-only web view, The official X API with your own developer credentials, no scraping and no session cookies, SQLite with tombstones for removals and a metrics time series, Every request priced before it is sent and reconciled after it returns, A budget-bounded sweep that re-checks the least recently verified posts. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. ===== BUILD_PLAN.md ===== # Build plan · GetXAPI A private X archive whose hard problems are staying inside a redistribution and update policy that makes deletion reconciliation mandatory, keeping a metered per-resource API from quietly becoming a bill, and backfilling without paying twice for the same page. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A client that knows what a call costs No request leaves the machine without a priced row written first and a budget that can refuse it. ### Steps 1. Put the priced wrapper in front of the network It is the only thing standing between a loop and an invoice. Files: `src/x/client.py`, `tests/test_no_direct_http.py` 2. Write the call row before the call, complete it after A crash mid-request should still leave evidence of what it spent. Files: `src/x/client.py`, `src/db/schema.sql` 3. Load prices from config with the date you checked them Published prices change and a hard-coded figure ages in silence. Files: `config/pricing.toml`, `src/x/pricing.py` ### Done when - [ ] Only the client wrapper touches the network - [ ] A call row exists before every request - [ ] The ceiling refuses reads and keeps the archive readable - [ ] A stale price-table date raises a warning ### Watch out - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. ## Phase 2 · Backfill and catch-up without paying twice A short watchlist is archived once and then kept current with the fewest reads that work. ### Steps 1. Persist the cursor after every page The page you already paid for is the one you must not read again. Files: `src/archive/backfill.py` 2. Append metrics, never overwrite them Like counts are a time series and the history is the interesting part. Files: `src/archive/metrics.py`, `tests/test_metrics_series.py` 3. Estimate a backfill before starting it Refusing a handle up front is cheaper than discovering the cost halfway. Files: `src/archive/estimate.py` ### Done when - [ ] A killed backfill resumes from its cursor - [ ] Reads and new rows are logged per run - [ ] Metrics append rather than overwrite - [ ] An unaffordable backfill is refused before any request ### Watch out - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. ## Phase 3 · Reconciliation, because a stale archive is the liability Stored posts are re-checked on a schedule and anything gone from X is tombstoned here. ### Steps 1. Sweep oldest-verified first, inside its own budget Age order is what makes a bounded sweep eventually cover everything. Files: `src/archive/reconcile.py` 2. Tombstone by nulling text and keeping the id You still need to know the post existed and when it went. Files: `src/archive/tombstone.py`, `tests/test_tombstone.py` 3. Make the oldest verification age a failing check A health number nobody can fail is a health number nobody reads. Files: `src/archive/health.py`, `tests/test_verification_age.py` ### Done when - [ ] A removed post is tombstoned within one sweep - [ ] Tombstones keep ids and timestamps, drop text - [ ] The sweep records budget exhaustion - [ ] Verification age beyond the window fails the suite ### Watch out - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. ## Phase 4 · Reading your own archive, offline Search and browsing work with the network unplugged and never trigger a paid call. ### Steps 1. Render every view from SQLite alone Offline is the test that proves reading costs nothing. Files: `src/web/views.py`, `tests/test_offline_render.py` 2. Show tombstones as stubs, not as gaps A missing row looks like a bug; a stub looks like the truth. Files: `src/web/templates/post.html` 3. Price any action that reaches the network Confirmation with a number attached is the whole guardrail. Files: `src/web/actions.py` ### Done when - [ ] All views render with the network blocked - [ ] Missing posts show a stub, not a fetch - [ ] Search never surfaces tombstoned text - [ ] Paid actions show a price and ask first ### Watch out - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. ## Not in this build - Posting, replying and direct messages. - Any access outside the official API. - Real-time push or streaming. - Redistribution of anything beyond identifiers. ## After v1, if you want it - A thread reconstructor that stitches replies you already hold, which costs nothing because it reads no new resources - A monthly spend report by endpoint, which is usually the fastest way to find the loop you forgot - A second archive profile for your own account, where owned reads are cheaper and the backfill is worth doing properly ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. The only credential the archiver uses. Rotate it from the developer portal if it leaks. X_BEARER_TOKEN=AAAA...redacted # Required. Posts, metrics, call log and budgets. This file is the whole archive. DATABASE_PATH=./data/archive.db # Required. Reads stop at this figure for the period and the archive stays readable. MONTHLY_READ_CEILING=5.00 # Required. Per-resource prices and the date you checked them. A stale date raises a warning. PRICE_TABLE_PATH=./config/pricing.toml # Required. Every stored post must be re-checked inside this window or the health check fails. VERIFY_WITHIN_DAYS=30 # Required. Handles to archive. Keep it short, because every handle is money. WATCHLIST_PATH=./config/watchlist.txt
You are building a production product version of GetXAPI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # GetXAPI · product brief ## Problem For personal, read-only scale this is genuinely kinda: twscrape plus a throwaway account gets you tweets from a few handles in a weekend. The gaps are the product: X actively fights automated access, so the DIY version breaks whenever X changes something, burns the accounts it runs on, cannot safely post or DM, has no real-time push, and does not scale. You pay for requests that keep working next week. ## Product outcome A private archive that reads through the sanctioned interface, prices every call before making it, keeps itself in step with deletions and edits, and has exactly one export shape you are allowed to hand to somebody else. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - a throwaway X account - patience for re-login breakage ## Explicit non-goals for v1 - Posting, replying and direct messages. - Any access outside the official API. - Real-time push or streaming. - Redistribution of anything beyond identifiers. - reliability: every X change is your outage until you patch it - account safety: automation flags and burns the login it runs on - write actions: posting and DMs from a scraper risk real bans - real-time push: no webhooks, you poll and you lag - scale: a few handles works, thousands is a different sport ## Success criteria - Only the priced client is confirmed to reach the network - The ceiling is confirmed to stop reads and leave the archive readable - A removed post is confirmed tombstoned within one sweep - Every view is confirmed to render with the network blocked - The identifier-only export is confirmed to contain no text fields ===== BRIEF.md ===== # Build brief · GetXAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local X archive that reads through the official API, prices every call before it makes it, and reconciles deletions to replace GetXAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not scrape X, and do not build this on a throwaway account. X's terms prohibit crawling or scraping the service without prior written consent, and the developer policy is the contract that governs API access. Read that policy before the first line of code, because it decides the schema, not just a paragraph in the README. Three clauses matter here. If you provide X content to third parties you may only distribute post ids, direct message ids and user ids, and no more than 1,500,000 post ids to any entity in any 30 day period without written permission. If you store X content offline you must keep it up to date with the current state of that content on X, deleting or modifying anything that was deleted or modified there, as soon as reasonably possible or within 24 hours of a request. So an archive is not a one-way write. The reconciliation sweep in Phase 3 is the product, and a schema with no tombstone cannot comply with the clause you agreed to when you took the key. The second constraint is money. Access is metered per resource returned, so a fetch loop without a ceiling is an invoice. Check the current published prices and caps yourself before choosing a poll interval: when this was checked in September 2026, reads were priced per resource, reads of your own account were cheaper than reads of anyone else's, and pay-per-usage carried a monthly read cap. Price every call before you send it, record what it actually cost, and stop at a ceiling you chose. ### Stack (fixed, do not substitute) - Runtime: A Python command line plus a small local read-only web view · this is a private archive, not a service, and nothing about it wants to be online - Access: The official X API with your own developer credentials, no scraping and no session cookies · the terms prohibit scraping and the policy is what makes keeping the data defensible - Storage: SQLite with tombstones for removals and a metrics time series · counts move and posts disappear, so neither can be one mutable column - Metering: Every request priced before it is sent and reconciled after it returns · a metered API with no ceiling is how a hobby becomes an invoice - Reconciliation: A budget-bounded sweep that re-checks the least recently verified posts · keeping the archive current is an obligation you accepted, not a nice extra ### Data model (create this before Phase 1) - accounts(id, handle, user_id, added_at, backfill_done, oldest_id_seen, newest_id_seen, active) - posts(id, author_id, created_at, text, lang, in_reply_to_id, quoted_id, first_fetched_at, last_verified_at, tombstoned_at, revision) · text is nulled on tombstone, the id and timestamps stay - post_metrics(post_id, at, like_count, repost_count, reply_count, quote_count) · counts change, so this is a time series and never a column on posts - api_calls(id, at, endpoint, params_hash, resources_returned, unit_price_minor, cost_minor, status, retry_of) · written before the request and completed after it - budgets(period_start, period_end, ceiling_minor, spent_minor, blocked_at) - reconcile_runs(id, started_at, finished_at, posts_checked, tombstoned, revised, cost_minor, budget_exhausted) - exports(id, at, kind, row_count, ids_only) · ids_only marks the only export shape you may hand to anybody else - There is no boolean called deleted. A tombstone is a timestamp, because you need to know when the archive learned the post was gone. ### Phase 1 · A client that knows what a call costs Build: Authenticate with your own developer credentials. Wrap every endpoint in one function that writes an api_calls row with the quoted unit price before the request, sends it, then completes the row with resources returned and actual cost. Load prices from a config file carrying the date you checked them. Refuse the call when the period ceiling is reached and say so plainly. Trap: Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. Done when: No module other than the client wrapper imports the HTTP library, asserted by a test that reads the source tree. Every request has an api_calls row written before it is sent, verified by killing the process mid-request and finding an incomplete row. Reaching the period ceiling refuses further reads with a clear message while the archive stays readable, tested at the boundary. The price table carries the date it was checked and the application warns when that date is older than a configured age. Do not build yet: No watchlist, no backfill, no reconciliation, no search. ### Phase 2 · Backfill and catch-up without paying twice Build: A watchlist of handles. Backfill oldest to newest with a cursor persisted after every page, then switch to incremental fetches from the newest stored id. Upsert on post id. Append a metrics row each time rather than overwriting, so engagement history survives. Trap: A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. Done when: Killing the process mid-backfill and restarting resumes from the stored cursor rather than the top, tested. Every run logs resources read and rows newly stored, and a run that reads without storing is flagged. A repeated page updates metrics with a new row and leaves the post row unchanged apart from last_verified_at, checked in the database. Adding a handle whose backfill would exceed the remaining budget is refused before any request is made. Do not build yet: No reconciliation, no search, no export. ### Phase 3 · Reconciliation, because a stale archive is the liability Build: A sweep that selects the least recently verified posts, in batches, within a read budget of its own. A post the API no longer returns gets a tombstoned_at timestamp and its text nulled. A post that comes back changed gets its revision incremented and the new text stored. Publish the oldest last_verified_at in the archive as the health number. Trap: Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. Done when: A post the fixture API stops returning is tombstoned within one sweep, tested against a fake. A tombstone nulls the text and retains the id, author and timestamps, checked directly in the database. The sweep stops at its own budget and records that it stopped rather than silently skipping the rest. The oldest last_verified_at is displayed and the test suite fails when it exceeds the configured window. Do not build yet: No search, no dashboard, no export. ### Phase 4 · Reading your own archive, offline Build: A local page listing recent posts across accounts, a per-account view with a daily count chart, and full text search over stored text. Tombstoned posts render as a stub showing the id and when it vanished. No view calls the API: if something is not stored, the page says so. Trap: A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. Done when: Every page renders with the network disconnected, tested by running the suite with outbound requests blocked. A missing post shows an explanatory stub rather than triggering a request. Search returns stored text only and never matches the text of a tombstoned post, tested. Any action that would spend money shows its estimated cost and requires a confirmation. Do not build yet: No export, no credential rotation, no operational documentation. ### Phase 5 · Two exports, and which one you may share Build: A full export of every table as JSONL for your own backup, and a separate identifier-only export containing post ids and user ids for anything handed to anyone else. Rotate the credential without losing cursors. Document the spend, the verification age and a restore drill you have actually run. Trap: One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution. Done when: Two export commands exist and the full export writes a leading line stating it is not for redistribution, checked in the output. The identifier-only export contains no post text, no display names and no profile text, asserted by a test over its output. Rotating the credential leaves cursors, budgets and the archive intact, tested by changing the token and running a catch-up. A restore from the full export into an empty database reproduces row counts per table and the same verification age. Do not build yet: Nothing after this. ### Out of scope (and why) - Posting, replying and direct messages, which are writes against your real account and a different class of risk from reading. - Any access through a logged-in session, a browser automation or a sacrificial account, which the terms prohibit and which the rest of this build exists to avoid. - Real-time push, because streaming access is a separate product and polling fast enough to feel live is priced accordingly. - Anything commercial or redistributed beyond identifiers, which is where the published distribution limits start applying to you. ### README must contain - That access is through the official API under the developer terms, with the date those terms were read - That stored content must be kept in step with X, and that the reconciliation sweep is how this build does it - That only identifiers may be handed to third parties, with the published limit and the date it was checked - That every call is priced and metered, where the ceiling lives, and what happens when it is reached - What the oldest verification age means, where to see it, and what to do when it starts growing ===== ARCHITECTURE.md ===== # Architecture · GetXAPI ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python command line plus a small local read-only web view | this is a private archive, not a service, and nothing about it wants to be online | | Access | The official X API with your own developer credentials, no scraping and no session cookies | the terms prohibit scraping and the policy is what makes keeping the data defensible | | Storage | SQLite with tombstones for removals and a metrics time series | counts move and posts disappear, so neither can be one mutable column | | Metering | Every request priced before it is sent and reconciled after it returns | a metered API with no ceiling is how a hobby becomes an invoice | | Reconciliation | A budget-bounded sweep that re-checks the least recently verified posts | keeping the archive current is an obligation you accepted, not a nice extra | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Client | Authentication, pricing, metering and the ceiling. | Any HTTP layer, provided it stays the only path to the network. | | Archive | Watchlist, backfill cursors, upserts and the metrics series. | Any store, provided metrics append and cursors persist per page. | | Reconcile | Age-ordered sweeps, tombstones and revisions. | Not swappable for a nightly full re-read, which is the design that gets switched off. | | Read | Offline browsing, search and stubs for tombstones. | Any front end, provided rendering never calls the API. | | Export | The full backup and the identifier-only share. | Any format, provided the two shapes stay separate and labelled. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `X_BEARER_TOKEN` · required, secret · The only credential the archiver uses. Rotate it from the developer portal if it leaks. - `DATABASE_PATH` · required · Posts, metrics, call log and budgets. This file is the whole archive. - `MONTHLY_READ_CEILING` · required · Reads stop at this figure for the period and the archive stays readable. - `PRICE_TABLE_PATH` · required · Per-resource prices and the date you checked them. A stale date raises a warning. - `VERIFY_WITHIN_DAYS` · required · Every stored post must be re-checked inside this window or the health check fails. - `WATCHLIST_PATH` · required · Handles to archive. Keep it short, because every handle is money. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · GetXAPI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A Python command line plus a small local read-only web view, The official X API with your own developer credentials, no scraping and no session cookies, SQLite with tombstones for removals and a metrics time series, Every request priced before it is sent and reconciled after it returns, A budget-bounded sweep that re-checks the least recently verified posts. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. - One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution. ===== MILESTONES.md ===== # Delivery milestones · GetXAPI Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A client that knows what a call costs No request leaves the machine without a priced row written first and a budget that can refuse it. ### Steps 1. Put the priced wrapper in front of the network It is the only thing standing between a loop and an invoice. Files: `src/x/client.py`, `tests/test_no_direct_http.py` 2. Write the call row before the call, complete it after A crash mid-request should still leave evidence of what it spent. Files: `src/x/client.py`, `src/db/schema.sql` 3. Load prices from config with the date you checked them Published prices change and a hard-coded figure ages in silence. Files: `config/pricing.toml`, `src/x/pricing.py` ### Done when - [ ] Only the client wrapper touches the network - [ ] A call row exists before every request - [ ] The ceiling refuses reads and keeps the archive readable - [ ] A stale price-table date raises a warning ### Watch out - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. ## M2 · Backfill and catch-up without paying twice A short watchlist is archived once and then kept current with the fewest reads that work. ### Steps 1. Persist the cursor after every page The page you already paid for is the one you must not read again. Files: `src/archive/backfill.py` 2. Append metrics, never overwrite them Like counts are a time series and the history is the interesting part. Files: `src/archive/metrics.py`, `tests/test_metrics_series.py` 3. Estimate a backfill before starting it Refusing a handle up front is cheaper than discovering the cost halfway. Files: `src/archive/estimate.py` ### Done when - [ ] A killed backfill resumes from its cursor - [ ] Reads and new rows are logged per run - [ ] Metrics append rather than overwrite - [ ] An unaffordable backfill is refused before any request ### Watch out - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. ## M3 · Reconciliation, because a stale archive is the liability Stored posts are re-checked on a schedule and anything gone from X is tombstoned here. ### Steps 1. Sweep oldest-verified first, inside its own budget Age order is what makes a bounded sweep eventually cover everything. Files: `src/archive/reconcile.py` 2. Tombstone by nulling text and keeping the id You still need to know the post existed and when it went. Files: `src/archive/tombstone.py`, `tests/test_tombstone.py` 3. Make the oldest verification age a failing check A health number nobody can fail is a health number nobody reads. Files: `src/archive/health.py`, `tests/test_verification_age.py` ### Done when - [ ] A removed post is tombstoned within one sweep - [ ] Tombstones keep ids and timestamps, drop text - [ ] The sweep records budget exhaustion - [ ] Verification age beyond the window fails the suite ### Watch out - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. ## M4 · Reading your own archive, offline Search and browsing work with the network unplugged and never trigger a paid call. ### Steps 1. Render every view from SQLite alone Offline is the test that proves reading costs nothing. Files: `src/web/views.py`, `tests/test_offline_render.py` 2. Show tombstones as stubs, not as gaps A missing row looks like a bug; a stub looks like the truth. Files: `src/web/templates/post.html` 3. Price any action that reaches the network Confirmation with a number attached is the whole guardrail. Files: `src/web/actions.py` ### Done when - [ ] All views render with the network blocked - [ ] Missing posts show a stub, not a fetch - [ ] Search never surfaces tombstoned text - [ ] Paid actions show a price and ask first ### Watch out - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. ## M5 · Two exports, and which one you may share (production only) Your backup is complete, anything leaving the machine is identifiers only, and the sweep keeps running after you stop watching. ### Steps 1. Ship the identifier-only export as the shareable one The naming is the control; the code just enforces it. Files: `src/export/ids_only.py`, `tests/test_ids_only.py` 2. Rotate the credential without losing state Tokens change more often than schemas do. Files: `src/x/auth.py` 3. Run the restore drill and write down what it printed A restore nobody has run is a paragraph, not a backup. Files: `docs/restore-drill.md` ### Done when - [ ] The full export is labelled not for redistribution - [ ] The identifier export contains no text fields - [ ] Credential rotation preserves cursors and budgets - [ ] A restore reproduces row counts and verification age ### Watch out - One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution. ===== OPERATIONS.md ===== # Operations · GetXAPI ## Backup Copy the SQLite file, and keep the full JSONL export as the format-independent copy. The call log matters as much as the posts, because it is the only record of what the archive has cost you. ## Restore Restore the database, run a catch-up, and confirm row counts per table and the oldest verification age match the export. A restore that resets last_verified_at everywhere silently restarts the compliance clock. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Three numbers: spend against the period ceiling, the oldest last_verified_at in the archive, and resources read against rows newly stored per run. The third one catches a backfill that has started paying for pages it already has. ## Incident checklist The two incidents are a runaway loop and a sweep that stopped. The first shows up in spend within an hour; the second shows up as a verification age that grows for days, which is why that number is on the front page rather than in a log. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Only the priced client is confirmed to reach the network - [ ] The ceiling is confirmed to stop reads and leave the archive readable - [ ] A removed post is confirmed tombstoned within one sweep - [ ] Every view is confirmed to render with the network blocked - [ ] The identifier-only export is confirmed to contain no text fields ## Launch constraint Do not market omitted GetXAPI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. The only credential the archiver uses. Rotate it from the developer portal if it leaks. X_BEARER_TOKEN=AAAA...redacted # Required. Posts, metrics, call log and budgets. This file is the whole archive. DATABASE_PATH=./data/archive.db # Required. Reads stop at this figure for the period and the archive stays readable. MONTHLY_READ_CEILING=5.00 # Required. Per-resource prices and the date you checked them. A stale date raises a warning. PRICE_TABLE_PATH=./config/pricing.toml # Required. Every stored post must be re-checked inside this window or the health check fails. VERIFY_WITHIN_DAYS=30 # Required. Handles to archive. Keep it short, because every handle is money. WATCHLIST_PATH=./config/watchlist.txt
README.md
# GetXAPI · indie build A private X archive whose hard problems are staying inside a redistribution and update policy that makes deletion reconciliation mandatory, keeping a metered per-resource API from quietly becoming a bill, and backfilling without paying twice for the same page. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python command line plus a small local read-only web view | this is a private archive, not a service, and nothing about it wants to be online | | Access | The official X API with your own developer credentials, no scraping and no session cookies | the terms prohibit scraping and the policy is what makes keeping the data defensible | | Storage | SQLite with tombstones for removals and a metrics time series | counts move and posts disappear, so neither can be one mutable column | | Metering | Every request priced before it is sent and reconciled after it returns | a metered API with no ceiling is how a hobby becomes an invoice | | Reconciliation | A budget-bounded sweep that re-checks the least recently verified posts | keeping the archive current is an obligation you accepted, not a nice extra | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the archiver and the local read-only view. - Get it: python.org, or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **An X developer account with its own API credentials** · metered per resource returned; check the current published prices and the monthly cap before you start - Why: This is the only sanctioned way to read the data, and registering is also how you end up reading the policy that shapes the schema. - Get it: Register in the X developer portal and create a project with credentials scoped to reading. - Verify: One authenticated read returns a post you can also see in a browser - [ ] **A written note of what you may keep and what you may share** · free - Why: The update obligation and the distribution limit decide the tombstone design and the export shapes, and deciding them afterwards means rewriting both. - Get it: Read the developer policy, write down the distribution limit, the update obligation and the date you read them. - Verify: The note is committed in the repository and the export code refers to it - [ ] **A monthly read ceiling in real money** · free to decide - Why: A metered API behind an unbounded loop is the only way this project can hurt you. - Get it: Pick a figure you would not mind losing, divide by the current per-resource price, and size the watchlist to fit. - Verify: The ceiling is set in the environment and a test proves reads stop at it ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Posting, replying and direct messages. - Any access outside the official API. - Real-time push or streaming. - Redistribution of anything beyond identifiers. - reliability: every X change is your outage until you patch it - account safety: automation flags and burns the login it runs on - write actions: posting and DMs from a scraper risk real bans - real-time push: no webhooks, you poll and you lag - scale: a few handles works, thousands is a different sport If one of those is essential to you, that is the reason to keep paying for GetXAPI, and the README should say so rather than pretend.
BRIEF.md
# Build brief · GetXAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local X archive that reads through the official API, prices every call before it makes it, and reconciles deletions to replace GetXAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not scrape X, and do not build this on a throwaway account. X's terms prohibit crawling or scraping the service without prior written consent, and the developer policy is the contract that governs API access. Read that policy before the first line of code, because it decides the schema, not just a paragraph in the README. Three clauses matter here. If you provide X content to third parties you may only distribute post ids, direct message ids and user ids, and no more than 1,500,000 post ids to any entity in any 30 day period without written permission. If you store X content offline you must keep it up to date with the current state of that content on X, deleting or modifying anything that was deleted or modified there, as soon as reasonably possible or within 24 hours of a request. So an archive is not a one-way write. The reconciliation sweep in Phase 3 is the product, and a schema with no tombstone cannot comply with the clause you agreed to when you took the key. The second constraint is money. Access is metered per resource returned, so a fetch loop without a ceiling is an invoice. Check the current published prices and caps yourself before choosing a poll interval: when this was checked in September 2026, reads were priced per resource, reads of your own account were cheaper than reads of anyone else's, and pay-per-usage carried a monthly read cap. Price every call before you send it, record what it actually cost, and stop at a ceiling you chose. ### Stack (fixed, do not substitute) - Runtime: A Python command line plus a small local read-only web view · this is a private archive, not a service, and nothing about it wants to be online - Access: The official X API with your own developer credentials, no scraping and no session cookies · the terms prohibit scraping and the policy is what makes keeping the data defensible - Storage: SQLite with tombstones for removals and a metrics time series · counts move and posts disappear, so neither can be one mutable column - Metering: Every request priced before it is sent and reconciled after it returns · a metered API with no ceiling is how a hobby becomes an invoice - Reconciliation: A budget-bounded sweep that re-checks the least recently verified posts · keeping the archive current is an obligation you accepted, not a nice extra ### Data model (create this before Phase 1) - accounts(id, handle, user_id, added_at, backfill_done, oldest_id_seen, newest_id_seen, active) - posts(id, author_id, created_at, text, lang, in_reply_to_id, quoted_id, first_fetched_at, last_verified_at, tombstoned_at, revision) · text is nulled on tombstone, the id and timestamps stay - post_metrics(post_id, at, like_count, repost_count, reply_count, quote_count) · counts change, so this is a time series and never a column on posts - api_calls(id, at, endpoint, params_hash, resources_returned, unit_price_minor, cost_minor, status, retry_of) · written before the request and completed after it - budgets(period_start, period_end, ceiling_minor, spent_minor, blocked_at) - reconcile_runs(id, started_at, finished_at, posts_checked, tombstoned, revised, cost_minor, budget_exhausted) - exports(id, at, kind, row_count, ids_only) · ids_only marks the only export shape you may hand to anybody else - There is no boolean called deleted. A tombstone is a timestamp, because you need to know when the archive learned the post was gone. ### Phase 1 · A client that knows what a call costs Build: Authenticate with your own developer credentials. Wrap every endpoint in one function that writes an api_calls row with the quoted unit price before the request, sends it, then completes the row with resources returned and actual cost. Load prices from a config file carrying the date you checked them. Refuse the call when the period ceiling is reached and say so plainly. Trap: Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. Done when: No module other than the client wrapper imports the HTTP library, asserted by a test that reads the source tree. Every request has an api_calls row written before it is sent, verified by killing the process mid-request and finding an incomplete row. Reaching the period ceiling refuses further reads with a clear message while the archive stays readable, tested at the boundary. The price table carries the date it was checked and the application warns when that date is older than a configured age. Do not build yet: No watchlist, no backfill, no reconciliation, no search. ### Phase 2 · Backfill and catch-up without paying twice Build: A watchlist of handles. Backfill oldest to newest with a cursor persisted after every page, then switch to incremental fetches from the newest stored id. Upsert on post id. Append a metrics row each time rather than overwriting, so engagement history survives. Trap: A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. Done when: Killing the process mid-backfill and restarting resumes from the stored cursor rather than the top, tested. Every run logs resources read and rows newly stored, and a run that reads without storing is flagged. A repeated page updates metrics with a new row and leaves the post row unchanged apart from last_verified_at, checked in the database. Adding a handle whose backfill would exceed the remaining budget is refused before any request is made. Do not build yet: No reconciliation, no search, no export. ### Phase 3 · Reconciliation, because a stale archive is the liability Build: A sweep that selects the least recently verified posts, in batches, within a read budget of its own. A post the API no longer returns gets a tombstoned_at timestamp and its text nulled. A post that comes back changed gets its revision incremented and the new text stored. Publish the oldest last_verified_at in the archive as the health number. Trap: Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. Done when: A post the fixture API stops returning is tombstoned within one sweep, tested against a fake. A tombstone nulls the text and retains the id, author and timestamps, checked directly in the database. The sweep stops at its own budget and records that it stopped rather than silently skipping the rest. The oldest last_verified_at is displayed and the test suite fails when it exceeds the configured window. Do not build yet: No search, no dashboard, no export. ### Phase 4 · Reading your own archive, offline Build: A local page listing recent posts across accounts, a per-account view with a daily count chart, and full text search over stored text. Tombstoned posts render as a stub showing the id and when it vanished. No view calls the API: if something is not stored, the page says so. Trap: A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. Done when: Every page renders with the network disconnected, tested by running the suite with outbound requests blocked. A missing post shows an explanatory stub rather than triggering a request. Search returns stored text only and never matches the text of a tombstoned post, tested. Any action that would spend money shows its estimated cost and requires a confirmation. Do not build yet: No export, no credential rotation, no operational documentation. ### Phase 5 · Two exports, and which one you may share Build: A full export of every table as JSONL for your own backup, and a separate identifier-only export containing post ids and user ids for anything handed to anyone else. Rotate the credential without losing cursors. Document the spend, the verification age and a restore drill you have actually run. Trap: One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution. Done when: Two export commands exist and the full export writes a leading line stating it is not for redistribution, checked in the output. The identifier-only export contains no post text, no display names and no profile text, asserted by a test over its output. Rotating the credential leaves cursors, budgets and the archive intact, tested by changing the token and running a catch-up. A restore from the full export into an empty database reproduces row counts per table and the same verification age. Do not build yet: Nothing after this. ### Out of scope (and why) - Posting, replying and direct messages, which are writes against your real account and a different class of risk from reading. - Any access through a logged-in session, a browser automation or a sacrificial account, which the terms prohibit and which the rest of this build exists to avoid. - Real-time push, because streaming access is a separate product and polling fast enough to feel live is priced accordingly. - Anything commercial or redistributed beyond identifiers, which is where the published distribution limits start applying to you. ### README must contain - That access is through the official API under the developer terms, with the date those terms were read - That stored content must be kept in step with X, and that the reconciliation sweep is how this build does it - That only identifiers may be handed to third parties, with the published limit and the date it was checked - That every call is priced and metered, where the ceiling lives, and what happens when it is reached - What the oldest verification age means, where to see it, and what to do when it starts growing
AGENTS.md
# Agent instructions · GetXAPI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python command line plus a small local read-only web view, The official X API with your own developer credentials, no scraping and no session cookies, SQLite with tombstones for removals and a metrics time series, Every request priced before it is sent and reconciled after it returns, A budget-bounded sweep that re-checks the least recently verified posts. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs.
BUILD_PLAN.md
# Build plan · GetXAPI A private X archive whose hard problems are staying inside a redistribution and update policy that makes deletion reconciliation mandatory, keeping a metered per-resource API from quietly becoming a bill, and backfilling without paying twice for the same page. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A client that knows what a call costs No request leaves the machine without a priced row written first and a budget that can refuse it. ### Steps 1. Put the priced wrapper in front of the network It is the only thing standing between a loop and an invoice. Files: `src/x/client.py`, `tests/test_no_direct_http.py` 2. Write the call row before the call, complete it after A crash mid-request should still leave evidence of what it spent. Files: `src/x/client.py`, `src/db/schema.sql` 3. Load prices from config with the date you checked them Published prices change and a hard-coded figure ages in silence. Files: `config/pricing.toml`, `src/x/pricing.py` ### Done when - [ ] Only the client wrapper touches the network - [ ] A call row exists before every request - [ ] The ceiling refuses reads and keeps the archive readable - [ ] A stale price-table date raises a warning ### Watch out - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. ## Phase 2 · Backfill and catch-up without paying twice A short watchlist is archived once and then kept current with the fewest reads that work. ### Steps 1. Persist the cursor after every page The page you already paid for is the one you must not read again. Files: `src/archive/backfill.py` 2. Append metrics, never overwrite them Like counts are a time series and the history is the interesting part. Files: `src/archive/metrics.py`, `tests/test_metrics_series.py` 3. Estimate a backfill before starting it Refusing a handle up front is cheaper than discovering the cost halfway. Files: `src/archive/estimate.py` ### Done when - [ ] A killed backfill resumes from its cursor - [ ] Reads and new rows are logged per run - [ ] Metrics append rather than overwrite - [ ] An unaffordable backfill is refused before any request ### Watch out - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. ## Phase 3 · Reconciliation, because a stale archive is the liability Stored posts are re-checked on a schedule and anything gone from X is tombstoned here. ### Steps 1. Sweep oldest-verified first, inside its own budget Age order is what makes a bounded sweep eventually cover everything. Files: `src/archive/reconcile.py` 2. Tombstone by nulling text and keeping the id You still need to know the post existed and when it went. Files: `src/archive/tombstone.py`, `tests/test_tombstone.py` 3. Make the oldest verification age a failing check A health number nobody can fail is a health number nobody reads. Files: `src/archive/health.py`, `tests/test_verification_age.py` ### Done when - [ ] A removed post is tombstoned within one sweep - [ ] Tombstones keep ids and timestamps, drop text - [ ] The sweep records budget exhaustion - [ ] Verification age beyond the window fails the suite ### Watch out - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. ## Phase 4 · Reading your own archive, offline Search and browsing work with the network unplugged and never trigger a paid call. ### Steps 1. Render every view from SQLite alone Offline is the test that proves reading costs nothing. Files: `src/web/views.py`, `tests/test_offline_render.py` 2. Show tombstones as stubs, not as gaps A missing row looks like a bug; a stub looks like the truth. Files: `src/web/templates/post.html` 3. Price any action that reaches the network Confirmation with a number attached is the whole guardrail. Files: `src/web/actions.py` ### Done when - [ ] All views render with the network blocked - [ ] Missing posts show a stub, not a fetch - [ ] Search never surfaces tombstoned text - [ ] Paid actions show a price and ask first ### Watch out - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. ## Not in this build - Posting, replying and direct messages. - Any access outside the official API. - Real-time push or streaming. - Redistribution of anything beyond identifiers. ## After v1, if you want it - A thread reconstructor that stitches replies you already hold, which costs nothing because it reads no new resources - A monthly spend report by endpoint, which is usually the fastest way to find the loop you forgot - A second archive profile for your own account, where owned reads are cheaper and the backfill is worth doing properly
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. The only credential the archiver uses. Rotate it from the developer portal if it leaks. X_BEARER_TOKEN=AAAA...redacted # Required. Posts, metrics, call log and budgets. This file is the whole archive. DATABASE_PATH=./data/archive.db # Required. Reads stop at this figure for the period and the archive stays readable. MONTHLY_READ_CEILING=5.00 # Required. Per-resource prices and the date you checked them. A stale date raises a warning. PRICE_TABLE_PATH=./config/pricing.toml # Required. Every stored post must be re-checked inside this window or the health check fails. VERIFY_WITHIN_DAYS=30 # Required. Handles to archive. Keep it short, because every handle is money. WATCHLIST_PATH=./config/watchlist.txt
PRODUCT.md
# GetXAPI · product brief ## Problem For personal, read-only scale this is genuinely kinda: twscrape plus a throwaway account gets you tweets from a few handles in a weekend. The gaps are the product: X actively fights automated access, so the DIY version breaks whenever X changes something, burns the accounts it runs on, cannot safely post or DM, has no real-time push, and does not scale. You pay for requests that keep working next week. ## Product outcome A private archive that reads through the sanctioned interface, prices every call before making it, keeps itself in step with deletions and edits, and has exactly one export shape you are allowed to hand to somebody else. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - a throwaway X account - patience for re-login breakage ## Explicit non-goals for v1 - Posting, replying and direct messages. - Any access outside the official API. - Real-time push or streaming. - Redistribution of anything beyond identifiers. - reliability: every X change is your outage until you patch it - account safety: automation flags and burns the login it runs on - write actions: posting and DMs from a scraper risk real bans - real-time push: no webhooks, you poll and you lag - scale: a few handles works, thousands is a different sport ## Success criteria - Only the priced client is confirmed to reach the network - The ceiling is confirmed to stop reads and leave the archive readable - A removed post is confirmed tombstoned within one sweep - Every view is confirmed to render with the network blocked - The identifier-only export is confirmed to contain no text fields
BRIEF.md
# Build brief · GetXAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a local X archive that reads through the official API, prices every call before it makes it, and reconciles deletions to replace GetXAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not scrape X, and do not build this on a throwaway account. X's terms prohibit crawling or scraping the service without prior written consent, and the developer policy is the contract that governs API access. Read that policy before the first line of code, because it decides the schema, not just a paragraph in the README. Three clauses matter here. If you provide X content to third parties you may only distribute post ids, direct message ids and user ids, and no more than 1,500,000 post ids to any entity in any 30 day period without written permission. If you store X content offline you must keep it up to date with the current state of that content on X, deleting or modifying anything that was deleted or modified there, as soon as reasonably possible or within 24 hours of a request. So an archive is not a one-way write. The reconciliation sweep in Phase 3 is the product, and a schema with no tombstone cannot comply with the clause you agreed to when you took the key. The second constraint is money. Access is metered per resource returned, so a fetch loop without a ceiling is an invoice. Check the current published prices and caps yourself before choosing a poll interval: when this was checked in September 2026, reads were priced per resource, reads of your own account were cheaper than reads of anyone else's, and pay-per-usage carried a monthly read cap. Price every call before you send it, record what it actually cost, and stop at a ceiling you chose. ### Stack (fixed, do not substitute) - Runtime: A Python command line plus a small local read-only web view · this is a private archive, not a service, and nothing about it wants to be online - Access: The official X API with your own developer credentials, no scraping and no session cookies · the terms prohibit scraping and the policy is what makes keeping the data defensible - Storage: SQLite with tombstones for removals and a metrics time series · counts move and posts disappear, so neither can be one mutable column - Metering: Every request priced before it is sent and reconciled after it returns · a metered API with no ceiling is how a hobby becomes an invoice - Reconciliation: A budget-bounded sweep that re-checks the least recently verified posts · keeping the archive current is an obligation you accepted, not a nice extra ### Data model (create this before Phase 1) - accounts(id, handle, user_id, added_at, backfill_done, oldest_id_seen, newest_id_seen, active) - posts(id, author_id, created_at, text, lang, in_reply_to_id, quoted_id, first_fetched_at, last_verified_at, tombstoned_at, revision) · text is nulled on tombstone, the id and timestamps stay - post_metrics(post_id, at, like_count, repost_count, reply_count, quote_count) · counts change, so this is a time series and never a column on posts - api_calls(id, at, endpoint, params_hash, resources_returned, unit_price_minor, cost_minor, status, retry_of) · written before the request and completed after it - budgets(period_start, period_end, ceiling_minor, spent_minor, blocked_at) - reconcile_runs(id, started_at, finished_at, posts_checked, tombstoned, revised, cost_minor, budget_exhausted) - exports(id, at, kind, row_count, ids_only) · ids_only marks the only export shape you may hand to anybody else - There is no boolean called deleted. A tombstone is a timestamp, because you need to know when the archive learned the post was gone. ### Phase 1 · A client that knows what a call costs Build: Authenticate with your own developer credentials. Wrap every endpoint in one function that writes an api_calls row with the quoted unit price before the request, sends it, then completes the row with resources returned and actual cost. Load prices from a config file carrying the date you checked them. Refuse the call when the period ceiling is reached and say so plainly. Trap: Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. Done when: No module other than the client wrapper imports the HTTP library, asserted by a test that reads the source tree. Every request has an api_calls row written before it is sent, verified by killing the process mid-request and finding an incomplete row. Reaching the period ceiling refuses further reads with a clear message while the archive stays readable, tested at the boundary. The price table carries the date it was checked and the application warns when that date is older than a configured age. Do not build yet: No watchlist, no backfill, no reconciliation, no search. ### Phase 2 · Backfill and catch-up without paying twice Build: A watchlist of handles. Backfill oldest to newest with a cursor persisted after every page, then switch to incremental fetches from the newest stored id. Upsert on post id. Append a metrics row each time rather than overwriting, so engagement history survives. Trap: A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. Done when: Killing the process mid-backfill and restarting resumes from the stored cursor rather than the top, tested. Every run logs resources read and rows newly stored, and a run that reads without storing is flagged. A repeated page updates metrics with a new row and leaves the post row unchanged apart from last_verified_at, checked in the database. Adding a handle whose backfill would exceed the remaining budget is refused before any request is made. Do not build yet: No reconciliation, no search, no export. ### Phase 3 · Reconciliation, because a stale archive is the liability Build: A sweep that selects the least recently verified posts, in batches, within a read budget of its own. A post the API no longer returns gets a tombstoned_at timestamp and its text nulled. A post that comes back changed gets its revision incremented and the new text stored. Publish the oldest last_verified_at in the archive as the health number. Trap: Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. Done when: A post the fixture API stops returning is tombstoned within one sweep, tested against a fake. A tombstone nulls the text and retains the id, author and timestamps, checked directly in the database. The sweep stops at its own budget and records that it stopped rather than silently skipping the rest. The oldest last_verified_at is displayed and the test suite fails when it exceeds the configured window. Do not build yet: No search, no dashboard, no export. ### Phase 4 · Reading your own archive, offline Build: A local page listing recent posts across accounts, a per-account view with a daily count chart, and full text search over stored text. Tombstoned posts render as a stub showing the id and when it vanished. No view calls the API: if something is not stored, the page says so. Trap: A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. Done when: Every page renders with the network disconnected, tested by running the suite with outbound requests blocked. A missing post shows an explanatory stub rather than triggering a request. Search returns stored text only and never matches the text of a tombstoned post, tested. Any action that would spend money shows its estimated cost and requires a confirmation. Do not build yet: No export, no credential rotation, no operational documentation. ### Phase 5 · Two exports, and which one you may share Build: A full export of every table as JSONL for your own backup, and a separate identifier-only export containing post ids and user ids for anything handed to anyone else. Rotate the credential without losing cursors. Document the spend, the verification age and a restore drill you have actually run. Trap: One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution. Done when: Two export commands exist and the full export writes a leading line stating it is not for redistribution, checked in the output. The identifier-only export contains no post text, no display names and no profile text, asserted by a test over its output. Rotating the credential leaves cursors, budgets and the archive intact, tested by changing the token and running a catch-up. A restore from the full export into an empty database reproduces row counts per table and the same verification age. Do not build yet: Nothing after this. ### Out of scope (and why) - Posting, replying and direct messages, which are writes against your real account and a different class of risk from reading. - Any access through a logged-in session, a browser automation or a sacrificial account, which the terms prohibit and which the rest of this build exists to avoid. - Real-time push, because streaming access is a separate product and polling fast enough to feel live is priced accordingly. - Anything commercial or redistributed beyond identifiers, which is where the published distribution limits start applying to you. ### README must contain - That access is through the official API under the developer terms, with the date those terms were read - That stored content must be kept in step with X, and that the reconciliation sweep is how this build does it - That only identifiers may be handed to third parties, with the published limit and the date it was checked - That every call is priced and metered, where the ceiling lives, and what happens when it is reached - What the oldest verification age means, where to see it, and what to do when it starts growing
ARCHITECTURE.md
# Architecture · GetXAPI ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python command line plus a small local read-only web view | this is a private archive, not a service, and nothing about it wants to be online | | Access | The official X API with your own developer credentials, no scraping and no session cookies | the terms prohibit scraping and the policy is what makes keeping the data defensible | | Storage | SQLite with tombstones for removals and a metrics time series | counts move and posts disappear, so neither can be one mutable column | | Metering | Every request priced before it is sent and reconciled after it returns | a metered API with no ceiling is how a hobby becomes an invoice | | Reconciliation | A budget-bounded sweep that re-checks the least recently verified posts | keeping the archive current is an obligation you accepted, not a nice extra | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Client | Authentication, pricing, metering and the ceiling. | Any HTTP layer, provided it stays the only path to the network. | | Archive | Watchlist, backfill cursors, upserts and the metrics series. | Any store, provided metrics append and cursors persist per page. | | Reconcile | Age-ordered sweeps, tombstones and revisions. | Not swappable for a nightly full re-read, which is the design that gets switched off. | | Read | Offline browsing, search and stubs for tombstones. | Any front end, provided rendering never calls the API. | | Export | The full backup and the identifier-only share. | Any format, provided the two shapes stay separate and labelled. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `X_BEARER_TOKEN` · required, secret · The only credential the archiver uses. Rotate it from the developer portal if it leaks. - `DATABASE_PATH` · required · Posts, metrics, call log and budgets. This file is the whole archive. - `MONTHLY_READ_CEILING` · required · Reads stop at this figure for the period and the archive stays readable. - `PRICE_TABLE_PATH` · required · Per-resource prices and the date you checked them. A stale date raises a warning. - `VERIFY_WITHIN_DAYS` · required · Every stored post must be re-checked inside this window or the health check fails. - `WATCHLIST_PATH` · required · Handles to archive. Keep it short, because every handle is money. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · GetXAPI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A Python command line plus a small local read-only web view, The official X API with your own developer credentials, no scraping and no session cookies, SQLite with tombstones for removals and a metrics time series, Every request priced before it is sent and reconciled after it returns, A budget-bounded sweep that re-checks the least recently verified posts. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. - One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution.
MILESTONES.md
# Delivery milestones · GetXAPI Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A client that knows what a call costs No request leaves the machine without a priced row written first and a budget that can refuse it. ### Steps 1. Put the priced wrapper in front of the network It is the only thing standing between a loop and an invoice. Files: `src/x/client.py`, `tests/test_no_direct_http.py` 2. Write the call row before the call, complete it after A crash mid-request should still leave evidence of what it spent. Files: `src/x/client.py`, `src/db/schema.sql` 3. Load prices from config with the date you checked them Published prices change and a hard-coded figure ages in silence. Files: `config/pricing.toml`, `src/x/pricing.py` ### Done when - [ ] Only the client wrapper touches the network - [ ] A call row exists before every request - [ ] The ceiling refuses reads and keeps the archive readable - [ ] A stale price-table date raises a warning ### Watch out - Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library. ## M2 · Backfill and catch-up without paying twice A short watchlist is archived once and then kept current with the fewest reads that work. ### Steps 1. Persist the cursor after every page The page you already paid for is the one you must not read again. Files: `src/archive/backfill.py` 2. Append metrics, never overwrite them Like counts are a time series and the history is the interesting part. Files: `src/archive/metrics.py`, `tests/test_metrics_series.py` 3. Estimate a backfill before starting it Refusing a handle up front is cheaper than discovering the cost halfway. Files: `src/archive/estimate.py` ### Done when - [ ] A killed backfill resumes from its cursor - [ ] Reads and new rows are logged per run - [ ] Metrics append rather than overwrite - [ ] An unaffordable backfill is refused before any request ### Watch out - A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible. ## M3 · Reconciliation, because a stale archive is the liability Stored posts are re-checked on a schedule and anything gone from X is tombstoned here. ### Steps 1. Sweep oldest-verified first, inside its own budget Age order is what makes a bounded sweep eventually cover everything. Files: `src/archive/reconcile.py` 2. Tombstone by nulling text and keeping the id You still need to know the post existed and when it went. Files: `src/archive/tombstone.py`, `tests/test_tombstone.py` 3. Make the oldest verification age a failing check A health number nobody can fail is a health number nobody reads. Files: `src/archive/health.py`, `tests/test_verification_age.py` ### Done when - [ ] A removed post is tombstoned within one sweep - [ ] Tombstones keep ids and timestamps, drop text - [ ] The sweep records budget exhaustion - [ ] Verification age beyond the window fails the suite ### Watch out - Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on. ## M4 · Reading your own archive, offline Search and browsing work with the network unplugged and never trigger a paid call. ### Steps 1. Render every view from SQLite alone Offline is the test that proves reading costs nothing. Files: `src/web/views.py`, `tests/test_offline_render.py` 2. Show tombstones as stubs, not as gaps A missing row looks like a bug; a stub looks like the truth. Files: `src/web/templates/post.html` 3. Price any action that reaches the network Confirmation with a number attached is the whole guardrail. Files: `src/web/actions.py` ### Done when - [ ] All views render with the network blocked - [ ] Missing posts show a stub, not a fetch - [ ] Search never surfaces tombstoned text - [ ] Paid actions show a price and ask first ### Watch out - A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs. ## M5 · Two exports, and which one you may share (production only) Your backup is complete, anything leaving the machine is identifiers only, and the sweep keeps running after you stop watching. ### Steps 1. Ship the identifier-only export as the shareable one The naming is the control; the code just enforces it. Files: `src/export/ids_only.py`, `tests/test_ids_only.py` 2. Rotate the credential without losing state Tokens change more often than schemas do. Files: `src/x/auth.py` 3. Run the restore drill and write down what it printed A restore nobody has run is a paragraph, not a backup. Files: `docs/restore-drill.md` ### Done when - [ ] The full export is labelled not for redistribution - [ ] The identifier export contains no text fields - [ ] Credential rotation preserves cursors and budgets - [ ] A restore reproduces row counts and verification age ### Watch out - One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution.
OPERATIONS.md
# Operations · GetXAPI ## Backup Copy the SQLite file, and keep the full JSONL export as the format-independent copy. The call log matters as much as the posts, because it is the only record of what the archive has cost you. ## Restore Restore the database, run a catch-up, and confirm row counts per table and the oldest verification age match the export. A restore that resets last_verified_at everywhere silently restarts the compliance clock. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Three numbers: spend against the period ceiling, the oldest last_verified_at in the archive, and resources read against rows newly stored per run. The third one catches a backfill that has started paying for pages it already has. ## Incident checklist The two incidents are a runaway loop and a sweep that stopped. The first shows up in spend within an hour; the second shows up as a verification age that grows for days, which is why that number is on the front page rather than in a log. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Only the priced client is confirmed to reach the network - [ ] The ceiling is confirmed to stop reads and leave the archive readable - [ ] A removed post is confirmed tombstoned within one sweep - [ ] Every view is confirmed to render with the network blocked - [ ] The identifier-only export is confirmed to contain no text fields ## Launch constraint Do not market omitted GetXAPI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. The only credential the archiver uses. Rotate it from the developer portal if it leaks. X_BEARER_TOKEN=AAAA...redacted # Required. Posts, metrics, call log and budgets. This file is the whole archive. DATABASE_PATH=./data/archive.db # Required. Reads stop at this figure for the period and the archive stays readable. MONTHLY_READ_CEILING=5.00 # Required. Per-resource prices and the date you checked them. A stale date raises a warning. PRICE_TABLE_PATH=./config/pricing.toml # Required. Every stored post must be re-checked inside this window or the health check fails. VERIFY_WITHIN_DAYS=30 # Required. Handles to archive. Keep it short, because every handle is money. WATCHLIST_PATH=./config/watchlist.txt
$ choose a build depth, inspect the files, then open the complete pack in your agent
X retired meaningful free API access and priced the official tiers for enterprises, so everyone else either scrapes or pays someone to keep scraping working. The subscription is not the endpoints, it is the ops: absorbing every anti-bot change, keeping sessions alive, and answering for uptime while you build your actual product.
xreliability: every X change is your outage until you patch it
xaccount safety: automation flags and burns the login it runs on
xwrite actions: posting and DMs from a scraper risk real bans
xreal-time push: no webhooks, you poll and you lag
xscale: a few handles works, thousands is a different sport
share on X ↗"I just replaced GetXAPI ($15/mo) with one prompt"
GetXAPI pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| api free signup | $0 | $0 | $0.10 in credits, roughly 100 standard calls or 2,000 returned tweets; no card. |
| api pay as you go | custom | — | Top-ups start at $10; most reads cost $0.001/call and heavier routes cost $0.002-$0.010/call. |
| api starter | custom | — | $4.50 in usable credits, roughly 4,500 standard calls; expires after 30 days. |
| api pro | $15 | — | $30 plan credit plus $10 bonus credit, roughly 40,000 standard calls/month. |
| api growth | $49 | — | $120 plan credit plus $20 bonus credit, roughly 140,000 standard calls/month. |
| api scale | $99 | — | $300 plan credit plus $50 bonus credit, roughly 350,000 standard calls/month. |
| api custom | custom | — | Custom prepaid volume and support. |
| monitoring starter | $19 | — | Monitor up to 6 accounts; approximately 15-second polling. |
| monitoring growth | $59 | — | Monitor up to 20 accounts; approximately 2-second fast-tier polling. |
| monitoring professional | $119 | — | Monitor up to 50 accounts. |
| monitoring enterprise | $249 | — | Monitor up to 150 accounts. |
| spaces pay as you go | custom | — | $0.05 per job plus $0.015 per transcript minute. |
free tier$0.10 signup credit, roughly 100 standard calls or 2,000 returned tweets; no card; starter credit never expires
billingmonthly subscriptions plus prepaid pay-as-you-go and one-time packs; no annual subscription price published
hidden costsRoute prices vary from $0.001 to $0.010 per call; Spaces has a separate job and transcript-minute charge; subscription rollover is lost when the plan lapses.
verified 2026-08-14 · source ↗
Vibecode GetXAPI
Kinda. The core of GetXAPI is buildable in a weekend with the prompt on this page, but there are real gaps: reliability: every X change is your outage until you patch it, account safety: automation flags and burns the login it runs on. Read the honest list above before committing.
How much does GetXAPI cost?
GetXAPI costs about $15/month (Pro, checked 2026-07-31), which is $180 per year.
What do I lose by replacing GetXAPI?
Honestly: reliability: every X change is your outage until you patch it; account safety: automation flags and burns the login it runs on; write actions: posting and DMs from a scraper risk real bans; real-time push: no webhooks, you poll and you lag; scale: a few handles works, thousands is a different sport. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to GetXAPI?
Yes: twscrape (account-based X scraper, the standard DIY starting point), twitter-scraper (TypeScript X scraper, no official API). Using prior art is also vibecoding; the prompt is for when you want it exactly your way.