Build GetXAPI

KINDAreplaces $15/mosaves $180/yrback to the verdict

0%0 of 16 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A private X archive whose hard problems are staying inside a redistribution and update policy that makes deletion reconciliation mandatory, keeping a metered per-resource API from quietly becoming a bill, and backfilling without paying twice for the same page.

estimated effort weekendthe files for this build are in the project pack

RuntimeA Python command line plus a small local read-only web viewAccessThe official X API with your own developer credentials, no scraping and no session cookiesStorageSQLite with tombstones for removals and a metrics time seriesMeteringEvery request priced before it is sent and reconciled after it returnsReconciliationA budget-bounded sweep that re-checks the least recently verified posts

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the archiver and the local read-only view.
Get it python.org, or your version manager. open ↗
Verify python3 --version prints 3.12 or higher
accountmetered per resource returned; check the current published prices and the monthly cap before you start
Why This is the only sanctioned way to read the data, and registering is also how you end up reading the policy that shapes the schema.
Get it Register in the X developer portal and create a project with credentials scoped to reading.
Verify One authenticated read returns a post you can also see in a browser
decidefree
Why The update obligation and the distribution limit decide the tombstone design and the export shapes, and deciding them afterwards means rewriting both.
Get it Read the developer policy, write down the distribution limit, the update obligation and the date you read them.
Verify The note is committed in the repository and the export code refers to it
decidefree to decide
Why A metered API behind an unbounded loop is the only way this project can hurt you.
Get it Pick a figure you would not mind losing, divide by the current per-resource price, and size the watchlist to fit.
Verify The ceiling is set in the environment and a test proves reads stop at it

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- accounts(id, handle, user_id, added_at, backfill_done, oldest_id_seen, newest_id_seen, active)
- posts(id, author_id, created_at, text, lang, in_reply_to_id, quoted_id, first_fetched_at, last_verified_at, tombstoned_at, revision) · text is nulled on tombstone, the id and timestamps stay
- post_metrics(post_id, at, like_count, repost_count, reply_count, quote_count) · counts change, so this is a time series and never a column on posts
- api_calls(id, at, endpoint, params_hash, resources_returned, unit_price_minor, cost_minor, status, retry_of) · written before the request and completed after it
- budgets(period_start, period_end, ceiling_minor, spent_minor, blocked_at)
- reconcile_runs(id, started_at, finished_at, posts_checked, tombstoned, revised, cost_minor, budget_exhausted)
- exports(id, at, kind, row_count, ids_only) · ids_only marks the only export shape you may hand to anybody else
- There is no boolean called deleted. A tombstone is a timestamp, because you need to know when the archive learned the post was gone.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`X_BEARER_TOKEN`secret	required	`AAAA...redacted`	The only credential the archiver uses. Rotate it from the developer portal if it leaks.
`DATABASE_PATH`	required	`./data/archive.db`	Posts, metrics, call log and budgets. This file is the whole archive.
`MONTHLY_READ_CEILING`	required	`5.00`	Reads stop at this figure for the period and the archive stays readable.
`PRICE_TABLE_PATH`	required	`./config/pricing.toml`	Per-resource prices and the date you checked them. A stale date raises a warning.
`VERIFY_WITHIN_DAYS`	required	`30`	Every stored post must be re-checked inside this window or the health check fails.
`WATCHLIST_PATH`	required	`./config/watchlist.txt`	Handles to archive. Keep it short, because every handle is money.

The build, in order

1
A client that knows what a call costs
No request leaves the machine without a priced row written first and a budget that can refuse it.
1. It is the only thing standing between a loop and an invoice.
  Files src/x/client.pytests/test_no_direct_http.py
2. A crash mid-request should still leave evidence of what it spent.
  Files src/x/client.pysrc/db/schema.sql
3. Published prices change and a hard-coded figure ages in silence.
  Files config/pricing.tomlsrc/x/pricing.py
done when · tick each as it passes
watch out
- Building the fetch loop first and bolting cost tracking on afterwards means the tracking wraps a code path that has already grown four ways to make a request, and the one you forgot is the one running hourly. Make the priced wrapper the only route to the network and assert in the suite that no other module imports the HTTP library.
next phase
2
Backfill and catch-up without paying twice
A short watchlist is archived once and then kept current with the fewest reads that work.
1. The page you already paid for is the one you must not read again.
  Files src/archive/backfill.py
2. Like counts are a time series and the history is the interesting part.
  Files src/archive/metrics.pytests/test_metrics_series.py
3. Refusing a handle up front is cheaper than discovering the cost halfway.
  Files src/archive/estimate.py
done when · tick each as it passes
watch out
- A restart that begins the backfill from the top re-reads pages you have already paid for, and because the upsert leaves the database looking identical you will not notice until the statement arrives. Persist the cursor after every page, resume from it on start, and log resources read against rows newly stored so a run that paid for nothing is visible.
next phase
3
Reconciliation, because a stale archive is the liability
Stored posts are re-checked on a schedule and anything gone from X is tombstoned here.
1. Age order is what makes a bounded sweep eventually cover everything.
  Files src/archive/reconcile.py
2. You still need to know the post existed and when it went.
  Files src/archive/tombstone.pytests/test_tombstone.py
3. A health number nobody can fail is a health number nobody reads.
  Files src/archive/health.pytests/test_verification_age.py
done when · tick each as it passes
watch out
- Re-reading everything nightly is the obvious design and it becomes unaffordable the moment the archive passes a few thousand posts, so the sweep gets disabled and the archive rots into exactly the content the policy asked you to remove. Make the sweep age-ordered and budget-bounded, and treat the oldest last_verified_at as a number a test can fail on.
next phase
4
Reading your own archive, offline
Search and browsing work with the network unplugged and never trigger a paid call.
1. Offline is the test that proves reading costs nothing.
  Files src/web/views.pytests/test_offline_render.py
2. A missing row looks like a bug; a stub looks like the truth.
  Files src/web/templates/post.html
3. Confirmation with a number attached is the whole guardrail.
  Files src/web/actions.py
done when · tick each as it passes
watch out
- A detail view that quietly fetches a missing post to fill the gap turns every scroll into spend and every offline session into an error page. Keep reading strictly local, and put any fetch behind an explicit action that shows the estimated price before it runs.
5
Two exports, and which one you may shareproduct builder
Your backup is complete, anything leaving the machine is identifiers only, and the sweep keeps running after you stop watching.
1. The naming is the control; the code just enforces it.
  Files src/export/ids_only.pytests/test_ids_only.py
2. Tokens change more often than schemas do.
  Files src/x/auth.py
3. A restore nobody has run is a paragraph, not a backup.
  Files docs/restore-drill.md
done when · tick each as it passes
watch out
- One export button that produces full post text is the file that ends up shared, and the distribution limits in rule zero are about precisely that file. Ship two exports, name the identifier-only one as the shareable one in the interface and the README, and have the full export write a first line stating that it is not for redistribution.

what this build does not replace

Posting, replying and direct messages.
Any access outside the official API.
Real-time push or streaming.
Redistribution of anything beyond identifiers.
reliability: every X change is your outage until you patch it
account safety: automation flags and burns the login it runs on
write actions: posting and DMs from a scraper risk real bans
real-time push: no webhooks, you poll and you lag
scale: a few handles works, thousands is a different sport

after v1, if you want it

A thread reconstructor that stitches replies you already hold, which costs nothing because it reads no new resources
A monthly spend report by endpoint, which is usually the fastest way to find the loop you forgot
A second archive profile for your own account, where owned reads are cheaper and the backfill is worth doing properly

Need the files? The project pack on the verdict page hands your agent the whole brief · more dev tools.