Build Import.io

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A personal extraction pipeline whose hard problems are politeness enforced by the protocol rather than by good intentions, noticing when a selector still matches but now returns the cookie banner, and replaying a run without refetching or double-delivering.

estimated effort multi-daythe files for this build are in the project pack

RuntimePython 3.12 with one worker process and a scheduler, no distributed queueFetchinghttpx with a per-host token bucket and conditional requestsStorageSQLite for runs and rows, compressed page snapshots on disk keyed by content hashExtractionDeclared selectors in a versioned spec file, validated against a JSON SchemaDeliveryOne sink at a time, idempotent per run and row key

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the fetcher, the extractor and the scheduler.
Get it python.org, or your package manager. open ↗
Verify python3 --version prints 3.12 or higher
decidefree
Why Everything here depends on having a target whose robots.txt permits it and whose data is not about people, and choosing that target is the decision with consequences.
Get it Read the site's robots.txt and terms, confirm the pages are public and impersonal, and write down the date you checked. open ↗
Verify A written note records the site, the rules that apply and the date
have readyfree
Why Identifying yourself is the difference between a crawler an administrator emails and one they block, and it costs nothing.
Get it An address you monitor, or a page describing the crawler.
Verify The User-Agent string contains the address and appears in the target's logs as sent
have readypennies per month
Why Replay without refetching is the whole design, and it is paid for in stored bytes that grow with every run until retention is set.
Get it A few gigabytes on the machine, or a mounted volume you can grow.
Verify The snapshot directory reports its size and an alert threshold is configured
decidefree
Why Idempotent delivery needs a stable key per row, and picking it after the first duplicate arrives means picking whatever is convenient rather than correct.
Get it Choose the sink (a file, a spreadsheet, a table) and the field or combination that uniquely identifies a row on the site.
Verify Both are in the spec and the delivery test asserts uniqueness on that key

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- sites(host, robots_fetched_at, robots_body_sha256, robots_status, rate_per_second, user_agent) · the cached exclusion file and its age
- specs(id, name, version, url_pattern, selectors_json, schema_json, created_at) · the extractor, versioned, because drift is a diff between versions
- runs(id, spec_id, started_at, finished_at, state, pages_fetched, pages_from_cache, rows_extracted, quarantined) · state is running, succeeded, quarantined or failed
- fetches(id, run_id, url, requested_at, status, etag, last_modified, bytes, snapshot_sha256, from_cache, robots_decision)
- rows(id, run_id, row_key, data_json, snapshot_sha256, valid) · row_key is declared in the spec and is what makes delivery idempotent
- deliveries(run_id, sink, row_key, delivered_at) · unique, so a replay delivers only what did not arrive
- A snapshot is stored once per content hash and referenced by every fetch that produced it, so a spec fixed after the fact is re-run against the exact bytes the site served.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_PATH`	required	`./data/pipeline.db`	Runs, fetches, rows and deliveries.
`SNAPSHOT_DIR`	required	`./data/snapshots`	Compressed pages keyed by content hash. Watch its size.
`USER_AGENT`	required	`example-extractor/1.0 (+mailto:you@example.com)`	Sent on every request. Must carry a contact address.
`DEFAULT_RATE_PER_SECOND`	required	`1`	Your own pacing, since the standard defines no delay. Lower it per host as needed.
`ROBOTS_CACHE_SECONDS`	required	`86400`	Never longer than 24 hours, which is the ceiling the standard gives.
`RUN_MAX_REQUESTS`	required	`2000`	Per run. Reaching it truncates the run rather than failing it.
`SNAPSHOT_RETENTION_DAYS`	required	`30`	The most recent snapshot per URL is kept regardless.
`DRIFT_ROW_COUNT_TOLERANCE`	required	`0.2`	Fractional change in row count that quarantines a run.

The build, in order

1
A fetcher that obeys the protocol
No request leaves the process until the exclusion rules for that host have been applied.
1. The deny-on-error branch is the one every naive crawler omits.
  Files src/fetch/robots.pytests/test_robots.py
  terminal
  python -m pytest tests/test_robots.py -q
2. The standard does not set a delay, so you set one and honour it.
  Files src/fetch/pacing.pysrc/fetch/client.py
3. So a broken extractor is fixed without touching the site again.
  Files src/store/snapshots.py
done when · tick each as it passes
watch out
- The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse.
next phase
2
Extraction from stored bytes, against a declared shape
A spec declares selectors, types and a row key, and extraction never touches the network.
1. Testable, replayable and free to run a hundred times.
  Files src/extract/run.pytests/test_extract.py
2. A type error found now is a support question avoided later.
  Files src/extract/validate.py
3. The one assertion that keeps the split honest.
  Files tests/conftest.py
  terminal
  python -m pytest tests/test_extract.py -q --disable-socket
done when · tick each as it passes
watch out
- An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly.
next phase
3
Noticing that the page changed shape
A run whose output looks wrong is quarantined before anything downstream sees it.
1. You cannot diff against a baseline you never recorded.
  Files src/quality/stats.py
2. A warning in a log is a delivery of bad data with extra steps.
  Files src/quality/gate.pytests/test_drift.py
3. This is what a cookie banner looks like in a dataset.
  Files src/quality/stats.py
done when · tick each as it passes
watch out
- The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right.
next phase
4
Delivery you can replay without duplicating
Re-running a half-delivered run sends only what did not arrive.
1. The unique index is what makes replay safe.
  Files src/deliver/sink.pytests/test_replay.py
2. Nobody presses replay confidently without one.
  Files src/deliver/dryrun.py
3. The gate is worthless if the replay button bypasses it.
  Files src/deliver/sink.py
done when · tick each as it passes
watch out
- Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat.
5
Running unattended without a surprise billproduct builder
Scheduled runs stay inside disk, time and request ceilings, and a failure is visible without reading a log.
1. An unattended crawler without a ceiling is a bill or a ban.
  Files src/schedule/limits.pytests/test_limits.py
2. Keep the most recent per URL whatever the retention says.
  Files src/store/retention.py
3. The two things you need and will not find in a log.
  Files src/web/templates/run.html
done when · tick each as it passes
watch out
- Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full.

what this build does not replace

Anything behind a login.
Personal data of any kind.
Address rotation or header spoofing.
A connector library.
hundreds of maintained connectors
OAuth app verification
durable execution at scale
schema drift handling and enterprise controls

after v1, if you want it

A second sink, which the delivery table already supports and which needs only a module
Change alerts per field, once the per-field statistics have a few weeks of history behind them
Rendering pages that need scripting, which is a different fetcher and should stay behind the same robots gate

Need the files? The project pack on the verdict page hands your agent the whole brief · more automation.