Vibecode Import.io
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The visible enterprise web data loop is buildable, but a credible replacement needs more than the first screen. Import.io earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Import.io. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Import.io · indie build A personal extraction pipeline whose hard problems are politeness enforced by the protocol rather than by good intentions, noticing when a selector still matches but now returns the cookie banner, and replaying a run without refetching or double-delivering. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with one worker process and a scheduler, no distributed queue | durable execution at scale is the part you are deliberately not rebuilding | | Fetching | httpx with a per-host token bucket and conditional requests | pacing and caching are the whole of being a good citizen | | Storage | SQLite for runs and rows, compressed page snapshots on disk keyed by content hash | re-extraction should never mean refetching | | Extraction | Declared selectors in a versioned spec file, validated against a JSON Schema | an extractor without a declared shape cannot detect drift | | Delivery | One sink at a time, idempotent per run and row key | a half-delivered run that reruns is how duplicates reach a spreadsheet | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the fetcher, the extractor and the scheduler. - Get it: python.org, or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **One site you are allowed to extract from** · free - Why: Everything here depends on having a target whose robots.txt permits it and whose data is not about people, and choosing that target is the decision with consequences. - Get it: Read the site's robots.txt and terms, confirm the pages are public and impersonal, and write down the date you checked. - Verify: A written note records the site, the rules that apply and the date - [ ] **A contact address for the User-Agent** · free - Why: Identifying yourself is the difference between a crawler an administrator emails and one they block, and it costs nothing. - Get it: An address you monitor, or a page describing the crawler. - Verify: The User-Agent string contains the address and appears in the target's logs as sent - [ ] **Disk for snapshots** · pennies per month - Why: Replay without refetching is the whole design, and it is paid for in stored bytes that grow with every run until retention is set. - Get it: A few gigabytes on the machine, or a mounted volume you can grow. - Verify: The snapshot directory reports its size and an alert threshold is configured - [ ] **One delivery sink and a row key** · free - Why: Idempotent delivery needs a stable key per row, and picking it after the first duplicate arrives means picking whatever is convenient rather than correct. - Get it: Choose the sink (a file, a spreadsheet, a table) and the field or combination that uniquely identifies a row on the site. - Verify: Both are in the spec and the delivery test asserts uniqueness on that key ## Quick start ```sh python -m pytest tests/test_robots.py -q ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Anything behind a login. - Personal data of any kind. - Address rotation or header spoofing. - A connector library. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls If one of those is essential to you, that is the reason to keep paying for Import.io, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Import.io The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a web data extraction pipeline that obeys the exclusion protocol, notices when a page changes shape, and can replay any run to replace Import.io. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three rules, and all three are settled before anything fetches a page. Follow the Robots Exclusion Protocol as the standard defines it rather than as folklore does. Fetch robots.txt per host and do not use a cached copy for more than 24 hours. If the file is unreachable or the server answers with a server error, you must assume a complete disallow, and only after a long period (the standard offers 30 days as the example) may you treat it as simply unavailable. A 4xx means no file exists and fetching is allowed. Parse at least 500 kibibytes of it. Crawl-delay is not part of the standard at all, so your own pacing is a courtesy you choose and enforce yourself, and one request per second per host is a defensible starting point. Second, identify the crawler in the User-Agent with a name and a contact address, so an administrator who wants you to stop has somewhere to write instead of a firewall rule. Third, only pages an anonymous visitor can reach. No logging in, no session cookies lifted from a browser, no data about identifiable people. A site behind a login has terms you accepted, and personal data turns a scraping project into a legal one. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with one worker process and a scheduler, no distributed queue · durable execution at scale is the part you are deliberately not rebuilding - Fetching: httpx with a per-host token bucket and conditional requests · pacing and caching are the whole of being a good citizen - Storage: SQLite for runs and rows, compressed page snapshots on disk keyed by content hash · re-extraction should never mean refetching - Extraction: Declared selectors in a versioned spec file, validated against a JSON Schema · an extractor without a declared shape cannot detect drift - Delivery: One sink at a time, idempotent per run and row key · a half-delivered run that reruns is how duplicates reach a spreadsheet ### Data model (create this before Phase 1) - sites(host, robots_fetched_at, robots_body_sha256, robots_status, rate_per_second, user_agent) · the cached exclusion file and its age - specs(id, name, version, url_pattern, selectors_json, schema_json, created_at) · the extractor, versioned, because drift is a diff between versions - runs(id, spec_id, started_at, finished_at, state, pages_fetched, pages_from_cache, rows_extracted, quarantined) · state is running, succeeded, quarantined or failed - fetches(id, run_id, url, requested_at, status, etag, last_modified, bytes, snapshot_sha256, from_cache, robots_decision) - rows(id, run_id, row_key, data_json, snapshot_sha256, valid) · row_key is declared in the spec and is what makes delivery idempotent - deliveries(run_id, sink, row_key, delivered_at) · unique, so a replay delivers only what did not arrive - A snapshot is stored once per content hash and referenced by every fetch that produced it, so a spec fixed after the fact is re-run against the exact bytes the site served. ### Phase 1 · A fetcher that obeys the protocol Build: Fetch and cache robots.txt per host with its status and age, apply the rules above including the disallow on a server error, and pace per host with a token bucket. Send a User-Agent naming the crawler with a contact address. Store every response body as a compressed snapshot keyed by its content hash, and use ETag and Last-Modified so an unchanged page costs one conditional request. Trap: The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. Done when: A host whose robots.txt returns a server error is not fetched, and the refusal is recorded against the run with its reason. A cached exclusion file older than 24 hours is refetched before the next request to that host, tested with a clock shim. A 4xx on robots.txt allows fetching, and a 200 with a matching disallow blocks it, both covered by tests. Every response is stored as a snapshot and an unchanged page on a second run is served from cache after one conditional request. Do not build yet: No extraction, schemas, drift detection or delivery. ### Phase 2 · Extraction from stored bytes, against a declared shape Build: Write extraction as a pure function from a snapshot to rows, driven by a versioned spec holding selectors, a JSON Schema for the row and a declared row key. Validate every row against the schema. Run the whole thing over stored snapshots with the network unavailable. Trap: An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. Done when: The extraction test suite runs with the network blocked and passes, which is the assertion. Every row validates against the spec's schema and an invalid row is recorded with the failing field rather than dropped. Changing a selector and re-running produces new rows from existing snapshots with no fetches recorded. Each row carries the snapshot hash it came from. Do not build yet: No drift detection, delivery or scheduling yet. ### Phase 3 · Noticing that the page changed shape Build: Compare each run against the previous successful one: row count, null rate per field, type failure count, and the distribution of a numeric field you nominate. Breach a stated threshold and the run is quarantined, not delivered, with a diff showing which fields moved. Trap: The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. Done when: A run in which one field becomes constant across all rows is quarantined, tested with a doctored snapshot set. A run whose row count moves by more than the stated threshold is quarantined with the previous and current counts shown. Quarantined runs are never delivered, asserted by the delivery code path. The drift report names the fields that moved and by how much. Do not build yet: No delivery, replay or scheduling yet. ### Phase 4 · Delivery you can replay without duplicating Build: Deliver rows to one sink keyed by run and row key, recording each delivery in the same transaction where possible and immediately after otherwise. A replay skips keys already recorded. Offer a manual replay for a single run from the interface, with a dry run that lists what it would send. Trap: Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. Done when: Killing the process mid-delivery and replaying results in every row delivered exactly once, tested by counting at a fake sink. A dry run lists exactly the rows a real replay would send and sends none. A replay of a fully delivered run sends nothing and says so. Delivery of a quarantined run is refused with the drift reason shown. Do not build yet: No scheduling, storage ceilings or export yet. ### Phase 5 · Running unattended without a surprise bill Build: Schedule specs, with a per-run ceiling on requests, wall clock time and bytes fetched, and a global daily request budget per host. Prune snapshots on a stated retention while keeping the most recent per URL. Put every credential in the environment. A run page shows fetches, cache hits, robots refusals, quarantine state and the drift report. Export runs and rows as files. Trap: Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full. Done when: A run that reaches its request, time or byte ceiling stops and is marked as truncated rather than failing silently or continuing. Snapshot storage is deduplicated by content hash, verified by running the same spec twice and observing no growth. Retention pruning keeps the most recent snapshot per URL, tested. The run page shows fetches, cache hits, robots refusals and the drift report, and export produces files that reimport into an empty database. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, including session cookies copied from your browser, because those pages come with terms you personally accepted. - Data about identifiable people, which turns an engineering decision into a legal one and needs a lawful basis you probably do not have. - Rotating addresses, header spoofing or any attempt to look like a person, which is evasion rather than extraction and ends in a ban you deserve. - Hundreds of maintained connectors, which is a full-time maintenance treadmill; a handful of specs you own is the honest scope. - Durable execution across machines, which is the part the subscription actually pays for. ### README must contain - The exclusion rules as implemented, including that an unreachable or erroring robots.txt is treated as a complete disallow - That the cached exclusion file is never reused beyond 24 hours, and that pacing is your own choice because the standard defines no delay - The User-Agent string, including the contact address an administrator can write to - That extraction runs from stored snapshots with the network disabled, so a fixed selector never means a refetch - That a quarantined run cannot be delivered, and what thresholds cause quarantine ===== AGENTS.md ===== # Agent instructions · Import.io indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with one worker process and a scheduler, no distributed queue, httpx with a per-host token bucket and conditional requests, SQLite for runs and rows, compressed page snapshots on disk keyed by content hash, Declared selectors in a versioned spec file, validated against a JSON Schema, One sink at a time, idempotent per run and row key. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. ===== BUILD_PLAN.md ===== # Build plan · Import.io A personal extraction pipeline whose hard problems are politeness enforced by the protocol rather than by good intentions, noticing when a selector still matches but now returns the cookie banner, and replaying a run without refetching or double-delivering. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A fetcher that obeys the protocol No request leaves the process until the exclusion rules for that host have been applied. ### Steps 1. Implement the exclusion rules including the deny cases The deny-on-error branch is the one every naive crawler omits. Files: `src/fetch/robots.py`, `tests/test_robots.py` ```sh python -m pytest tests/test_robots.py -q ``` 2. Pace per host with a token bucket and a real User-Agent The standard does not set a delay, so you set one and honour it. Files: `src/fetch/pacing.py`, `src/fetch/client.py` 3. Store snapshots by content hash with conditional requests So a broken extractor is fixed without touching the site again. Files: `src/store/snapshots.py` ### Done when - [ ] A server error on robots.txt blocks fetching and is recorded - [ ] A cache older than 24 hours is refetched - [ ] Disallow rules block and a missing file allows - [ ] An unchanged page costs one conditional request ### Watch out - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. ## Phase 2 · Extraction from stored bytes, against a declared shape A spec declares selectors, types and a row key, and extraction never touches the network. ### Steps 1. Make extraction a pure function of a snapshot Testable, replayable and free to run a hundred times. Files: `src/extract/run.py`, `tests/test_extract.py` 2. Validate rows against the spec's schema A type error found now is a support question avoided later. Files: `src/extract/validate.py` 3. Block the network in the extraction tests The one assertion that keeps the split honest. Files: `tests/conftest.py` ```sh python -m pytest tests/test_extract.py -q --disable-socket ``` ### Done when - [ ] Extraction tests pass with networking disabled - [ ] Invalid rows are recorded with the failing field - [ ] A selector change re-runs with zero fetches - [ ] Every row carries its snapshot hash ### Watch out - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. ## Phase 3 · Noticing that the page changed shape A run whose output looks wrong is quarantined before anything downstream sees it. ### Steps 1. Store per-field statistics on every successful run You cannot diff against a baseline you never recorded. Files: `src/quality/stats.py` 2. Quarantine on a threshold breach, do not warn A warning in a log is a delivery of bad data with extra steps. Files: `src/quality/gate.py`, `tests/test_drift.py` 3. Flag a field that has become constant This is what a cookie banner looks like in a dataset. Files: `src/quality/stats.py` ### Done when - [ ] A constant field quarantines the run - [ ] A row count swing beyond the threshold quarantines the run - [ ] Quarantined runs are never delivered - [ ] The drift report names the moved fields ### Watch out - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. ## Phase 4 · Delivery you can replay without duplicating Re-running a half-delivered run sends only what did not arrive. ### Steps 1. Record delivery per row key, uniquely The unique index is what makes replay safe. Files: `src/deliver/sink.py`, `tests/test_replay.py` 2. Offer a dry run that lists and sends nothing Nobody presses replay confidently without one. Files: `src/deliver/dryrun.py` 3. Refuse delivery of a quarantined run The gate is worthless if the replay button bypasses it. Files: `src/deliver/sink.py` ### Done when - [ ] A mid-delivery crash plus replay yields one copy per row - [ ] A dry run sends nothing and lists accurately - [ ] Replaying a complete run sends nothing - [ ] Quarantined runs cannot be delivered ### Watch out - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. ## Not in this build - Anything behind a login. - Personal data of any kind. - Address rotation or header spoofing. - A connector library. ## After v1, if you want it - A second sink, which the delivery table already supports and which needs only a module - Change alerts per field, once the per-field statistics have a few weeks of history behind them - Rendering pages that need scripting, which is a different fetcher and should stay behind the same robots gate ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, fetches, rows and deliveries. DATABASE_PATH=./data/pipeline.db # Required. Compressed pages keyed by content hash. Watch its size. SNAPSHOT_DIR=./data/snapshots # Required. Sent on every request. Must carry a contact address. USER_AGENT=example-extractor/1.0 (+mailto:you@example.com) # Required. Your own pacing, since the standard defines no delay. Lower it per host as needed. DEFAULT_RATE_PER_SECOND=1 # Required. Never longer than 24 hours, which is the ceiling the standard gives. ROBOTS_CACHE_SECONDS=86400 # Required. Per run. Reaching it truncates the run rather than failing it. RUN_MAX_REQUESTS=2000 # Required. The most recent snapshot per URL is kept regardless. SNAPSHOT_RETENTION_DAYS=30 # Required. Fractional change in row count that quarantines a run. DRIFT_ROW_COUNT_TOLERANCE=0.2
You are building a lean indie version of Import.io. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Import.io · indie build A personal extraction pipeline whose hard problems are politeness enforced by the protocol rather than by good intentions, noticing when a selector still matches but now returns the cookie banner, and replaying a run without refetching or double-delivering. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with one worker process and a scheduler, no distributed queue | durable execution at scale is the part you are deliberately not rebuilding | | Fetching | httpx with a per-host token bucket and conditional requests | pacing and caching are the whole of being a good citizen | | Storage | SQLite for runs and rows, compressed page snapshots on disk keyed by content hash | re-extraction should never mean refetching | | Extraction | Declared selectors in a versioned spec file, validated against a JSON Schema | an extractor without a declared shape cannot detect drift | | Delivery | One sink at a time, idempotent per run and row key | a half-delivered run that reruns is how duplicates reach a spreadsheet | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the fetcher, the extractor and the scheduler. - Get it: python.org, or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **One site you are allowed to extract from** · free - Why: Everything here depends on having a target whose robots.txt permits it and whose data is not about people, and choosing that target is the decision with consequences. - Get it: Read the site's robots.txt and terms, confirm the pages are public and impersonal, and write down the date you checked. - Verify: A written note records the site, the rules that apply and the date - [ ] **A contact address for the User-Agent** · free - Why: Identifying yourself is the difference between a crawler an administrator emails and one they block, and it costs nothing. - Get it: An address you monitor, or a page describing the crawler. - Verify: The User-Agent string contains the address and appears in the target's logs as sent - [ ] **Disk for snapshots** · pennies per month - Why: Replay without refetching is the whole design, and it is paid for in stored bytes that grow with every run until retention is set. - Get it: A few gigabytes on the machine, or a mounted volume you can grow. - Verify: The snapshot directory reports its size and an alert threshold is configured - [ ] **One delivery sink and a row key** · free - Why: Idempotent delivery needs a stable key per row, and picking it after the first duplicate arrives means picking whatever is convenient rather than correct. - Get it: Choose the sink (a file, a spreadsheet, a table) and the field or combination that uniquely identifies a row on the site. - Verify: Both are in the spec and the delivery test asserts uniqueness on that key ## Quick start ```sh python -m pytest tests/test_robots.py -q ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Anything behind a login. - Personal data of any kind. - Address rotation or header spoofing. - A connector library. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls If one of those is essential to you, that is the reason to keep paying for Import.io, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Import.io The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a web data extraction pipeline that obeys the exclusion protocol, notices when a page changes shape, and can replay any run to replace Import.io. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three rules, and all three are settled before anything fetches a page. Follow the Robots Exclusion Protocol as the standard defines it rather than as folklore does. Fetch robots.txt per host and do not use a cached copy for more than 24 hours. If the file is unreachable or the server answers with a server error, you must assume a complete disallow, and only after a long period (the standard offers 30 days as the example) may you treat it as simply unavailable. A 4xx means no file exists and fetching is allowed. Parse at least 500 kibibytes of it. Crawl-delay is not part of the standard at all, so your own pacing is a courtesy you choose and enforce yourself, and one request per second per host is a defensible starting point. Second, identify the crawler in the User-Agent with a name and a contact address, so an administrator who wants you to stop has somewhere to write instead of a firewall rule. Third, only pages an anonymous visitor can reach. No logging in, no session cookies lifted from a browser, no data about identifiable people. A site behind a login has terms you accepted, and personal data turns a scraping project into a legal one. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with one worker process and a scheduler, no distributed queue · durable execution at scale is the part you are deliberately not rebuilding - Fetching: httpx with a per-host token bucket and conditional requests · pacing and caching are the whole of being a good citizen - Storage: SQLite for runs and rows, compressed page snapshots on disk keyed by content hash · re-extraction should never mean refetching - Extraction: Declared selectors in a versioned spec file, validated against a JSON Schema · an extractor without a declared shape cannot detect drift - Delivery: One sink at a time, idempotent per run and row key · a half-delivered run that reruns is how duplicates reach a spreadsheet ### Data model (create this before Phase 1) - sites(host, robots_fetched_at, robots_body_sha256, robots_status, rate_per_second, user_agent) · the cached exclusion file and its age - specs(id, name, version, url_pattern, selectors_json, schema_json, created_at) · the extractor, versioned, because drift is a diff between versions - runs(id, spec_id, started_at, finished_at, state, pages_fetched, pages_from_cache, rows_extracted, quarantined) · state is running, succeeded, quarantined or failed - fetches(id, run_id, url, requested_at, status, etag, last_modified, bytes, snapshot_sha256, from_cache, robots_decision) - rows(id, run_id, row_key, data_json, snapshot_sha256, valid) · row_key is declared in the spec and is what makes delivery idempotent - deliveries(run_id, sink, row_key, delivered_at) · unique, so a replay delivers only what did not arrive - A snapshot is stored once per content hash and referenced by every fetch that produced it, so a spec fixed after the fact is re-run against the exact bytes the site served. ### Phase 1 · A fetcher that obeys the protocol Build: Fetch and cache robots.txt per host with its status and age, apply the rules above including the disallow on a server error, and pace per host with a token bucket. Send a User-Agent naming the crawler with a contact address. Store every response body as a compressed snapshot keyed by its content hash, and use ETag and Last-Modified so an unchanged page costs one conditional request. Trap: The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. Done when: A host whose robots.txt returns a server error is not fetched, and the refusal is recorded against the run with its reason. A cached exclusion file older than 24 hours is refetched before the next request to that host, tested with a clock shim. A 4xx on robots.txt allows fetching, and a 200 with a matching disallow blocks it, both covered by tests. Every response is stored as a snapshot and an unchanged page on a second run is served from cache after one conditional request. Do not build yet: No extraction, schemas, drift detection or delivery. ### Phase 2 · Extraction from stored bytes, against a declared shape Build: Write extraction as a pure function from a snapshot to rows, driven by a versioned spec holding selectors, a JSON Schema for the row and a declared row key. Validate every row against the schema. Run the whole thing over stored snapshots with the network unavailable. Trap: An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. Done when: The extraction test suite runs with the network blocked and passes, which is the assertion. Every row validates against the spec's schema and an invalid row is recorded with the failing field rather than dropped. Changing a selector and re-running produces new rows from existing snapshots with no fetches recorded. Each row carries the snapshot hash it came from. Do not build yet: No drift detection, delivery or scheduling yet. ### Phase 3 · Noticing that the page changed shape Build: Compare each run against the previous successful one: row count, null rate per field, type failure count, and the distribution of a numeric field you nominate. Breach a stated threshold and the run is quarantined, not delivered, with a diff showing which fields moved. Trap: The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. Done when: A run in which one field becomes constant across all rows is quarantined, tested with a doctored snapshot set. A run whose row count moves by more than the stated threshold is quarantined with the previous and current counts shown. Quarantined runs are never delivered, asserted by the delivery code path. The drift report names the fields that moved and by how much. Do not build yet: No delivery, replay or scheduling yet. ### Phase 4 · Delivery you can replay without duplicating Build: Deliver rows to one sink keyed by run and row key, recording each delivery in the same transaction where possible and immediately after otherwise. A replay skips keys already recorded. Offer a manual replay for a single run from the interface, with a dry run that lists what it would send. Trap: Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. Done when: Killing the process mid-delivery and replaying results in every row delivered exactly once, tested by counting at a fake sink. A dry run lists exactly the rows a real replay would send and sends none. A replay of a fully delivered run sends nothing and says so. Delivery of a quarantined run is refused with the drift reason shown. Do not build yet: No scheduling, storage ceilings or export yet. ### Phase 5 · Running unattended without a surprise bill Build: Schedule specs, with a per-run ceiling on requests, wall clock time and bytes fetched, and a global daily request budget per host. Prune snapshots on a stated retention while keeping the most recent per URL. Put every credential in the environment. A run page shows fetches, cache hits, robots refusals, quarantine state and the drift report. Export runs and rows as files. Trap: Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full. Done when: A run that reaches its request, time or byte ceiling stops and is marked as truncated rather than failing silently or continuing. Snapshot storage is deduplicated by content hash, verified by running the same spec twice and observing no growth. Retention pruning keeps the most recent snapshot per URL, tested. The run page shows fetches, cache hits, robots refusals and the drift report, and export produces files that reimport into an empty database. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, including session cookies copied from your browser, because those pages come with terms you personally accepted. - Data about identifiable people, which turns an engineering decision into a legal one and needs a lawful basis you probably do not have. - Rotating addresses, header spoofing or any attempt to look like a person, which is evasion rather than extraction and ends in a ban you deserve. - Hundreds of maintained connectors, which is a full-time maintenance treadmill; a handful of specs you own is the honest scope. - Durable execution across machines, which is the part the subscription actually pays for. ### README must contain - The exclusion rules as implemented, including that an unreachable or erroring robots.txt is treated as a complete disallow - That the cached exclusion file is never reused beyond 24 hours, and that pacing is your own choice because the standard defines no delay - The User-Agent string, including the contact address an administrator can write to - That extraction runs from stored snapshots with the network disabled, so a fixed selector never means a refetch - That a quarantined run cannot be delivered, and what thresholds cause quarantine ===== AGENTS.md ===== # Agent instructions · Import.io indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with one worker process and a scheduler, no distributed queue, httpx with a per-host token bucket and conditional requests, SQLite for runs and rows, compressed page snapshots on disk keyed by content hash, Declared selectors in a versioned spec file, validated against a JSON Schema, One sink at a time, idempotent per run and row key. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. ===== BUILD_PLAN.md ===== # Build plan · Import.io A personal extraction pipeline whose hard problems are politeness enforced by the protocol rather than by good intentions, noticing when a selector still matches but now returns the cookie banner, and replaying a run without refetching or double-delivering. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A fetcher that obeys the protocol No request leaves the process until the exclusion rules for that host have been applied. ### Steps 1. Implement the exclusion rules including the deny cases The deny-on-error branch is the one every naive crawler omits. Files: `src/fetch/robots.py`, `tests/test_robots.py` ```sh python -m pytest tests/test_robots.py -q ``` 2. Pace per host with a token bucket and a real User-Agent The standard does not set a delay, so you set one and honour it. Files: `src/fetch/pacing.py`, `src/fetch/client.py` 3. Store snapshots by content hash with conditional requests So a broken extractor is fixed without touching the site again. Files: `src/store/snapshots.py` ### Done when - [ ] A server error on robots.txt blocks fetching and is recorded - [ ] A cache older than 24 hours is refetched - [ ] Disallow rules block and a missing file allows - [ ] An unchanged page costs one conditional request ### Watch out - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. ## Phase 2 · Extraction from stored bytes, against a declared shape A spec declares selectors, types and a row key, and extraction never touches the network. ### Steps 1. Make extraction a pure function of a snapshot Testable, replayable and free to run a hundred times. Files: `src/extract/run.py`, `tests/test_extract.py` 2. Validate rows against the spec's schema A type error found now is a support question avoided later. Files: `src/extract/validate.py` 3. Block the network in the extraction tests The one assertion that keeps the split honest. Files: `tests/conftest.py` ```sh python -m pytest tests/test_extract.py -q --disable-socket ``` ### Done when - [ ] Extraction tests pass with networking disabled - [ ] Invalid rows are recorded with the failing field - [ ] A selector change re-runs with zero fetches - [ ] Every row carries its snapshot hash ### Watch out - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. ## Phase 3 · Noticing that the page changed shape A run whose output looks wrong is quarantined before anything downstream sees it. ### Steps 1. Store per-field statistics on every successful run You cannot diff against a baseline you never recorded. Files: `src/quality/stats.py` 2. Quarantine on a threshold breach, do not warn A warning in a log is a delivery of bad data with extra steps. Files: `src/quality/gate.py`, `tests/test_drift.py` 3. Flag a field that has become constant This is what a cookie banner looks like in a dataset. Files: `src/quality/stats.py` ### Done when - [ ] A constant field quarantines the run - [ ] A row count swing beyond the threshold quarantines the run - [ ] Quarantined runs are never delivered - [ ] The drift report names the moved fields ### Watch out - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. ## Phase 4 · Delivery you can replay without duplicating Re-running a half-delivered run sends only what did not arrive. ### Steps 1. Record delivery per row key, uniquely The unique index is what makes replay safe. Files: `src/deliver/sink.py`, `tests/test_replay.py` 2. Offer a dry run that lists and sends nothing Nobody presses replay confidently without one. Files: `src/deliver/dryrun.py` 3. Refuse delivery of a quarantined run The gate is worthless if the replay button bypasses it. Files: `src/deliver/sink.py` ### Done when - [ ] A mid-delivery crash plus replay yields one copy per row - [ ] A dry run sends nothing and lists accurately - [ ] Replaying a complete run sends nothing - [ ] Quarantined runs cannot be delivered ### Watch out - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. ## Not in this build - Anything behind a login. - Personal data of any kind. - Address rotation or header spoofing. - A connector library. ## After v1, if you want it - A second sink, which the delivery table already supports and which needs only a module - Change alerts per field, once the per-field statistics have a few weeks of history behind them - Rendering pages that need scripting, which is a different fetcher and should stay behind the same robots gate ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, fetches, rows and deliveries. DATABASE_PATH=./data/pipeline.db # Required. Compressed pages keyed by content hash. Watch its size. SNAPSHOT_DIR=./data/snapshots # Required. Sent on every request. Must carry a contact address. USER_AGENT=example-extractor/1.0 (+mailto:you@example.com) # Required. Your own pacing, since the standard defines no delay. Lower it per host as needed. DEFAULT_RATE_PER_SECOND=1 # Required. Never longer than 24 hours, which is the ceiling the standard gives. ROBOTS_CACHE_SECONDS=86400 # Required. Per run. Reaching it truncates the run rather than failing it. RUN_MAX_REQUESTS=2000 # Required. The most recent snapshot per URL is kept regardless. SNAPSHOT_RETENTION_DAYS=30 # Required. Fractional change in row count that quarantines a run. DRIFT_ROW_COUNT_TOLERANCE=0.2
You are building a production product version of Import.io. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Import.io · product brief ## Problem The visible enterprise web data loop is buildable, but a credible replacement needs more than the first screen. Import.io earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A pipeline that refuses to fetch what it should not, keeps the bytes it did fetch so a fix never costs the site another request, quarantines a run that produced plausible rubbish, and replays a half-delivered run without sending anything twice. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - hosted PostgreSQL and Redis - API credentials for the two chosen services - Node.js 22 ## Explicit non-goals for v1 - Anything behind a login. - Personal data of any kind. - Address rotation or header spoofing. - A connector library. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls ## Success criteria - A server error on robots.txt is confirmed to block fetching - Extraction is confirmed to pass with networking disabled - A constant field is confirmed to quarantine the run - A mid-delivery crash plus replay is confirmed to yield one copy per row - Repeating a spec is confirmed to add no snapshot storage ===== BRIEF.md ===== # Build brief · Import.io The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a web data extraction pipeline that obeys the exclusion protocol, notices when a page changes shape, and can replay any run to replace Import.io. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three rules, and all three are settled before anything fetches a page. Follow the Robots Exclusion Protocol as the standard defines it rather than as folklore does. Fetch robots.txt per host and do not use a cached copy for more than 24 hours. If the file is unreachable or the server answers with a server error, you must assume a complete disallow, and only after a long period (the standard offers 30 days as the example) may you treat it as simply unavailable. A 4xx means no file exists and fetching is allowed. Parse at least 500 kibibytes of it. Crawl-delay is not part of the standard at all, so your own pacing is a courtesy you choose and enforce yourself, and one request per second per host is a defensible starting point. Second, identify the crawler in the User-Agent with a name and a contact address, so an administrator who wants you to stop has somewhere to write instead of a firewall rule. Third, only pages an anonymous visitor can reach. No logging in, no session cookies lifted from a browser, no data about identifiable people. A site behind a login has terms you accepted, and personal data turns a scraping project into a legal one. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with one worker process and a scheduler, no distributed queue · durable execution at scale is the part you are deliberately not rebuilding - Fetching: httpx with a per-host token bucket and conditional requests · pacing and caching are the whole of being a good citizen - Storage: SQLite for runs and rows, compressed page snapshots on disk keyed by content hash · re-extraction should never mean refetching - Extraction: Declared selectors in a versioned spec file, validated against a JSON Schema · an extractor without a declared shape cannot detect drift - Delivery: One sink at a time, idempotent per run and row key · a half-delivered run that reruns is how duplicates reach a spreadsheet ### Data model (create this before Phase 1) - sites(host, robots_fetched_at, robots_body_sha256, robots_status, rate_per_second, user_agent) · the cached exclusion file and its age - specs(id, name, version, url_pattern, selectors_json, schema_json, created_at) · the extractor, versioned, because drift is a diff between versions - runs(id, spec_id, started_at, finished_at, state, pages_fetched, pages_from_cache, rows_extracted, quarantined) · state is running, succeeded, quarantined or failed - fetches(id, run_id, url, requested_at, status, etag, last_modified, bytes, snapshot_sha256, from_cache, robots_decision) - rows(id, run_id, row_key, data_json, snapshot_sha256, valid) · row_key is declared in the spec and is what makes delivery idempotent - deliveries(run_id, sink, row_key, delivered_at) · unique, so a replay delivers only what did not arrive - A snapshot is stored once per content hash and referenced by every fetch that produced it, so a spec fixed after the fact is re-run against the exact bytes the site served. ### Phase 1 · A fetcher that obeys the protocol Build: Fetch and cache robots.txt per host with its status and age, apply the rules above including the disallow on a server error, and pace per host with a token bucket. Send a User-Agent naming the crawler with a contact address. Store every response body as a compressed snapshot keyed by its content hash, and use ETag and Last-Modified so an unchanged page costs one conditional request. Trap: The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. Done when: A host whose robots.txt returns a server error is not fetched, and the refusal is recorded against the run with its reason. A cached exclusion file older than 24 hours is refetched before the next request to that host, tested with a clock shim. A 4xx on robots.txt allows fetching, and a 200 with a matching disallow blocks it, both covered by tests. Every response is stored as a snapshot and an unchanged page on a second run is served from cache after one conditional request. Do not build yet: No extraction, schemas, drift detection or delivery. ### Phase 2 · Extraction from stored bytes, against a declared shape Build: Write extraction as a pure function from a snapshot to rows, driven by a versioned spec holding selectors, a JSON Schema for the row and a declared row key. Validate every row against the schema. Run the whole thing over stored snapshots with the network unavailable. Trap: An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. Done when: The extraction test suite runs with the network blocked and passes, which is the assertion. Every row validates against the spec's schema and an invalid row is recorded with the failing field rather than dropped. Changing a selector and re-running produces new rows from existing snapshots with no fetches recorded. Each row carries the snapshot hash it came from. Do not build yet: No drift detection, delivery or scheduling yet. ### Phase 3 · Noticing that the page changed shape Build: Compare each run against the previous successful one: row count, null rate per field, type failure count, and the distribution of a numeric field you nominate. Breach a stated threshold and the run is quarantined, not delivered, with a diff showing which fields moved. Trap: The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. Done when: A run in which one field becomes constant across all rows is quarantined, tested with a doctored snapshot set. A run whose row count moves by more than the stated threshold is quarantined with the previous and current counts shown. Quarantined runs are never delivered, asserted by the delivery code path. The drift report names the fields that moved and by how much. Do not build yet: No delivery, replay or scheduling yet. ### Phase 4 · Delivery you can replay without duplicating Build: Deliver rows to one sink keyed by run and row key, recording each delivery in the same transaction where possible and immediately after otherwise. A replay skips keys already recorded. Offer a manual replay for a single run from the interface, with a dry run that lists what it would send. Trap: Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. Done when: Killing the process mid-delivery and replaying results in every row delivered exactly once, tested by counting at a fake sink. A dry run lists exactly the rows a real replay would send and sends none. A replay of a fully delivered run sends nothing and says so. Delivery of a quarantined run is refused with the drift reason shown. Do not build yet: No scheduling, storage ceilings or export yet. ### Phase 5 · Running unattended without a surprise bill Build: Schedule specs, with a per-run ceiling on requests, wall clock time and bytes fetched, and a global daily request budget per host. Prune snapshots on a stated retention while keeping the most recent per URL. Put every credential in the environment. A run page shows fetches, cache hits, robots refusals, quarantine state and the drift report. Export runs and rows as files. Trap: Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full. Done when: A run that reaches its request, time or byte ceiling stops and is marked as truncated rather than failing silently or continuing. Snapshot storage is deduplicated by content hash, verified by running the same spec twice and observing no growth. Retention pruning keeps the most recent snapshot per URL, tested. The run page shows fetches, cache hits, robots refusals and the drift report, and export produces files that reimport into an empty database. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, including session cookies copied from your browser, because those pages come with terms you personally accepted. - Data about identifiable people, which turns an engineering decision into a legal one and needs a lawful basis you probably do not have. - Rotating addresses, header spoofing or any attempt to look like a person, which is evasion rather than extraction and ends in a ban you deserve. - Hundreds of maintained connectors, which is a full-time maintenance treadmill; a handful of specs you own is the honest scope. - Durable execution across machines, which is the part the subscription actually pays for. ### README must contain - The exclusion rules as implemented, including that an unreachable or erroring robots.txt is treated as a complete disallow - That the cached exclusion file is never reused beyond 24 hours, and that pacing is your own choice because the standard defines no delay - The User-Agent string, including the contact address an administrator can write to - That extraction runs from stored snapshots with the network disabled, so a fixed selector never means a refetch - That a quarantined run cannot be delivered, and what thresholds cause quarantine ===== ARCHITECTURE.md ===== # Architecture · Import.io ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with one worker process and a scheduler, no distributed queue | durable execution at scale is the part you are deliberately not rebuilding | | Fetching | httpx with a per-host token bucket and conditional requests | pacing and caching are the whole of being a good citizen | | Storage | SQLite for runs and rows, compressed page snapshots on disk keyed by content hash | re-extraction should never mean refetching | | Extraction | Declared selectors in a versioned spec file, validated against a JSON Schema | an extractor without a declared shape cannot detect drift | | Delivery | One sink at a time, idempotent per run and row key | a half-delivered run that reruns is how duplicates reach a spreadsheet | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Robots | Per-host rules, caching and the deny cases. | Any parser that implements the standard, including the error branches. | | Fetcher | Pacing, conditional requests and snapshots. | Any client, provided every response is snapshotted by content hash. | | Extractor | Pure snapshot to rows, schema validated. | Any selector language, provided extraction never touches the network. | | Quality | Per-field statistics, drift and quarantine. | Any statistic, provided quarantine blocks delivery rather than warning. | | Delivery | Per row key idempotency and replay. | Any sink, provided the delivered set is recorded per key. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Runs, fetches, rows and deliveries. - `SNAPSHOT_DIR` · required · Compressed pages keyed by content hash. Watch its size. - `USER_AGENT` · required · Sent on every request. Must carry a contact address. - `DEFAULT_RATE_PER_SECOND` · required · Your own pacing, since the standard defines no delay. Lower it per host as needed. - `ROBOTS_CACHE_SECONDS` · required · Never longer than 24 hours, which is the ceiling the standard gives. - `RUN_MAX_REQUESTS` · required · Per run. Reaching it truncates the run rather than failing it. - `SNAPSHOT_RETENTION_DAYS` · required · The most recent snapshot per URL is kept regardless. - `DRIFT_ROW_COUNT_TOLERANCE` · required · Fractional change in row count that quarantines a run. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Import.io product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with one worker process and a scheduler, no distributed queue, httpx with a per-host token bucket and conditional requests, SQLite for runs and rows, compressed page snapshots on disk keyed by content hash, Declared selectors in a versioned spec file, validated against a JSON Schema, One sink at a time, idempotent per run and row key. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. - Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full. ===== MILESTONES.md ===== # Delivery milestones · Import.io Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A fetcher that obeys the protocol No request leaves the process until the exclusion rules for that host have been applied. ### Steps 1. Implement the exclusion rules including the deny cases The deny-on-error branch is the one every naive crawler omits. Files: `src/fetch/robots.py`, `tests/test_robots.py` ```sh python -m pytest tests/test_robots.py -q ``` 2. Pace per host with a token bucket and a real User-Agent The standard does not set a delay, so you set one and honour it. Files: `src/fetch/pacing.py`, `src/fetch/client.py` 3. Store snapshots by content hash with conditional requests So a broken extractor is fixed without touching the site again. Files: `src/store/snapshots.py` ### Done when - [ ] A server error on robots.txt blocks fetching and is recorded - [ ] A cache older than 24 hours is refetched - [ ] Disallow rules block and a missing file allows - [ ] An unchanged page costs one conditional request ### Watch out - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. ## M2 · Extraction from stored bytes, against a declared shape A spec declares selectors, types and a row key, and extraction never touches the network. ### Steps 1. Make extraction a pure function of a snapshot Testable, replayable and free to run a hundred times. Files: `src/extract/run.py`, `tests/test_extract.py` 2. Validate rows against the spec's schema A type error found now is a support question avoided later. Files: `src/extract/validate.py` 3. Block the network in the extraction tests The one assertion that keeps the split honest. Files: `tests/conftest.py` ```sh python -m pytest tests/test_extract.py -q --disable-socket ``` ### Done when - [ ] Extraction tests pass with networking disabled - [ ] Invalid rows are recorded with the failing field - [ ] A selector change re-runs with zero fetches - [ ] Every row carries its snapshot hash ### Watch out - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. ## M3 · Noticing that the page changed shape A run whose output looks wrong is quarantined before anything downstream sees it. ### Steps 1. Store per-field statistics on every successful run You cannot diff against a baseline you never recorded. Files: `src/quality/stats.py` 2. Quarantine on a threshold breach, do not warn A warning in a log is a delivery of bad data with extra steps. Files: `src/quality/gate.py`, `tests/test_drift.py` 3. Flag a field that has become constant This is what a cookie banner looks like in a dataset. Files: `src/quality/stats.py` ### Done when - [ ] A constant field quarantines the run - [ ] A row count swing beyond the threshold quarantines the run - [ ] Quarantined runs are never delivered - [ ] The drift report names the moved fields ### Watch out - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. ## M4 · Delivery you can replay without duplicating Re-running a half-delivered run sends only what did not arrive. ### Steps 1. Record delivery per row key, uniquely The unique index is what makes replay safe. Files: `src/deliver/sink.py`, `tests/test_replay.py` 2. Offer a dry run that lists and sends nothing Nobody presses replay confidently without one. Files: `src/deliver/dryrun.py` 3. Refuse delivery of a quarantined run The gate is worthless if the replay button bypasses it. Files: `src/deliver/sink.py` ### Done when - [ ] A mid-delivery crash plus replay yields one copy per row - [ ] A dry run sends nothing and lists accurately - [ ] Replaying a complete run sends nothing - [ ] Quarantined runs cannot be delivered ### Watch out - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. ## M5 · Running unattended without a surprise bill (production only) Scheduled runs stay inside disk, time and request ceilings, and a failure is visible without reading a log. ### Steps 1. Enforce per-run and per-day ceilings An unattended crawler without a ceiling is a bill or a ban. Files: `src/schedule/limits.py`, `tests/test_limits.py` 2. Deduplicate and prune snapshots Keep the most recent per URL whatever the retention says. Files: `src/store/retention.py` 3. Show refusals and drift on the run page The two things you need and will not find in a log. Files: `src/web/templates/run.html` ### Done when - [ ] Ceilings truncate a run and mark it truncated - [ ] Repeating a spec adds no snapshot storage - [ ] Pruning keeps the most recent snapshot per URL - [ ] The run page shows refusals, cache hits and drift ### Watch out - Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full. ===== OPERATIONS.md ===== # Operations · Import.io ## Backup The database and the snapshot directory together, since rows reference snapshot hashes and a restore of one without the other leaves a history you cannot replay. The database alone is enough to prove what was delivered. ## Restore Restore both, then mark any run left in the running state as failed rather than resuming it, and replay through the delivery table so nothing is sent twice. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Snapshot directory size, quarantined runs, robots refusals per host, truncated runs and the age of each host's cached rules. A growing refusal count usually means the site changed its rules and nobody read the diff. ## Incident checklist Duplicated rows downstream mean the delivery table was bypassed or the row key is not unique on the site. Rubbish rows delivered mean the drift thresholds are too loose, and the per-field statistics from the last good run show exactly which field moved. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A server error on robots.txt is confirmed to block fetching - [ ] Extraction is confirmed to pass with networking disabled - [ ] A constant field is confirmed to quarantine the run - [ ] A mid-delivery crash plus replay is confirmed to yield one copy per row - [ ] Repeating a spec is confirmed to add no snapshot storage ## Launch constraint Do not market omitted Import.io capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, fetches, rows and deliveries. DATABASE_PATH=./data/pipeline.db # Required. Compressed pages keyed by content hash. Watch its size. SNAPSHOT_DIR=./data/snapshots # Required. Sent on every request. Must carry a contact address. USER_AGENT=example-extractor/1.0 (+mailto:you@example.com) # Required. Your own pacing, since the standard defines no delay. Lower it per host as needed. DEFAULT_RATE_PER_SECOND=1 # Required. Never longer than 24 hours, which is the ceiling the standard gives. ROBOTS_CACHE_SECONDS=86400 # Required. Per run. Reaching it truncates the run rather than failing it. RUN_MAX_REQUESTS=2000 # Required. The most recent snapshot per URL is kept regardless. SNAPSHOT_RETENTION_DAYS=30 # Required. Fractional change in row count that quarantines a run. DRIFT_ROW_COUNT_TOLERANCE=0.2
README.md
# Import.io · indie build A personal extraction pipeline whose hard problems are politeness enforced by the protocol rather than by good intentions, noticing when a selector still matches but now returns the cookie banner, and replaying a run without refetching or double-delivering. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with one worker process and a scheduler, no distributed queue | durable execution at scale is the part you are deliberately not rebuilding | | Fetching | httpx with a per-host token bucket and conditional requests | pacing and caching are the whole of being a good citizen | | Storage | SQLite for runs and rows, compressed page snapshots on disk keyed by content hash | re-extraction should never mean refetching | | Extraction | Declared selectors in a versioned spec file, validated against a JSON Schema | an extractor without a declared shape cannot detect drift | | Delivery | One sink at a time, idempotent per run and row key | a half-delivered run that reruns is how duplicates reach a spreadsheet | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the fetcher, the extractor and the scheduler. - Get it: python.org, or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **One site you are allowed to extract from** · free - Why: Everything here depends on having a target whose robots.txt permits it and whose data is not about people, and choosing that target is the decision with consequences. - Get it: Read the site's robots.txt and terms, confirm the pages are public and impersonal, and write down the date you checked. - Verify: A written note records the site, the rules that apply and the date - [ ] **A contact address for the User-Agent** · free - Why: Identifying yourself is the difference between a crawler an administrator emails and one they block, and it costs nothing. - Get it: An address you monitor, or a page describing the crawler. - Verify: The User-Agent string contains the address and appears in the target's logs as sent - [ ] **Disk for snapshots** · pennies per month - Why: Replay without refetching is the whole design, and it is paid for in stored bytes that grow with every run until retention is set. - Get it: A few gigabytes on the machine, or a mounted volume you can grow. - Verify: The snapshot directory reports its size and an alert threshold is configured - [ ] **One delivery sink and a row key** · free - Why: Idempotent delivery needs a stable key per row, and picking it after the first duplicate arrives means picking whatever is convenient rather than correct. - Get it: Choose the sink (a file, a spreadsheet, a table) and the field or combination that uniquely identifies a row on the site. - Verify: Both are in the spec and the delivery test asserts uniqueness on that key ## Quick start ```sh python -m pytest tests/test_robots.py -q ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Anything behind a login. - Personal data of any kind. - Address rotation or header spoofing. - A connector library. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls If one of those is essential to you, that is the reason to keep paying for Import.io, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Import.io The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a web data extraction pipeline that obeys the exclusion protocol, notices when a page changes shape, and can replay any run to replace Import.io. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three rules, and all three are settled before anything fetches a page. Follow the Robots Exclusion Protocol as the standard defines it rather than as folklore does. Fetch robots.txt per host and do not use a cached copy for more than 24 hours. If the file is unreachable or the server answers with a server error, you must assume a complete disallow, and only after a long period (the standard offers 30 days as the example) may you treat it as simply unavailable. A 4xx means no file exists and fetching is allowed. Parse at least 500 kibibytes of it. Crawl-delay is not part of the standard at all, so your own pacing is a courtesy you choose and enforce yourself, and one request per second per host is a defensible starting point. Second, identify the crawler in the User-Agent with a name and a contact address, so an administrator who wants you to stop has somewhere to write instead of a firewall rule. Third, only pages an anonymous visitor can reach. No logging in, no session cookies lifted from a browser, no data about identifiable people. A site behind a login has terms you accepted, and personal data turns a scraping project into a legal one. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with one worker process and a scheduler, no distributed queue · durable execution at scale is the part you are deliberately not rebuilding - Fetching: httpx with a per-host token bucket and conditional requests · pacing and caching are the whole of being a good citizen - Storage: SQLite for runs and rows, compressed page snapshots on disk keyed by content hash · re-extraction should never mean refetching - Extraction: Declared selectors in a versioned spec file, validated against a JSON Schema · an extractor without a declared shape cannot detect drift - Delivery: One sink at a time, idempotent per run and row key · a half-delivered run that reruns is how duplicates reach a spreadsheet ### Data model (create this before Phase 1) - sites(host, robots_fetched_at, robots_body_sha256, robots_status, rate_per_second, user_agent) · the cached exclusion file and its age - specs(id, name, version, url_pattern, selectors_json, schema_json, created_at) · the extractor, versioned, because drift is a diff between versions - runs(id, spec_id, started_at, finished_at, state, pages_fetched, pages_from_cache, rows_extracted, quarantined) · state is running, succeeded, quarantined or failed - fetches(id, run_id, url, requested_at, status, etag, last_modified, bytes, snapshot_sha256, from_cache, robots_decision) - rows(id, run_id, row_key, data_json, snapshot_sha256, valid) · row_key is declared in the spec and is what makes delivery idempotent - deliveries(run_id, sink, row_key, delivered_at) · unique, so a replay delivers only what did not arrive - A snapshot is stored once per content hash and referenced by every fetch that produced it, so a spec fixed after the fact is re-run against the exact bytes the site served. ### Phase 1 · A fetcher that obeys the protocol Build: Fetch and cache robots.txt per host with its status and age, apply the rules above including the disallow on a server error, and pace per host with a token bucket. Send a User-Agent naming the crawler with a contact address. Store every response body as a compressed snapshot keyed by its content hash, and use ETag and Last-Modified so an unchanged page costs one conditional request. Trap: The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. Done when: A host whose robots.txt returns a server error is not fetched, and the refusal is recorded against the run with its reason. A cached exclusion file older than 24 hours is refetched before the next request to that host, tested with a clock shim. A 4xx on robots.txt allows fetching, and a 200 with a matching disallow blocks it, both covered by tests. Every response is stored as a snapshot and an unchanged page on a second run is served from cache after one conditional request. Do not build yet: No extraction, schemas, drift detection or delivery. ### Phase 2 · Extraction from stored bytes, against a declared shape Build: Write extraction as a pure function from a snapshot to rows, driven by a versioned spec holding selectors, a JSON Schema for the row and a declared row key. Validate every row against the schema. Run the whole thing over stored snapshots with the network unavailable. Trap: An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. Done when: The extraction test suite runs with the network blocked and passes, which is the assertion. Every row validates against the spec's schema and an invalid row is recorded with the failing field rather than dropped. Changing a selector and re-running produces new rows from existing snapshots with no fetches recorded. Each row carries the snapshot hash it came from. Do not build yet: No drift detection, delivery or scheduling yet. ### Phase 3 · Noticing that the page changed shape Build: Compare each run against the previous successful one: row count, null rate per field, type failure count, and the distribution of a numeric field you nominate. Breach a stated threshold and the run is quarantined, not delivered, with a diff showing which fields moved. Trap: The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. Done when: A run in which one field becomes constant across all rows is quarantined, tested with a doctored snapshot set. A run whose row count moves by more than the stated threshold is quarantined with the previous and current counts shown. Quarantined runs are never delivered, asserted by the delivery code path. The drift report names the fields that moved and by how much. Do not build yet: No delivery, replay or scheduling yet. ### Phase 4 · Delivery you can replay without duplicating Build: Deliver rows to one sink keyed by run and row key, recording each delivery in the same transaction where possible and immediately after otherwise. A replay skips keys already recorded. Offer a manual replay for a single run from the interface, with a dry run that lists what it would send. Trap: Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. Done when: Killing the process mid-delivery and replaying results in every row delivered exactly once, tested by counting at a fake sink. A dry run lists exactly the rows a real replay would send and sends none. A replay of a fully delivered run sends nothing and says so. Delivery of a quarantined run is refused with the drift reason shown. Do not build yet: No scheduling, storage ceilings or export yet. ### Phase 5 · Running unattended without a surprise bill Build: Schedule specs, with a per-run ceiling on requests, wall clock time and bytes fetched, and a global daily request budget per host. Prune snapshots on a stated retention while keeping the most recent per URL. Put every credential in the environment. A run page shows fetches, cache hits, robots refusals, quarantine state and the drift report. Export runs and rows as files. Trap: Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full. Done when: A run that reaches its request, time or byte ceiling stops and is marked as truncated rather than failing silently or continuing. Snapshot storage is deduplicated by content hash, verified by running the same spec twice and observing no growth. Retention pruning keeps the most recent snapshot per URL, tested. The run page shows fetches, cache hits, robots refusals and the drift report, and export produces files that reimport into an empty database. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, including session cookies copied from your browser, because those pages come with terms you personally accepted. - Data about identifiable people, which turns an engineering decision into a legal one and needs a lawful basis you probably do not have. - Rotating addresses, header spoofing or any attempt to look like a person, which is evasion rather than extraction and ends in a ban you deserve. - Hundreds of maintained connectors, which is a full-time maintenance treadmill; a handful of specs you own is the honest scope. - Durable execution across machines, which is the part the subscription actually pays for. ### README must contain - The exclusion rules as implemented, including that an unreachable or erroring robots.txt is treated as a complete disallow - That the cached exclusion file is never reused beyond 24 hours, and that pacing is your own choice because the standard defines no delay - The User-Agent string, including the contact address an administrator can write to - That extraction runs from stored snapshots with the network disabled, so a fixed selector never means a refetch - That a quarantined run cannot be delivered, and what thresholds cause quarantine
AGENTS.md
# Agent instructions · Import.io indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with one worker process and a scheduler, no distributed queue, httpx with a per-host token bucket and conditional requests, SQLite for runs and rows, compressed page snapshots on disk keyed by content hash, Declared selectors in a versioned spec file, validated against a JSON Schema, One sink at a time, idempotent per run and row key. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat.
BUILD_PLAN.md
# Build plan · Import.io A personal extraction pipeline whose hard problems are politeness enforced by the protocol rather than by good intentions, noticing when a selector still matches but now returns the cookie banner, and replaying a run without refetching or double-delivering. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A fetcher that obeys the protocol No request leaves the process until the exclusion rules for that host have been applied. ### Steps 1. Implement the exclusion rules including the deny cases The deny-on-error branch is the one every naive crawler omits. Files: `src/fetch/robots.py`, `tests/test_robots.py` ```sh python -m pytest tests/test_robots.py -q ``` 2. Pace per host with a token bucket and a real User-Agent The standard does not set a delay, so you set one and honour it. Files: `src/fetch/pacing.py`, `src/fetch/client.py` 3. Store snapshots by content hash with conditional requests So a broken extractor is fixed without touching the site again. Files: `src/store/snapshots.py` ### Done when - [ ] A server error on robots.txt blocks fetching and is recorded - [ ] A cache older than 24 hours is refetched - [ ] Disallow rules block and a missing file allows - [ ] An unchanged page costs one conditional request ### Watch out - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. ## Phase 2 · Extraction from stored bytes, against a declared shape A spec declares selectors, types and a row key, and extraction never touches the network. ### Steps 1. Make extraction a pure function of a snapshot Testable, replayable and free to run a hundred times. Files: `src/extract/run.py`, `tests/test_extract.py` 2. Validate rows against the spec's schema A type error found now is a support question avoided later. Files: `src/extract/validate.py` 3. Block the network in the extraction tests The one assertion that keeps the split honest. Files: `tests/conftest.py` ```sh python -m pytest tests/test_extract.py -q --disable-socket ``` ### Done when - [ ] Extraction tests pass with networking disabled - [ ] Invalid rows are recorded with the failing field - [ ] A selector change re-runs with zero fetches - [ ] Every row carries its snapshot hash ### Watch out - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. ## Phase 3 · Noticing that the page changed shape A run whose output looks wrong is quarantined before anything downstream sees it. ### Steps 1. Store per-field statistics on every successful run You cannot diff against a baseline you never recorded. Files: `src/quality/stats.py` 2. Quarantine on a threshold breach, do not warn A warning in a log is a delivery of bad data with extra steps. Files: `src/quality/gate.py`, `tests/test_drift.py` 3. Flag a field that has become constant This is what a cookie banner looks like in a dataset. Files: `src/quality/stats.py` ### Done when - [ ] A constant field quarantines the run - [ ] A row count swing beyond the threshold quarantines the run - [ ] Quarantined runs are never delivered - [ ] The drift report names the moved fields ### Watch out - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. ## Phase 4 · Delivery you can replay without duplicating Re-running a half-delivered run sends only what did not arrive. ### Steps 1. Record delivery per row key, uniquely The unique index is what makes replay safe. Files: `src/deliver/sink.py`, `tests/test_replay.py` 2. Offer a dry run that lists and sends nothing Nobody presses replay confidently without one. Files: `src/deliver/dryrun.py` 3. Refuse delivery of a quarantined run The gate is worthless if the replay button bypasses it. Files: `src/deliver/sink.py` ### Done when - [ ] A mid-delivery crash plus replay yields one copy per row - [ ] A dry run sends nothing and lists accurately - [ ] Replaying a complete run sends nothing - [ ] Quarantined runs cannot be delivered ### Watch out - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. ## Not in this build - Anything behind a login. - Personal data of any kind. - Address rotation or header spoofing. - A connector library. ## After v1, if you want it - A second sink, which the delivery table already supports and which needs only a module - Change alerts per field, once the per-field statistics have a few weeks of history behind them - Rendering pages that need scripting, which is a different fetcher and should stay behind the same robots gate
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, fetches, rows and deliveries. DATABASE_PATH=./data/pipeline.db # Required. Compressed pages keyed by content hash. Watch its size. SNAPSHOT_DIR=./data/snapshots # Required. Sent on every request. Must carry a contact address. USER_AGENT=example-extractor/1.0 (+mailto:you@example.com) # Required. Your own pacing, since the standard defines no delay. Lower it per host as needed. DEFAULT_RATE_PER_SECOND=1 # Required. Never longer than 24 hours, which is the ceiling the standard gives. ROBOTS_CACHE_SECONDS=86400 # Required. Per run. Reaching it truncates the run rather than failing it. RUN_MAX_REQUESTS=2000 # Required. The most recent snapshot per URL is kept regardless. SNAPSHOT_RETENTION_DAYS=30 # Required. Fractional change in row count that quarantines a run. DRIFT_ROW_COUNT_TOLERANCE=0.2
PRODUCT.md
# Import.io · product brief ## Problem The visible enterprise web data loop is buildable, but a credible replacement needs more than the first screen. Import.io earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A pipeline that refuses to fetch what it should not, keeps the bytes it did fetch so a fix never costs the site another request, quarantines a run that produced plausible rubbish, and replays a half-delivered run without sending anything twice. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - hosted PostgreSQL and Redis - API credentials for the two chosen services - Node.js 22 ## Explicit non-goals for v1 - Anything behind a login. - Personal data of any kind. - Address rotation or header spoofing. - A connector library. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls ## Success criteria - A server error on robots.txt is confirmed to block fetching - Extraction is confirmed to pass with networking disabled - A constant field is confirmed to quarantine the run - A mid-delivery crash plus replay is confirmed to yield one copy per row - Repeating a spec is confirmed to add no snapshot storage
BRIEF.md
# Build brief · Import.io The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a web data extraction pipeline that obeys the exclusion protocol, notices when a page changes shape, and can replay any run to replace Import.io. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Three rules, and all three are settled before anything fetches a page. Follow the Robots Exclusion Protocol as the standard defines it rather than as folklore does. Fetch robots.txt per host and do not use a cached copy for more than 24 hours. If the file is unreachable or the server answers with a server error, you must assume a complete disallow, and only after a long period (the standard offers 30 days as the example) may you treat it as simply unavailable. A 4xx means no file exists and fetching is allowed. Parse at least 500 kibibytes of it. Crawl-delay is not part of the standard at all, so your own pacing is a courtesy you choose and enforce yourself, and one request per second per host is a defensible starting point. Second, identify the crawler in the User-Agent with a name and a contact address, so an administrator who wants you to stop has somewhere to write instead of a firewall rule. Third, only pages an anonymous visitor can reach. No logging in, no session cookies lifted from a browser, no data about identifiable people. A site behind a login has terms you accepted, and personal data turns a scraping project into a legal one. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with one worker process and a scheduler, no distributed queue · durable execution at scale is the part you are deliberately not rebuilding - Fetching: httpx with a per-host token bucket and conditional requests · pacing and caching are the whole of being a good citizen - Storage: SQLite for runs and rows, compressed page snapshots on disk keyed by content hash · re-extraction should never mean refetching - Extraction: Declared selectors in a versioned spec file, validated against a JSON Schema · an extractor without a declared shape cannot detect drift - Delivery: One sink at a time, idempotent per run and row key · a half-delivered run that reruns is how duplicates reach a spreadsheet ### Data model (create this before Phase 1) - sites(host, robots_fetched_at, robots_body_sha256, robots_status, rate_per_second, user_agent) · the cached exclusion file and its age - specs(id, name, version, url_pattern, selectors_json, schema_json, created_at) · the extractor, versioned, because drift is a diff between versions - runs(id, spec_id, started_at, finished_at, state, pages_fetched, pages_from_cache, rows_extracted, quarantined) · state is running, succeeded, quarantined or failed - fetches(id, run_id, url, requested_at, status, etag, last_modified, bytes, snapshot_sha256, from_cache, robots_decision) - rows(id, run_id, row_key, data_json, snapshot_sha256, valid) · row_key is declared in the spec and is what makes delivery idempotent - deliveries(run_id, sink, row_key, delivered_at) · unique, so a replay delivers only what did not arrive - A snapshot is stored once per content hash and referenced by every fetch that produced it, so a spec fixed after the fact is re-run against the exact bytes the site served. ### Phase 1 · A fetcher that obeys the protocol Build: Fetch and cache robots.txt per host with its status and age, apply the rules above including the disallow on a server error, and pace per host with a token bucket. Send a User-Agent naming the crawler with a contact address. Store every response body as a compressed snapshot keyed by its content hash, and use ETag and Last-Modified so an unchanged page costs one conditional request. Trap: The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. Done when: A host whose robots.txt returns a server error is not fetched, and the refusal is recorded against the run with its reason. A cached exclusion file older than 24 hours is refetched before the next request to that host, tested with a clock shim. A 4xx on robots.txt allows fetching, and a 200 with a matching disallow blocks it, both covered by tests. Every response is stored as a snapshot and an unchanged page on a second run is served from cache after one conditional request. Do not build yet: No extraction, schemas, drift detection or delivery. ### Phase 2 · Extraction from stored bytes, against a declared shape Build: Write extraction as a pure function from a snapshot to rows, driven by a versioned spec holding selectors, a JSON Schema for the row and a declared row key. Validate every row against the schema. Run the whole thing over stored snapshots with the network unavailable. Trap: An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. Done when: The extraction test suite runs with the network blocked and passes, which is the assertion. Every row validates against the spec's schema and an invalid row is recorded with the failing field rather than dropped. Changing a selector and re-running produces new rows from existing snapshots with no fetches recorded. Each row carries the snapshot hash it came from. Do not build yet: No drift detection, delivery or scheduling yet. ### Phase 3 · Noticing that the page changed shape Build: Compare each run against the previous successful one: row count, null rate per field, type failure count, and the distribution of a numeric field you nominate. Breach a stated threshold and the run is quarantined, not delivered, with a diff showing which fields moved. Trap: The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. Done when: A run in which one field becomes constant across all rows is quarantined, tested with a doctored snapshot set. A run whose row count moves by more than the stated threshold is quarantined with the previous and current counts shown. Quarantined runs are never delivered, asserted by the delivery code path. The drift report names the fields that moved and by how much. Do not build yet: No delivery, replay or scheduling yet. ### Phase 4 · Delivery you can replay without duplicating Build: Deliver rows to one sink keyed by run and row key, recording each delivery in the same transaction where possible and immediately after otherwise. A replay skips keys already recorded. Offer a manual replay for a single run from the interface, with a dry run that lists what it would send. Trap: Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. Done when: Killing the process mid-delivery and replaying results in every row delivered exactly once, tested by counting at a fake sink. A dry run lists exactly the rows a real replay would send and sends none. A replay of a fully delivered run sends nothing and says so. Delivery of a quarantined run is refused with the drift reason shown. Do not build yet: No scheduling, storage ceilings or export yet. ### Phase 5 · Running unattended without a surprise bill Build: Schedule specs, with a per-run ceiling on requests, wall clock time and bytes fetched, and a global daily request budget per host. Prune snapshots on a stated retention while keeping the most recent per URL. Put every credential in the environment. A run page shows fetches, cache hits, robots refusals, quarantine state and the drift report. Export runs and rows as files. Trap: Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full. Done when: A run that reaches its request, time or byte ceiling stops and is marked as truncated rather than failing silently or continuing. Snapshot storage is deduplicated by content hash, verified by running the same spec twice and observing no growth. Retention pruning keeps the most recent snapshot per URL, tested. The run page shows fetches, cache hits, robots refusals and the drift report, and export produces files that reimport into an empty database. Do not build yet: Nothing after this. ### Out of scope (and why) - Anything behind a login, including session cookies copied from your browser, because those pages come with terms you personally accepted. - Data about identifiable people, which turns an engineering decision into a legal one and needs a lawful basis you probably do not have. - Rotating addresses, header spoofing or any attempt to look like a person, which is evasion rather than extraction and ends in a ban you deserve. - Hundreds of maintained connectors, which is a full-time maintenance treadmill; a handful of specs you own is the honest scope. - Durable execution across machines, which is the part the subscription actually pays for. ### README must contain - The exclusion rules as implemented, including that an unreachable or erroring robots.txt is treated as a complete disallow - That the cached exclusion file is never reused beyond 24 hours, and that pacing is your own choice because the standard defines no delay - The User-Agent string, including the contact address an administrator can write to - That extraction runs from stored snapshots with the network disabled, so a fixed selector never means a refetch - That a quarantined run cannot be delivered, and what thresholds cause quarantine
ARCHITECTURE.md
# Architecture · Import.io ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with one worker process and a scheduler, no distributed queue | durable execution at scale is the part you are deliberately not rebuilding | | Fetching | httpx with a per-host token bucket and conditional requests | pacing and caching are the whole of being a good citizen | | Storage | SQLite for runs and rows, compressed page snapshots on disk keyed by content hash | re-extraction should never mean refetching | | Extraction | Declared selectors in a versioned spec file, validated against a JSON Schema | an extractor without a declared shape cannot detect drift | | Delivery | One sink at a time, idempotent per run and row key | a half-delivered run that reruns is how duplicates reach a spreadsheet | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Robots | Per-host rules, caching and the deny cases. | Any parser that implements the standard, including the error branches. | | Fetcher | Pacing, conditional requests and snapshots. | Any client, provided every response is snapshotted by content hash. | | Extractor | Pure snapshot to rows, schema validated. | Any selector language, provided extraction never touches the network. | | Quality | Per-field statistics, drift and quarantine. | Any statistic, provided quarantine blocks delivery rather than warning. | | Delivery | Per row key idempotency and replay. | Any sink, provided the delivered set is recorded per key. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Runs, fetches, rows and deliveries. - `SNAPSHOT_DIR` · required · Compressed pages keyed by content hash. Watch its size. - `USER_AGENT` · required · Sent on every request. Must carry a contact address. - `DEFAULT_RATE_PER_SECOND` · required · Your own pacing, since the standard defines no delay. Lower it per host as needed. - `ROBOTS_CACHE_SECONDS` · required · Never longer than 24 hours, which is the ceiling the standard gives. - `RUN_MAX_REQUESTS` · required · Per run. Reaching it truncates the run rather than failing it. - `SNAPSHOT_RETENTION_DAYS` · required · The most recent snapshot per URL is kept regardless. - `DRIFT_ROW_COUNT_TOLERANCE` · required · Fractional change in row count that quarantines a run. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Import.io product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with one worker process and a scheduler, no distributed queue, httpx with a per-host token bucket and conditional requests, SQLite for runs and rows, compressed page snapshots on disk keyed by content hash, Declared selectors in a versioned spec file, validated against a JSON Schema, One sink at a time, idempotent per run and row key. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. - Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full.
MILESTONES.md
# Delivery milestones · Import.io Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A fetcher that obeys the protocol No request leaves the process until the exclusion rules for that host have been applied. ### Steps 1. Implement the exclusion rules including the deny cases The deny-on-error branch is the one every naive crawler omits. Files: `src/fetch/robots.py`, `tests/test_robots.py` ```sh python -m pytest tests/test_robots.py -q ``` 2. Pace per host with a token bucket and a real User-Agent The standard does not set a delay, so you set one and honour it. Files: `src/fetch/pacing.py`, `src/fetch/client.py` 3. Store snapshots by content hash with conditional requests So a broken extractor is fixed without touching the site again. Files: `src/store/snapshots.py` ### Done when - [ ] A server error on robots.txt blocks fetching and is recorded - [ ] A cache older than 24 hours is refetched - [ ] Disallow rules block and a missing file allows - [ ] An unchanged page costs one conditional request ### Watch out - The usual implementation treats an unreachable robots.txt as permission, because a timeout returns no rules and no rules reads as no restrictions, so the first thing your crawler does when a site is struggling is hammer it. Make unreachable and server-error results an explicit deny with the decision recorded per fetch, and make a cache older than 24 hours trigger a refetch rather than a reuse. ## M2 · Extraction from stored bytes, against a declared shape A spec declares selectors, types and a row key, and extraction never touches the network. ### Steps 1. Make extraction a pure function of a snapshot Testable, replayable and free to run a hundred times. Files: `src/extract/run.py`, `tests/test_extract.py` 2. Validate rows against the spec's schema A type error found now is a support question avoided later. Files: `src/extract/validate.py` 3. Block the network in the extraction tests The one assertion that keeps the split honest. Files: `tests/conftest.py` ```sh python -m pytest tests/test_extract.py -q --disable-socket ``` ### Done when - [ ] Extraction tests pass with networking disabled - [ ] Invalid rows are recorded with the failing field - [ ] A selector change re-runs with zero fetches - [ ] Every row carries its snapshot hash ### Watch out - An extractor that fetches as it parses cannot be tested, cannot be fixed without refetching, and quietly makes every experiment another load on the site. Split the two: fetch writes snapshots, extract reads them, and the extraction test suite runs with networking disabled so a stray request fails loudly. ## M3 · Noticing that the page changed shape A run whose output looks wrong is quarantined before anything downstream sees it. ### Steps 1. Store per-field statistics on every successful run You cannot diff against a baseline you never recorded. Files: `src/quality/stats.py` 2. Quarantine on a threshold breach, do not warn A warning in a log is a delivery of bad data with extra steps. Files: `src/quality/gate.py`, `tests/test_drift.py` 3. Flag a field that has become constant This is what a cookie banner looks like in a dataset. Files: `src/quality/stats.py` ### Done when - [ ] A constant field quarantines the run - [ ] A row count swing beyond the threshold quarantines the run - [ ] Quarantined runs are never delivered - [ ] The drift report names the moved fields ### Watch out - The failure that costs you is not a selector that stops matching, because that returns zero rows and is obvious; it is a selector that still matches and now returns the cookie banner text in every row, which delivers a full-looking dataset of rubbish. Compare against the previous run's field statistics, and treat a sudden jump in identical values across rows as a drift signal in its own right. ## M4 · Delivery you can replay without duplicating Re-running a half-delivered run sends only what did not arrive. ### Steps 1. Record delivery per row key, uniquely The unique index is what makes replay safe. Files: `src/deliver/sink.py`, `tests/test_replay.py` 2. Offer a dry run that lists and sends nothing Nobody presses replay confidently without one. Files: `src/deliver/dryrun.py` 3. Refuse delivery of a quarantined run The gate is worthless if the replay button bypasses it. Files: `src/deliver/sink.py` ### Done when - [ ] A mid-delivery crash plus replay yields one copy per row - [ ] A dry run sends nothing and lists accurately - [ ] Replaying a complete run sends nothing - [ ] Quarantined runs cannot be delivered ### Watch out - Delivering a run as one batch and marking the run delivered afterwards means a crash at row nine hundred replays all thousand rows into the spreadsheet, and the duplicates are found by whoever reads it. Record delivery per row key with a unique constraint, so a replay is naturally a difference rather than a repeat. ## M5 · Running unattended without a surprise bill (production only) Scheduled runs stay inside disk, time and request ceilings, and a failure is visible without reading a log. ### Steps 1. Enforce per-run and per-day ceilings An unattended crawler without a ceiling is a bill or a ban. Files: `src/schedule/limits.py`, `tests/test_limits.py` 2. Deduplicate and prune snapshots Keep the most recent per URL whatever the retention says. Files: `src/store/retention.py` 3. Show refusals and drift on the run page The two things you need and will not find in a log. Files: `src/web/templates/run.html` ### Done when - [ ] Ceilings truncate a run and mark it truncated - [ ] Repeating a spec adds no snapshot storage - [ ] Pruning keeps the most recent snapshot per URL - [ ] The run page shows refusals, cache hits and drift ### Watch out - Snapshots are the feature that makes replay possible and they are also what fills the disk, because a daily run over ten thousand pages keeps ten thousand copies a day until the machine stops. Deduplicate by content hash so an unchanged page costs nothing, set a retention, and alert on the snapshot directory crossing a stated size rather than on the disk being full.
OPERATIONS.md
# Operations · Import.io ## Backup The database and the snapshot directory together, since rows reference snapshot hashes and a restore of one without the other leaves a history you cannot replay. The database alone is enough to prove what was delivered. ## Restore Restore both, then mark any run left in the running state as failed rather than resuming it, and replay through the delivery table so nothing is sent twice. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Snapshot directory size, quarantined runs, robots refusals per host, truncated runs and the age of each host's cached rules. A growing refusal count usually means the site changed its rules and nobody read the diff. ## Incident checklist Duplicated rows downstream mean the delivery table was bypassed or the row key is not unique on the site. Rubbish rows delivered mean the drift thresholds are too loose, and the per-field statistics from the last good run show exactly which field moved. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A server error on robots.txt is confirmed to block fetching - [ ] Extraction is confirmed to pass with networking disabled - [ ] A constant field is confirmed to quarantine the run - [ ] A mid-delivery crash plus replay is confirmed to yield one copy per row - [ ] Repeating a spec is confirmed to add no snapshot storage ## Launch constraint Do not market omitted Import.io capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Runs, fetches, rows and deliveries. DATABASE_PATH=./data/pipeline.db # Required. Compressed pages keyed by content hash. Watch its size. SNAPSHOT_DIR=./data/snapshots # Required. Sent on every request. Must carry a contact address. USER_AGENT=example-extractor/1.0 (+mailto:you@example.com) # Required. Your own pacing, since the standard defines no delay. Lower it per host as needed. DEFAULT_RATE_PER_SECOND=1 # Required. Never longer than 24 hours, which is the ceiling the standard gives. ROBOTS_CACHE_SECONDS=86400 # Required. Per run. Reaching it truncates the run rather than failing it. RUN_MAX_REQUESTS=2000 # Required. The most recent snapshot per URL is kept regardless. SNAPSHOT_RETENTION_DAYS=30 # Required. Fractional change in row count that quarantines a run. DRIFT_ROW_COUNT_TOLERANCE=0.2
$ choose a build depth, inspect the files, then open the complete pack in your agent
Import.io: One automation is easy. Subscribers pay for maintained integrations, credentials, retries, observability, and someone else owning breakage.
xhundreds of maintained connectors
xOAuth app verification
xdurable execution at scale
xschema drift handling and enterprise controls
share on X ↗"I just replaced Import.io with one prompt"
Import.io pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| standard | $249 | $199 | 50,000 successful queries/month; $7 per additional 1,000 successful queries. |
| professional | $499 | $399 | 200,000 successful queries/month; $3.50 per additional 1,000 successful queries. |
| advanced | $899 | $699 | 500,000 successful queries/month; $2.50 per additional 1,000 successful queries. |
| managed web data | custom | — | Custom volume above 500,000 successful queries/month; managed extraction and delivery. |
| aperture pricing intelligence | custom | — | Custom product, competitor and pricing-intelligence scope; numeric limits not publicly listed. |
free tierno permanent free tier; 30-day trial includes 5,000 successful queries, requires no card, and hard-stops at the cap
billingmonthly + annual; annual commitment is about 20% lower than monthly
hidden costsSuccessful-query overages are automatic and tiered at $7, $3.50 or $2.50 per 1,000; managed services and custom delivery are separately quoted.
verified 2026-08-11 · source ↗
Vibecode Import.io
Kinda. The core of Import.io is buildable in a weekend with the prompt on this page, but there are real gaps: hundreds of maintained connectors, OAuth app verification. Read the honest list above before committing.
How much does Import.io cost?
Import.io's pricing is usage-based or varies by plan · The pricing model is not safely comparable as one fixed monthly USD amount. Review the official source before publication..
What do I lose by replacing Import.io?
Honestly: hundreds of maintained connectors; OAuth app verification; durable execution at scale; schema drift handling and enterprise controls. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Import.io?
Yes: Activepieces (Open-source automation builder with connectors and self-hosting.), n8n (Source-available workflow automation engine and connector reference.). Using prior art is also vibecoding; the prompt is for when you want it exactly your way.