Vibecode Browse AI
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
price $19/moyou'd save $228/yrbuild time weekend to multi-daycategory 🤖 automationreplaced by 0 people
The visible web data extraction loop is buildable, but a credible replacement needs more than the first screen. Browse AI earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Browse AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Browse AI · indie build A visual scraper builder where the extraction is the easy half. The hard half is selectors that survive a site redesign, change detection that does not fire on a rotating advertisement, and being a polite enough client that the sites you depend on do not block you. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Browser | A headless browser for pages that need one, plain requests where they do not | running a browser per page is expensive and most pages do not need it | | Selection | Selectors with several fallback strategies per field | one brittle path breaks on the first redesign and takes the robot with it | | Politeness | Per-host rate limits, stated rules honoured, honest identification | the cost you impose is on someone else | | Change detection | Normalised comparison ignoring known-volatile regions | a rotating advertisement will otherwise fire an alert every run | | Runtime | TypeScript with PostgreSQL and a worker queue | scheduling, retries and history are the actual system | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the server, the workers and the headless browser driver. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally - Why: Runs, results and change history with per-host state. - Get it: Docker or your package manager. - Verify: psql connects and creates a table - [ ] **A headless browser** · free - Why: Some pages render their content in the browser and cannot be fetched as documents, though many can and running a browser for those is waste. - Get it: A maintained headless browser driver, plus a way to decide per site whether one is needed. - Verify: A page requiring rendering extracts correctly and a static page is fetched without the browser - [ ] **Whether you may scrape each target** · free to read; advice if this is commercial and contested - Why: The site's terms, whether the data is personal, and whether you are behind a login are three separate questions with different answers, and all three belong before the first run. - Get it: For each target, read the terms and the stated crawling rules, note whether the data includes personal information, and write down your conclusion. - Verify: A written note per target states the terms position, the personal data position and the login position - [ ] **Three real target sites** · free - Why: Selector fragility, rendering requirements and volatile regions only appear on real sites, and one of the three should be a site that has been redesigned so you can test fallbacks. - Get it: Pick sites you have a legitimate reason to monitor, ideally including one whose old markup you have saved. - Verify: Three targets exist, one static, one requiring rendering, one with rotating content ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Evasion of any kind. - Scraping behind a login. - Breadth of site coverage. - Legal advice about a site's terms. - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors If one of those is essential to you, that is the reason to keep paying for Browse AI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Browse AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled web extraction robots that stay within what a site actually permits to replace Browse AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero A scraper is a program that imposes cost on someone else's server to take data they published for people to read. That can be entirely legitimate and it can be a nuisance or worse, and which one it is depends on decisions you make before writing any code. The ones this build makes for you: respect the site's stated crawling rules, identify yourself honestly in the user agent with a way to be contacted, rate limit conservatively per host, honour an explicit instruction to back off, and stop entirely after repeated refusals rather than working around them. The ones you have to make: whether the site's terms permit automated access, whether the data includes personal information and what obligations that carries, and whether you are behind a login, which changes the legal picture considerably. Write down your answers before the first run. The build deliberately does not include evasion. No rotating identities to look like different people, no defeating access controls, no pretending to be a browser you are not. If a site has made it clear it does not want automated access, the answer is to stop, not to be harder to detect. ### Stack (fixed, do not substitute) - Browser: A headless browser for pages that need one, plain requests where they do not · running a browser per page is expensive and most pages do not need it - Selection: Selectors with several fallback strategies per field · one brittle path breaks on the first redesign and takes the robot with it - Politeness: Per-host rate limits, stated rules honoured, honest identification · the cost you impose is on someone else - Change detection: Normalised comparison ignoring known-volatile regions · a rotating advertisement will otherwise fire an alert every run - Runtime: TypeScript with PostgreSQL and a worker queue · scheduling, retries and history are the actual system ### Data model (create this before Phase 1) - robots(id, name, start_url, host, schedule, enabled, created_at, disabled_reason) - fields(id, robot_id, name, selector_strategies_json, required, last_successful_strategy) - runs(id, robot_id, started_at, ended_at, status, pages_fetched, bytes, outcome, error) - results(id, run_id, row_ordinal, data_json, content_hash) - changes(id, robot_id, field, before, after, detected_at, significant) · significant excludes known-volatile regions - host_policy(host, crawl_delay_ms, rules_fetched_at, disallowed_paths_json, backoff_until) - A robot records which selector strategy last worked, so a redesign is visible as a fallback being used rather than as silent success. ### Phase 1 · Being a client sites do not mind Build: Fetch and cache each host's stated crawling rules and honour them per path. Identify the client honestly with a contact route. Rate limit per host with a conservative default. Honour an explicit backoff instruction over your own schedule. Trap: Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. Done when: Stated crawling rules are fetched, cached with an expiry and honoured per path. A path becoming disallowed disables the robot and reports why, tested by changing the rules. The user agent identifies the client and a contact route. An explicit backoff instruction is honoured exactly rather than overridden by the schedule. Do not build yet: No extraction, scheduling or change detection. ### Phase 2 · Selectors that survive a redesign Build: Capture several selector strategies per field when the robot is built: structural path, nearby label text, and attribute patterns. Try them in order and record which worked. Report a fallback being used rather than treating it as success. Trap: A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. Done when: Each field has several strategies recorded at build time. A changed structural path falls back to a label-based strategy, tested by modifying a fixture page. The run reports which strategy was used and warns when it was not the primary. A field matching nothing fails the run rather than recording an empty value. Do not build yet: No scheduling, change detection or delivery yet. ### Phase 3 · Running on a schedule without piling up Build: A worker queue with per-host concurrency limits, retries with backoff, and a cap on run duration. Isolate failures per robot. Detect a robot that has failed repeatedly and disable it rather than retrying forever. Trap: Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. Done when: Twenty robots on one host issue requests serially at the host's crawl delay, verified from the run log. A slow site does not block robots on other hosts. A repeatedly failing robot is disabled with a reason. A run exceeding its duration cap is terminated rather than running indefinitely. Do not build yet: No change detection or delivery yet. ### Phase 4 · Noticing real changes only Build: Compare runs on normalised values, excluding regions marked volatile. Let the user mark a region as volatile from an alert they did not want. Report changes with before and after values rather than only that something differed. Trap: Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. Done when: A page with a rotating advertisement and a timestamp does not fire an alert, tested across ten runs. Marking a region volatile from an alert prevents future alerts on it. Changes report before and after values. A genuine content change fires reliably, tested by editing a fixture. Do not build yet: No delivery or export yet. ### Phase 5 · Delivery, and what you collected Build: Deliver results by webhook, file or scheduled export with retries. Flag fields that look like personal data at build time and record the decision. Retain runs and results on a stated policy. Trap: Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each. Done when: Results deliver by webhook with retries and a signature. A field that looks like personal data is flagged at build time and requires a recorded decision. Retention deletes old runs and results on the stated policy. Export contains robots, fields and results and re-imports. Do not build yet: Nothing after this. ### Out of scope (and why) - Any form of evasion: rotating identities, defeating access controls, or bypassing measures a site has taken to stop automated access. - Scraping behind a login, which changes the legal picture substantially and is not something to do casually. - Breadth of site coverage, which is what the hosted products actually maintain and which needs constant attention. - Legal advice about whether a particular site's terms permit what you intend. ### README must contain - That stated crawling rules are honoured and re-fetched, and a newly disallowed path disables the robot - That the user agent identifies the client with a way to be contacted - That concurrency is limited per host regardless of how many robots want to run - That a fallback selector succeeding is reported as a warning rather than a clean run - That the build contains no evasion, deliberately ===== AGENTS.md ===== # Agent instructions · Browse AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A headless browser for pages that need one, plain requests where they do not, Selectors with several fallback strategies per field, Per-host rate limits, stated rules honoured, honest identification, Normalised comparison ignoring known-volatile regions, TypeScript with PostgreSQL and a worker queue. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. ===== BUILD_PLAN.md ===== # Build plan · Browse AI A visual scraper builder where the extraction is the easy half. The hard half is selectors that survive a site redesign, change detection that does not fire on a rotating advertisement, and being a polite enough client that the sites you depend on do not block you. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Being a client sites do not mind Your fetching respects what each host has said and asked. ### Steps 1. Fetch and re-fetch stated rules With an expiry. A permanent cache ignores a site changing its mind. Files: `src/policy.ts` 2. Identify honestly with a contact So an operator with a problem can reach you instead of blocking you. Files: `src/fetch.ts` 3. Honour backoff over your schedule An explicit instruction outranks your plan. Always. Files: `src/ratelimit.ts`, `src/ratelimit.test.ts` ### Done when - [ ] Stated rules are honoured per path and re-fetched - [ ] A newly disallowed path disables the robot - [ ] The user agent identifies the client and a contact - [ ] Backoff instructions are honoured exactly ### Watch out - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. ## Phase 2 · Selectors that survive a redesign A robot keeps working after the site changes its markup. ### Steps 1. Capture alternatives at build time Structure, label proximity, attributes. One path is not enough. Files: `src/selectors.ts` 2. Record which strategy worked A fallback succeeding is a warning, not a clean run. Files: `src/extract.ts` 3. Fail rather than recording empty An empty value that looks like data is worse than a failure. Files: `src/extract.ts` ### Done when - [ ] Each field has several strategies - [ ] A changed path falls back successfully - [ ] The strategy used is reported and warned on - [ ] A field matching nothing fails the run ### Watch out - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. ## Phase 3 · Running on a schedule without piling up Robots run reliably and a slow site does not stop everything. ### Steps 1. Limit concurrency per host Not globally. Twenty robots on one site is an attack from their side. Files: `src/queue.ts` 2. Isolate failures per robot One dead site must not stop the others. Files: `src/worker.ts` 3. Disable persistent failures And say why. Retrying a dead robot forever helps nobody. Files: `src/health.ts` ### Done when - [ ] Twenty robots on one host run serially at its delay - [ ] A slow host does not block others - [ ] A repeatedly failing robot is disabled with a reason - [ ] Runs are terminated at a duration cap ### Watch out - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. ## Phase 4 · Noticing real changes only An alert means something changed that you care about. ### Steps 1. Normalise before comparing Whitespace, ordering, formatting. Raw comparison fires constantly. Files: `src/normalise.ts` 2. Let volatile regions be marked from the alert One click, at the moment of annoyance. Otherwise it never happens. Files: `src/volatile.ts` 3. Report before and after Something changed is not information. Files: `src/changes.ts` ### Done when - [ ] Rotating content does not fire alerts - [ ] Marking volatile from an alert works - [ ] Changes report before and after values - [ ] A genuine change fires reliably ### Watch out - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. ## Not in this build - Evasion of any kind. - Scraping behind a login. - Breadth of site coverage. - Legal advice about a site's terms. ## After v1, if you want it - A diff view across runs, which is what makes change history worth keeping - Robot templates for common page shapes, once selectors have proven durable - A shared host policy cache, so several robots on one site cooperate rather than compete ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Robots, runs and extracted results, which may include personal data. DATABASE_URL=postgres://localhost:5432/robots # Required. Identifies the client with a contact route. Anonymous clients get blocked. USER_AGENT=MyRobot/1.0 (+https://example.com/robot) # Required. Per host, unless the host states its own. Conservative by default. DEFAULT_CRAWL_DELAY_MS=5000 # Required. Regardless of how many robots target it. Twenty at once looks like an attack. MAX_CONCURRENCY_PER_HOST=1 # Required. A run that will not finish should be stopped rather than left going. RUN_DURATION_CAP_SECONDS=600 # Required. Present as a constant so the answer is visible. There is no other value. EVASION_FEATURES=none
You are building a lean indie version of Browse AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Browse AI · indie build A visual scraper builder where the extraction is the easy half. The hard half is selectors that survive a site redesign, change detection that does not fire on a rotating advertisement, and being a polite enough client that the sites you depend on do not block you. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Browser | A headless browser for pages that need one, plain requests where they do not | running a browser per page is expensive and most pages do not need it | | Selection | Selectors with several fallback strategies per field | one brittle path breaks on the first redesign and takes the robot with it | | Politeness | Per-host rate limits, stated rules honoured, honest identification | the cost you impose is on someone else | | Change detection | Normalised comparison ignoring known-volatile regions | a rotating advertisement will otherwise fire an alert every run | | Runtime | TypeScript with PostgreSQL and a worker queue | scheduling, retries and history are the actual system | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the server, the workers and the headless browser driver. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally - Why: Runs, results and change history with per-host state. - Get it: Docker or your package manager. - Verify: psql connects and creates a table - [ ] **A headless browser** · free - Why: Some pages render their content in the browser and cannot be fetched as documents, though many can and running a browser for those is waste. - Get it: A maintained headless browser driver, plus a way to decide per site whether one is needed. - Verify: A page requiring rendering extracts correctly and a static page is fetched without the browser - [ ] **Whether you may scrape each target** · free to read; advice if this is commercial and contested - Why: The site's terms, whether the data is personal, and whether you are behind a login are three separate questions with different answers, and all three belong before the first run. - Get it: For each target, read the terms and the stated crawling rules, note whether the data includes personal information, and write down your conclusion. - Verify: A written note per target states the terms position, the personal data position and the login position - [ ] **Three real target sites** · free - Why: Selector fragility, rendering requirements and volatile regions only appear on real sites, and one of the three should be a site that has been redesigned so you can test fallbacks. - Get it: Pick sites you have a legitimate reason to monitor, ideally including one whose old markup you have saved. - Verify: Three targets exist, one static, one requiring rendering, one with rotating content ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Evasion of any kind. - Scraping behind a login. - Breadth of site coverage. - Legal advice about a site's terms. - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors If one of those is essential to you, that is the reason to keep paying for Browse AI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Browse AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled web extraction robots that stay within what a site actually permits to replace Browse AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero A scraper is a program that imposes cost on someone else's server to take data they published for people to read. That can be entirely legitimate and it can be a nuisance or worse, and which one it is depends on decisions you make before writing any code. The ones this build makes for you: respect the site's stated crawling rules, identify yourself honestly in the user agent with a way to be contacted, rate limit conservatively per host, honour an explicit instruction to back off, and stop entirely after repeated refusals rather than working around them. The ones you have to make: whether the site's terms permit automated access, whether the data includes personal information and what obligations that carries, and whether you are behind a login, which changes the legal picture considerably. Write down your answers before the first run. The build deliberately does not include evasion. No rotating identities to look like different people, no defeating access controls, no pretending to be a browser you are not. If a site has made it clear it does not want automated access, the answer is to stop, not to be harder to detect. ### Stack (fixed, do not substitute) - Browser: A headless browser for pages that need one, plain requests where they do not · running a browser per page is expensive and most pages do not need it - Selection: Selectors with several fallback strategies per field · one brittle path breaks on the first redesign and takes the robot with it - Politeness: Per-host rate limits, stated rules honoured, honest identification · the cost you impose is on someone else - Change detection: Normalised comparison ignoring known-volatile regions · a rotating advertisement will otherwise fire an alert every run - Runtime: TypeScript with PostgreSQL and a worker queue · scheduling, retries and history are the actual system ### Data model (create this before Phase 1) - robots(id, name, start_url, host, schedule, enabled, created_at, disabled_reason) - fields(id, robot_id, name, selector_strategies_json, required, last_successful_strategy) - runs(id, robot_id, started_at, ended_at, status, pages_fetched, bytes, outcome, error) - results(id, run_id, row_ordinal, data_json, content_hash) - changes(id, robot_id, field, before, after, detected_at, significant) · significant excludes known-volatile regions - host_policy(host, crawl_delay_ms, rules_fetched_at, disallowed_paths_json, backoff_until) - A robot records which selector strategy last worked, so a redesign is visible as a fallback being used rather than as silent success. ### Phase 1 · Being a client sites do not mind Build: Fetch and cache each host's stated crawling rules and honour them per path. Identify the client honestly with a contact route. Rate limit per host with a conservative default. Honour an explicit backoff instruction over your own schedule. Trap: Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. Done when: Stated crawling rules are fetched, cached with an expiry and honoured per path. A path becoming disallowed disables the robot and reports why, tested by changing the rules. The user agent identifies the client and a contact route. An explicit backoff instruction is honoured exactly rather than overridden by the schedule. Do not build yet: No extraction, scheduling or change detection. ### Phase 2 · Selectors that survive a redesign Build: Capture several selector strategies per field when the robot is built: structural path, nearby label text, and attribute patterns. Try them in order and record which worked. Report a fallback being used rather than treating it as success. Trap: A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. Done when: Each field has several strategies recorded at build time. A changed structural path falls back to a label-based strategy, tested by modifying a fixture page. The run reports which strategy was used and warns when it was not the primary. A field matching nothing fails the run rather than recording an empty value. Do not build yet: No scheduling, change detection or delivery yet. ### Phase 3 · Running on a schedule without piling up Build: A worker queue with per-host concurrency limits, retries with backoff, and a cap on run duration. Isolate failures per robot. Detect a robot that has failed repeatedly and disable it rather than retrying forever. Trap: Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. Done when: Twenty robots on one host issue requests serially at the host's crawl delay, verified from the run log. A slow site does not block robots on other hosts. A repeatedly failing robot is disabled with a reason. A run exceeding its duration cap is terminated rather than running indefinitely. Do not build yet: No change detection or delivery yet. ### Phase 4 · Noticing real changes only Build: Compare runs on normalised values, excluding regions marked volatile. Let the user mark a region as volatile from an alert they did not want. Report changes with before and after values rather than only that something differed. Trap: Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. Done when: A page with a rotating advertisement and a timestamp does not fire an alert, tested across ten runs. Marking a region volatile from an alert prevents future alerts on it. Changes report before and after values. A genuine content change fires reliably, tested by editing a fixture. Do not build yet: No delivery or export yet. ### Phase 5 · Delivery, and what you collected Build: Deliver results by webhook, file or scheduled export with retries. Flag fields that look like personal data at build time and record the decision. Retain runs and results on a stated policy. Trap: Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each. Done when: Results deliver by webhook with retries and a signature. A field that looks like personal data is flagged at build time and requires a recorded decision. Retention deletes old runs and results on the stated policy. Export contains robots, fields and results and re-imports. Do not build yet: Nothing after this. ### Out of scope (and why) - Any form of evasion: rotating identities, defeating access controls, or bypassing measures a site has taken to stop automated access. - Scraping behind a login, which changes the legal picture substantially and is not something to do casually. - Breadth of site coverage, which is what the hosted products actually maintain and which needs constant attention. - Legal advice about whether a particular site's terms permit what you intend. ### README must contain - That stated crawling rules are honoured and re-fetched, and a newly disallowed path disables the robot - That the user agent identifies the client with a way to be contacted - That concurrency is limited per host regardless of how many robots want to run - That a fallback selector succeeding is reported as a warning rather than a clean run - That the build contains no evasion, deliberately ===== AGENTS.md ===== # Agent instructions · Browse AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A headless browser for pages that need one, plain requests where they do not, Selectors with several fallback strategies per field, Per-host rate limits, stated rules honoured, honest identification, Normalised comparison ignoring known-volatile regions, TypeScript with PostgreSQL and a worker queue. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. ===== BUILD_PLAN.md ===== # Build plan · Browse AI A visual scraper builder where the extraction is the easy half. The hard half is selectors that survive a site redesign, change detection that does not fire on a rotating advertisement, and being a polite enough client that the sites you depend on do not block you. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Being a client sites do not mind Your fetching respects what each host has said and asked. ### Steps 1. Fetch and re-fetch stated rules With an expiry. A permanent cache ignores a site changing its mind. Files: `src/policy.ts` 2. Identify honestly with a contact So an operator with a problem can reach you instead of blocking you. Files: `src/fetch.ts` 3. Honour backoff over your schedule An explicit instruction outranks your plan. Always. Files: `src/ratelimit.ts`, `src/ratelimit.test.ts` ### Done when - [ ] Stated rules are honoured per path and re-fetched - [ ] A newly disallowed path disables the robot - [ ] The user agent identifies the client and a contact - [ ] Backoff instructions are honoured exactly ### Watch out - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. ## Phase 2 · Selectors that survive a redesign A robot keeps working after the site changes its markup. ### Steps 1. Capture alternatives at build time Structure, label proximity, attributes. One path is not enough. Files: `src/selectors.ts` 2. Record which strategy worked A fallback succeeding is a warning, not a clean run. Files: `src/extract.ts` 3. Fail rather than recording empty An empty value that looks like data is worse than a failure. Files: `src/extract.ts` ### Done when - [ ] Each field has several strategies - [ ] A changed path falls back successfully - [ ] The strategy used is reported and warned on - [ ] A field matching nothing fails the run ### Watch out - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. ## Phase 3 · Running on a schedule without piling up Robots run reliably and a slow site does not stop everything. ### Steps 1. Limit concurrency per host Not globally. Twenty robots on one site is an attack from their side. Files: `src/queue.ts` 2. Isolate failures per robot One dead site must not stop the others. Files: `src/worker.ts` 3. Disable persistent failures And say why. Retrying a dead robot forever helps nobody. Files: `src/health.ts` ### Done when - [ ] Twenty robots on one host run serially at its delay - [ ] A slow host does not block others - [ ] A repeatedly failing robot is disabled with a reason - [ ] Runs are terminated at a duration cap ### Watch out - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. ## Phase 4 · Noticing real changes only An alert means something changed that you care about. ### Steps 1. Normalise before comparing Whitespace, ordering, formatting. Raw comparison fires constantly. Files: `src/normalise.ts` 2. Let volatile regions be marked from the alert One click, at the moment of annoyance. Otherwise it never happens. Files: `src/volatile.ts` 3. Report before and after Something changed is not information. Files: `src/changes.ts` ### Done when - [ ] Rotating content does not fire alerts - [ ] Marking volatile from an alert works - [ ] Changes report before and after values - [ ] A genuine change fires reliably ### Watch out - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. ## Not in this build - Evasion of any kind. - Scraping behind a login. - Breadth of site coverage. - Legal advice about a site's terms. ## After v1, if you want it - A diff view across runs, which is what makes change history worth keeping - Robot templates for common page shapes, once selectors have proven durable - A shared host policy cache, so several robots on one site cooperate rather than compete ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Robots, runs and extracted results, which may include personal data. DATABASE_URL=postgres://localhost:5432/robots # Required. Identifies the client with a contact route. Anonymous clients get blocked. USER_AGENT=MyRobot/1.0 (+https://example.com/robot) # Required. Per host, unless the host states its own. Conservative by default. DEFAULT_CRAWL_DELAY_MS=5000 # Required. Regardless of how many robots target it. Twenty at once looks like an attack. MAX_CONCURRENCY_PER_HOST=1 # Required. A run that will not finish should be stopped rather than left going. RUN_DURATION_CAP_SECONDS=600 # Required. Present as a constant so the answer is visible. There is no other value. EVASION_FEATURES=none
You are building a production product version of Browse AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Browse AI · product brief ## Problem The visible web data extraction loop is buildable, but a credible replacement needs more than the first screen. Browse AI earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A scraper that sites do not need to block, whose robots survive a redesign visibly rather than silently, and whose alerts mean something actually changed. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - hosted PostgreSQL and Redis - API credentials for the two chosen services - Node.js 22 ## Explicit non-goals for v1 - Evasion of any kind. - Scraping behind a login. - Breadth of site coverage. - Legal advice about a site's terms. - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors ## Success criteria - Stated rules are confirmed honoured and re-fetched - A newly disallowed path is confirmed to disable the robot - Twenty robots on one host are confirmed to run serially - A fallback strategy is confirmed reported as a warning - Rotating content is confirmed not to fire alerts ===== BRIEF.md ===== # Build brief · Browse AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled web extraction robots that stay within what a site actually permits to replace Browse AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero A scraper is a program that imposes cost on someone else's server to take data they published for people to read. That can be entirely legitimate and it can be a nuisance or worse, and which one it is depends on decisions you make before writing any code. The ones this build makes for you: respect the site's stated crawling rules, identify yourself honestly in the user agent with a way to be contacted, rate limit conservatively per host, honour an explicit instruction to back off, and stop entirely after repeated refusals rather than working around them. The ones you have to make: whether the site's terms permit automated access, whether the data includes personal information and what obligations that carries, and whether you are behind a login, which changes the legal picture considerably. Write down your answers before the first run. The build deliberately does not include evasion. No rotating identities to look like different people, no defeating access controls, no pretending to be a browser you are not. If a site has made it clear it does not want automated access, the answer is to stop, not to be harder to detect. ### Stack (fixed, do not substitute) - Browser: A headless browser for pages that need one, plain requests where they do not · running a browser per page is expensive and most pages do not need it - Selection: Selectors with several fallback strategies per field · one brittle path breaks on the first redesign and takes the robot with it - Politeness: Per-host rate limits, stated rules honoured, honest identification · the cost you impose is on someone else - Change detection: Normalised comparison ignoring known-volatile regions · a rotating advertisement will otherwise fire an alert every run - Runtime: TypeScript with PostgreSQL and a worker queue · scheduling, retries and history are the actual system ### Data model (create this before Phase 1) - robots(id, name, start_url, host, schedule, enabled, created_at, disabled_reason) - fields(id, robot_id, name, selector_strategies_json, required, last_successful_strategy) - runs(id, robot_id, started_at, ended_at, status, pages_fetched, bytes, outcome, error) - results(id, run_id, row_ordinal, data_json, content_hash) - changes(id, robot_id, field, before, after, detected_at, significant) · significant excludes known-volatile regions - host_policy(host, crawl_delay_ms, rules_fetched_at, disallowed_paths_json, backoff_until) - A robot records which selector strategy last worked, so a redesign is visible as a fallback being used rather than as silent success. ### Phase 1 · Being a client sites do not mind Build: Fetch and cache each host's stated crawling rules and honour them per path. Identify the client honestly with a contact route. Rate limit per host with a conservative default. Honour an explicit backoff instruction over your own schedule. Trap: Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. Done when: Stated crawling rules are fetched, cached with an expiry and honoured per path. A path becoming disallowed disables the robot and reports why, tested by changing the rules. The user agent identifies the client and a contact route. An explicit backoff instruction is honoured exactly rather than overridden by the schedule. Do not build yet: No extraction, scheduling or change detection. ### Phase 2 · Selectors that survive a redesign Build: Capture several selector strategies per field when the robot is built: structural path, nearby label text, and attribute patterns. Try them in order and record which worked. Report a fallback being used rather than treating it as success. Trap: A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. Done when: Each field has several strategies recorded at build time. A changed structural path falls back to a label-based strategy, tested by modifying a fixture page. The run reports which strategy was used and warns when it was not the primary. A field matching nothing fails the run rather than recording an empty value. Do not build yet: No scheduling, change detection or delivery yet. ### Phase 3 · Running on a schedule without piling up Build: A worker queue with per-host concurrency limits, retries with backoff, and a cap on run duration. Isolate failures per robot. Detect a robot that has failed repeatedly and disable it rather than retrying forever. Trap: Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. Done when: Twenty robots on one host issue requests serially at the host's crawl delay, verified from the run log. A slow site does not block robots on other hosts. A repeatedly failing robot is disabled with a reason. A run exceeding its duration cap is terminated rather than running indefinitely. Do not build yet: No change detection or delivery yet. ### Phase 4 · Noticing real changes only Build: Compare runs on normalised values, excluding regions marked volatile. Let the user mark a region as volatile from an alert they did not want. Report changes with before and after values rather than only that something differed. Trap: Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. Done when: A page with a rotating advertisement and a timestamp does not fire an alert, tested across ten runs. Marking a region volatile from an alert prevents future alerts on it. Changes report before and after values. A genuine content change fires reliably, tested by editing a fixture. Do not build yet: No delivery or export yet. ### Phase 5 · Delivery, and what you collected Build: Deliver results by webhook, file or scheduled export with retries. Flag fields that look like personal data at build time and record the decision. Retain runs and results on a stated policy. Trap: Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each. Done when: Results deliver by webhook with retries and a signature. A field that looks like personal data is flagged at build time and requires a recorded decision. Retention deletes old runs and results on the stated policy. Export contains robots, fields and results and re-imports. Do not build yet: Nothing after this. ### Out of scope (and why) - Any form of evasion: rotating identities, defeating access controls, or bypassing measures a site has taken to stop automated access. - Scraping behind a login, which changes the legal picture substantially and is not something to do casually. - Breadth of site coverage, which is what the hosted products actually maintain and which needs constant attention. - Legal advice about whether a particular site's terms permit what you intend. ### README must contain - That stated crawling rules are honoured and re-fetched, and a newly disallowed path disables the robot - That the user agent identifies the client with a way to be contacted - That concurrency is limited per host regardless of how many robots want to run - That a fallback selector succeeding is reported as a warning rather than a clean run - That the build contains no evasion, deliberately ===== ARCHITECTURE.md ===== # Architecture · Browse AI ## Stack | Part | Choice | Why | | --- | --- | --- | | Browser | A headless browser for pages that need one, plain requests where they do not | running a browser per page is expensive and most pages do not need it | | Selection | Selectors with several fallback strategies per field | one brittle path breaks on the first redesign and takes the robot with it | | Politeness | Per-host rate limits, stated rules honoured, honest identification | the cost you impose is on someone else | | Change detection | Normalised comparison ignoring known-volatile regions | a rotating advertisement will otherwise fire an alert every run | | Runtime | TypeScript with PostgreSQL and a worker queue | scheduling, retries and history are the actual system | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Policy | Stated rules, delays and backoff. | Any implementation, provided rules are re-fetched and an explicit backoff outranks the schedule. | | Selection | Multiple strategies and fallback reporting. | Any strategy set, provided a fallback is reported rather than treated as success. | | Queue | Per-host concurrency and isolation. | Any queue, provided the limit is per host rather than global. | | Change detection | Normalised comparison and volatile regions. | Any comparison, provided volatile regions can be marked from an alert. | | Delivery | Retried, signed delivery and personal data flags. | Any transport, provided personal fields are flagged when the robot is built. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Robots, runs and extracted results, which may include personal data. - `USER_AGENT` · required · Identifies the client with a contact route. Anonymous clients get blocked. - `DEFAULT_CRAWL_DELAY_MS` · required · Per host, unless the host states its own. Conservative by default. - `MAX_CONCURRENCY_PER_HOST` · required · Regardless of how many robots target it. Twenty at once looks like an attack. - `RUN_DURATION_CAP_SECONDS` · required · A run that will not finish should be stopped rather than left going. - `EVASION_FEATURES` · required · Present as a constant so the answer is visible. There is no other value. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Browse AI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A headless browser for pages that need one, plain requests where they do not, Selectors with several fallback strategies per field, Per-host rate limits, stated rules honoured, honest identification, Normalised comparison ignoring known-volatile regions, TypeScript with PostgreSQL and a worker queue. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. - Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each. ===== MILESTONES.md ===== # Delivery milestones · Browse AI Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Being a client sites do not mind Your fetching respects what each host has said and asked. ### Steps 1. Fetch and re-fetch stated rules With an expiry. A permanent cache ignores a site changing its mind. Files: `src/policy.ts` 2. Identify honestly with a contact So an operator with a problem can reach you instead of blocking you. Files: `src/fetch.ts` 3. Honour backoff over your schedule An explicit instruction outranks your plan. Always. Files: `src/ratelimit.ts`, `src/ratelimit.test.ts` ### Done when - [ ] Stated rules are honoured per path and re-fetched - [ ] A newly disallowed path disables the robot - [ ] The user agent identifies the client and a contact - [ ] Backoff instructions are honoured exactly ### Watch out - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. ## M2 · Selectors that survive a redesign A robot keeps working after the site changes its markup. ### Steps 1. Capture alternatives at build time Structure, label proximity, attributes. One path is not enough. Files: `src/selectors.ts` 2. Record which strategy worked A fallback succeeding is a warning, not a clean run. Files: `src/extract.ts` 3. Fail rather than recording empty An empty value that looks like data is worse than a failure. Files: `src/extract.ts` ### Done when - [ ] Each field has several strategies - [ ] A changed path falls back successfully - [ ] The strategy used is reported and warned on - [ ] A field matching nothing fails the run ### Watch out - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. ## M3 · Running on a schedule without piling up Robots run reliably and a slow site does not stop everything. ### Steps 1. Limit concurrency per host Not globally. Twenty robots on one site is an attack from their side. Files: `src/queue.ts` 2. Isolate failures per robot One dead site must not stop the others. Files: `src/worker.ts` 3. Disable persistent failures And say why. Retrying a dead robot forever helps nobody. Files: `src/health.ts` ### Done when - [ ] Twenty robots on one host run serially at its delay - [ ] A slow host does not block others - [ ] A repeatedly failing robot is disabled with a reason - [ ] Runs are terminated at a duration cap ### Watch out - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. ## M4 · Noticing real changes only An alert means something changed that you care about. ### Steps 1. Normalise before comparing Whitespace, ordering, formatting. Raw comparison fires constantly. Files: `src/normalise.ts` 2. Let volatile regions be marked from the alert One click, at the moment of annoyance. Otherwise it never happens. Files: `src/volatile.ts` 3. Report before and after Something changed is not information. Files: `src/changes.ts` ### Done when - [ ] Rotating content does not fire alerts - [ ] Marking volatile from an alert works - [ ] Changes report before and after values - [ ] A genuine change fires reliably ### Watch out - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. ## M5 · Delivery, and what you collected (production only) Results reach where you need them and you know what you are holding. ### Steps 1. Deliver with retries and signing A receiving endpoint being down should not lose the run. Files: `src/deliver.ts` 2. Flag likely personal fields at build When someone is looking. Not at scale, six months later. Files: `src/personal.ts` 3. Retain on a stated policy Scraped data accumulates. Decide how long before it does. Files: `src/retention.ts` ### Done when - [ ] Webhook delivery retries and is signed - [ ] Personal-looking fields are flagged at build time - [ ] Retention deletes runs on the stated policy - [ ] Export round trips robots, fields and results ### Watch out - Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each. ===== OPERATIONS.md ===== # Operations · Browse AI ## Backup Robot definitions are the work; results are usually reproducible by running again, subject to the site still having the data. Back up definitions and change history, and apply retention to results rather than accumulating them. ## Restore Restore and confirm host policy caches are expired rather than restored stale, since a restored cache may permit a path the site has since disallowed. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Fallback selector usage and per-host error rates are the health metrics. A robot silently running on its third fallback strategy is one redesign from producing wrong data. ## Incident checklist Being blocked by a site is the incident, and the cause is nearly always concurrency or ignoring a stated rule. The run log records what your client actually did, which is what lets you answer an operator rather than guess. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Stated rules are confirmed honoured and re-fetched - [ ] A newly disallowed path is confirmed to disable the robot - [ ] Twenty robots on one host are confirmed to run serially - [ ] A fallback strategy is confirmed reported as a warning - [ ] Rotating content is confirmed not to fire alerts ## Launch constraint Do not market omitted Browse AI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Robots, runs and extracted results, which may include personal data. DATABASE_URL=postgres://localhost:5432/robots # Required. Identifies the client with a contact route. Anonymous clients get blocked. USER_AGENT=MyRobot/1.0 (+https://example.com/robot) # Required. Per host, unless the host states its own. Conservative by default. DEFAULT_CRAWL_DELAY_MS=5000 # Required. Regardless of how many robots target it. Twenty at once looks like an attack. MAX_CONCURRENCY_PER_HOST=1 # Required. A run that will not finish should be stopped rather than left going. RUN_DURATION_CAP_SECONDS=600 # Required. Present as a constant so the answer is visible. There is no other value. EVASION_FEATURES=none
README.md
# Browse AI · indie build A visual scraper builder where the extraction is the easy half. The hard half is selectors that survive a site redesign, change detection that does not fire on a rotating advertisement, and being a polite enough client that the sites you depend on do not block you. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Browser | A headless browser for pages that need one, plain requests where they do not | running a browser per page is expensive and most pages do not need it | | Selection | Selectors with several fallback strategies per field | one brittle path breaks on the first redesign and takes the robot with it | | Politeness | Per-host rate limits, stated rules honoured, honest identification | the cost you impose is on someone else | | Change detection | Normalised comparison ignoring known-volatile regions | a rotating advertisement will otherwise fire an alert every run | | Runtime | TypeScript with PostgreSQL and a worker queue | scheduling, retries and history are the actual system | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the server, the workers and the headless browser driver. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally - Why: Runs, results and change history with per-host state. - Get it: Docker or your package manager. - Verify: psql connects and creates a table - [ ] **A headless browser** · free - Why: Some pages render their content in the browser and cannot be fetched as documents, though many can and running a browser for those is waste. - Get it: A maintained headless browser driver, plus a way to decide per site whether one is needed. - Verify: A page requiring rendering extracts correctly and a static page is fetched without the browser - [ ] **Whether you may scrape each target** · free to read; advice if this is commercial and contested - Why: The site's terms, whether the data is personal, and whether you are behind a login are three separate questions with different answers, and all three belong before the first run. - Get it: For each target, read the terms and the stated crawling rules, note whether the data includes personal information, and write down your conclusion. - Verify: A written note per target states the terms position, the personal data position and the login position - [ ] **Three real target sites** · free - Why: Selector fragility, rendering requirements and volatile regions only appear on real sites, and one of the three should be a site that has been redesigned so you can test fallbacks. - Get it: Pick sites you have a legitimate reason to monitor, ideally including one whose old markup you have saved. - Verify: Three targets exist, one static, one requiring rendering, one with rotating content ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Evasion of any kind. - Scraping behind a login. - Breadth of site coverage. - Legal advice about a site's terms. - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors If one of those is essential to you, that is the reason to keep paying for Browse AI, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Browse AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled web extraction robots that stay within what a site actually permits to replace Browse AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero A scraper is a program that imposes cost on someone else's server to take data they published for people to read. That can be entirely legitimate and it can be a nuisance or worse, and which one it is depends on decisions you make before writing any code. The ones this build makes for you: respect the site's stated crawling rules, identify yourself honestly in the user agent with a way to be contacted, rate limit conservatively per host, honour an explicit instruction to back off, and stop entirely after repeated refusals rather than working around them. The ones you have to make: whether the site's terms permit automated access, whether the data includes personal information and what obligations that carries, and whether you are behind a login, which changes the legal picture considerably. Write down your answers before the first run. The build deliberately does not include evasion. No rotating identities to look like different people, no defeating access controls, no pretending to be a browser you are not. If a site has made it clear it does not want automated access, the answer is to stop, not to be harder to detect. ### Stack (fixed, do not substitute) - Browser: A headless browser for pages that need one, plain requests where they do not · running a browser per page is expensive and most pages do not need it - Selection: Selectors with several fallback strategies per field · one brittle path breaks on the first redesign and takes the robot with it - Politeness: Per-host rate limits, stated rules honoured, honest identification · the cost you impose is on someone else - Change detection: Normalised comparison ignoring known-volatile regions · a rotating advertisement will otherwise fire an alert every run - Runtime: TypeScript with PostgreSQL and a worker queue · scheduling, retries and history are the actual system ### Data model (create this before Phase 1) - robots(id, name, start_url, host, schedule, enabled, created_at, disabled_reason) - fields(id, robot_id, name, selector_strategies_json, required, last_successful_strategy) - runs(id, robot_id, started_at, ended_at, status, pages_fetched, bytes, outcome, error) - results(id, run_id, row_ordinal, data_json, content_hash) - changes(id, robot_id, field, before, after, detected_at, significant) · significant excludes known-volatile regions - host_policy(host, crawl_delay_ms, rules_fetched_at, disallowed_paths_json, backoff_until) - A robot records which selector strategy last worked, so a redesign is visible as a fallback being used rather than as silent success. ### Phase 1 · Being a client sites do not mind Build: Fetch and cache each host's stated crawling rules and honour them per path. Identify the client honestly with a contact route. Rate limit per host with a conservative default. Honour an explicit backoff instruction over your own schedule. Trap: Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. Done when: Stated crawling rules are fetched, cached with an expiry and honoured per path. A path becoming disallowed disables the robot and reports why, tested by changing the rules. The user agent identifies the client and a contact route. An explicit backoff instruction is honoured exactly rather than overridden by the schedule. Do not build yet: No extraction, scheduling or change detection. ### Phase 2 · Selectors that survive a redesign Build: Capture several selector strategies per field when the robot is built: structural path, nearby label text, and attribute patterns. Try them in order and record which worked. Report a fallback being used rather than treating it as success. Trap: A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. Done when: Each field has several strategies recorded at build time. A changed structural path falls back to a label-based strategy, tested by modifying a fixture page. The run reports which strategy was used and warns when it was not the primary. A field matching nothing fails the run rather than recording an empty value. Do not build yet: No scheduling, change detection or delivery yet. ### Phase 3 · Running on a schedule without piling up Build: A worker queue with per-host concurrency limits, retries with backoff, and a cap on run duration. Isolate failures per robot. Detect a robot that has failed repeatedly and disable it rather than retrying forever. Trap: Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. Done when: Twenty robots on one host issue requests serially at the host's crawl delay, verified from the run log. A slow site does not block robots on other hosts. A repeatedly failing robot is disabled with a reason. A run exceeding its duration cap is terminated rather than running indefinitely. Do not build yet: No change detection or delivery yet. ### Phase 4 · Noticing real changes only Build: Compare runs on normalised values, excluding regions marked volatile. Let the user mark a region as volatile from an alert they did not want. Report changes with before and after values rather than only that something differed. Trap: Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. Done when: A page with a rotating advertisement and a timestamp does not fire an alert, tested across ten runs. Marking a region volatile from an alert prevents future alerts on it. Changes report before and after values. A genuine content change fires reliably, tested by editing a fixture. Do not build yet: No delivery or export yet. ### Phase 5 · Delivery, and what you collected Build: Deliver results by webhook, file or scheduled export with retries. Flag fields that look like personal data at build time and record the decision. Retain runs and results on a stated policy. Trap: Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each. Done when: Results deliver by webhook with retries and a signature. A field that looks like personal data is flagged at build time and requires a recorded decision. Retention deletes old runs and results on the stated policy. Export contains robots, fields and results and re-imports. Do not build yet: Nothing after this. ### Out of scope (and why) - Any form of evasion: rotating identities, defeating access controls, or bypassing measures a site has taken to stop automated access. - Scraping behind a login, which changes the legal picture substantially and is not something to do casually. - Breadth of site coverage, which is what the hosted products actually maintain and which needs constant attention. - Legal advice about whether a particular site's terms permit what you intend. ### README must contain - That stated crawling rules are honoured and re-fetched, and a newly disallowed path disables the robot - That the user agent identifies the client with a way to be contacted - That concurrency is limited per host regardless of how many robots want to run - That a fallback selector succeeding is reported as a warning rather than a clean run - That the build contains no evasion, deliberately
AGENTS.md
# Agent instructions · Browse AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A headless browser for pages that need one, plain requests where they do not, Selectors with several fallback strategies per field, Per-host rate limits, stated rules honoured, honest identification, Normalised comparison ignoring known-volatile regions, TypeScript with PostgreSQL and a worker queue. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself.
BUILD_PLAN.md
# Build plan · Browse AI A visual scraper builder where the extraction is the easy half. The hard half is selectors that survive a site redesign, change detection that does not fire on a rotating advertisement, and being a polite enough client that the sites you depend on do not block you. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Being a client sites do not mind Your fetching respects what each host has said and asked. ### Steps 1. Fetch and re-fetch stated rules With an expiry. A permanent cache ignores a site changing its mind. Files: `src/policy.ts` 2. Identify honestly with a contact So an operator with a problem can reach you instead of blocking you. Files: `src/fetch.ts` 3. Honour backoff over your schedule An explicit instruction outranks your plan. Always. Files: `src/ratelimit.ts`, `src/ratelimit.test.ts` ### Done when - [ ] Stated rules are honoured per path and re-fetched - [ ] A newly disallowed path disables the robot - [ ] The user agent identifies the client and a contact - [ ] Backoff instructions are honoured exactly ### Watch out - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. ## Phase 2 · Selectors that survive a redesign A robot keeps working after the site changes its markup. ### Steps 1. Capture alternatives at build time Structure, label proximity, attributes. One path is not enough. Files: `src/selectors.ts` 2. Record which strategy worked A fallback succeeding is a warning, not a clean run. Files: `src/extract.ts` 3. Fail rather than recording empty An empty value that looks like data is worse than a failure. Files: `src/extract.ts` ### Done when - [ ] Each field has several strategies - [ ] A changed path falls back successfully - [ ] The strategy used is reported and warned on - [ ] A field matching nothing fails the run ### Watch out - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. ## Phase 3 · Running on a schedule without piling up Robots run reliably and a slow site does not stop everything. ### Steps 1. Limit concurrency per host Not globally. Twenty robots on one site is an attack from their side. Files: `src/queue.ts` 2. Isolate failures per robot One dead site must not stop the others. Files: `src/worker.ts` 3. Disable persistent failures And say why. Retrying a dead robot forever helps nobody. Files: `src/health.ts` ### Done when - [ ] Twenty robots on one host run serially at its delay - [ ] A slow host does not block others - [ ] A repeatedly failing robot is disabled with a reason - [ ] Runs are terminated at a duration cap ### Watch out - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. ## Phase 4 · Noticing real changes only An alert means something changed that you care about. ### Steps 1. Normalise before comparing Whitespace, ordering, formatting. Raw comparison fires constantly. Files: `src/normalise.ts` 2. Let volatile regions be marked from the alert One click, at the moment of annoyance. Otherwise it never happens. Files: `src/volatile.ts` 3. Report before and after Something changed is not information. Files: `src/changes.ts` ### Done when - [ ] Rotating content does not fire alerts - [ ] Marking volatile from an alert works - [ ] Changes report before and after values - [ ] A genuine change fires reliably ### Watch out - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. ## Not in this build - Evasion of any kind. - Scraping behind a login. - Breadth of site coverage. - Legal advice about a site's terms. ## After v1, if you want it - A diff view across runs, which is what makes change history worth keeping - Robot templates for common page shapes, once selectors have proven durable - A shared host policy cache, so several robots on one site cooperate rather than compete
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Robots, runs and extracted results, which may include personal data. DATABASE_URL=postgres://localhost:5432/robots # Required. Identifies the client with a contact route. Anonymous clients get blocked. USER_AGENT=MyRobot/1.0 (+https://example.com/robot) # Required. Per host, unless the host states its own. Conservative by default. DEFAULT_CRAWL_DELAY_MS=5000 # Required. Regardless of how many robots target it. Twenty at once looks like an attack. MAX_CONCURRENCY_PER_HOST=1 # Required. A run that will not finish should be stopped rather than left going. RUN_DURATION_CAP_SECONDS=600 # Required. Present as a constant so the answer is visible. There is no other value. EVASION_FEATURES=none
PRODUCT.md
# Browse AI · product brief ## Problem The visible web data extraction loop is buildable, but a credible replacement needs more than the first screen. Browse AI earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A scraper that sites do not need to block, whose robots survive a redesign visibly rather than silently, and whose alerts mean something actually changed. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - hosted PostgreSQL and Redis - API credentials for the two chosen services - Node.js 22 ## Explicit non-goals for v1 - Evasion of any kind. - Scraping behind a login. - Breadth of site coverage. - Legal advice about a site's terms. - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors ## Success criteria - Stated rules are confirmed honoured and re-fetched - A newly disallowed path is confirmed to disable the robot - Twenty robots on one host are confirmed to run serially - A fallback strategy is confirmed reported as a warning - Rotating content is confirmed not to fire alerts
BRIEF.md
# Build brief · Browse AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled web extraction robots that stay within what a site actually permits to replace Browse AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero A scraper is a program that imposes cost on someone else's server to take data they published for people to read. That can be entirely legitimate and it can be a nuisance or worse, and which one it is depends on decisions you make before writing any code. The ones this build makes for you: respect the site's stated crawling rules, identify yourself honestly in the user agent with a way to be contacted, rate limit conservatively per host, honour an explicit instruction to back off, and stop entirely after repeated refusals rather than working around them. The ones you have to make: whether the site's terms permit automated access, whether the data includes personal information and what obligations that carries, and whether you are behind a login, which changes the legal picture considerably. Write down your answers before the first run. The build deliberately does not include evasion. No rotating identities to look like different people, no defeating access controls, no pretending to be a browser you are not. If a site has made it clear it does not want automated access, the answer is to stop, not to be harder to detect. ### Stack (fixed, do not substitute) - Browser: A headless browser for pages that need one, plain requests where they do not · running a browser per page is expensive and most pages do not need it - Selection: Selectors with several fallback strategies per field · one brittle path breaks on the first redesign and takes the robot with it - Politeness: Per-host rate limits, stated rules honoured, honest identification · the cost you impose is on someone else - Change detection: Normalised comparison ignoring known-volatile regions · a rotating advertisement will otherwise fire an alert every run - Runtime: TypeScript with PostgreSQL and a worker queue · scheduling, retries and history are the actual system ### Data model (create this before Phase 1) - robots(id, name, start_url, host, schedule, enabled, created_at, disabled_reason) - fields(id, robot_id, name, selector_strategies_json, required, last_successful_strategy) - runs(id, robot_id, started_at, ended_at, status, pages_fetched, bytes, outcome, error) - results(id, run_id, row_ordinal, data_json, content_hash) - changes(id, robot_id, field, before, after, detected_at, significant) · significant excludes known-volatile regions - host_policy(host, crawl_delay_ms, rules_fetched_at, disallowed_paths_json, backoff_until) - A robot records which selector strategy last worked, so a redesign is visible as a fallback being used rather than as silent success. ### Phase 1 · Being a client sites do not mind Build: Fetch and cache each host's stated crawling rules and honour them per path. Identify the client honestly with a contact route. Rate limit per host with a conservative default. Honour an explicit backoff instruction over your own schedule. Trap: Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. Done when: Stated crawling rules are fetched, cached with an expiry and honoured per path. A path becoming disallowed disables the robot and reports why, tested by changing the rules. The user agent identifies the client and a contact route. An explicit backoff instruction is honoured exactly rather than overridden by the schedule. Do not build yet: No extraction, scheduling or change detection. ### Phase 2 · Selectors that survive a redesign Build: Capture several selector strategies per field when the robot is built: structural path, nearby label text, and attribute patterns. Try them in order and record which worked. Report a fallback being used rather than treating it as success. Trap: A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. Done when: Each field has several strategies recorded at build time. A changed structural path falls back to a label-based strategy, tested by modifying a fixture page. The run reports which strategy was used and warns when it was not the primary. A field matching nothing fails the run rather than recording an empty value. Do not build yet: No scheduling, change detection or delivery yet. ### Phase 3 · Running on a schedule without piling up Build: A worker queue with per-host concurrency limits, retries with backoff, and a cap on run duration. Isolate failures per robot. Detect a robot that has failed repeatedly and disable it rather than retrying forever. Trap: Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. Done when: Twenty robots on one host issue requests serially at the host's crawl delay, verified from the run log. A slow site does not block robots on other hosts. A repeatedly failing robot is disabled with a reason. A run exceeding its duration cap is terminated rather than running indefinitely. Do not build yet: No change detection or delivery yet. ### Phase 4 · Noticing real changes only Build: Compare runs on normalised values, excluding regions marked volatile. Let the user mark a region as volatile from an alert they did not want. Report changes with before and after values rather than only that something differed. Trap: Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. Done when: A page with a rotating advertisement and a timestamp does not fire an alert, tested across ten runs. Marking a region volatile from an alert prevents future alerts on it. Changes report before and after values. A genuine content change fires reliably, tested by editing a fixture. Do not build yet: No delivery or export yet. ### Phase 5 · Delivery, and what you collected Build: Deliver results by webhook, file or scheduled export with retries. Flag fields that look like personal data at build time and record the decision. Retain runs and results on a stated policy. Trap: Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each. Done when: Results deliver by webhook with retries and a signature. A field that looks like personal data is flagged at build time and requires a recorded decision. Retention deletes old runs and results on the stated policy. Export contains robots, fields and results and re-imports. Do not build yet: Nothing after this. ### Out of scope (and why) - Any form of evasion: rotating identities, defeating access controls, or bypassing measures a site has taken to stop automated access. - Scraping behind a login, which changes the legal picture substantially and is not something to do casually. - Breadth of site coverage, which is what the hosted products actually maintain and which needs constant attention. - Legal advice about whether a particular site's terms permit what you intend. ### README must contain - That stated crawling rules are honoured and re-fetched, and a newly disallowed path disables the robot - That the user agent identifies the client with a way to be contacted - That concurrency is limited per host regardless of how many robots want to run - That a fallback selector succeeding is reported as a warning rather than a clean run - That the build contains no evasion, deliberately
ARCHITECTURE.md
# Architecture · Browse AI ## Stack | Part | Choice | Why | | --- | --- | --- | | Browser | A headless browser for pages that need one, plain requests where they do not | running a browser per page is expensive and most pages do not need it | | Selection | Selectors with several fallback strategies per field | one brittle path breaks on the first redesign and takes the robot with it | | Politeness | Per-host rate limits, stated rules honoured, honest identification | the cost you impose is on someone else | | Change detection | Normalised comparison ignoring known-volatile regions | a rotating advertisement will otherwise fire an alert every run | | Runtime | TypeScript with PostgreSQL and a worker queue | scheduling, retries and history are the actual system | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Policy | Stated rules, delays and backoff. | Any implementation, provided rules are re-fetched and an explicit backoff outranks the schedule. | | Selection | Multiple strategies and fallback reporting. | Any strategy set, provided a fallback is reported rather than treated as success. | | Queue | Per-host concurrency and isolation. | Any queue, provided the limit is per host rather than global. | | Change detection | Normalised comparison and volatile regions. | Any comparison, provided volatile regions can be marked from an alert. | | Delivery | Retried, signed delivery and personal data flags. | Any transport, provided personal fields are flagged when the robot is built. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Robots, runs and extracted results, which may include personal data. - `USER_AGENT` · required · Identifies the client with a contact route. Anonymous clients get blocked. - `DEFAULT_CRAWL_DELAY_MS` · required · Per host, unless the host states its own. Conservative by default. - `MAX_CONCURRENCY_PER_HOST` · required · Regardless of how many robots target it. Twenty at once looks like an attack. - `RUN_DURATION_CAP_SECONDS` · required · A run that will not finish should be stopped rather than left going. - `EVASION_FEATURES` · required · Present as a constant so the answer is visible. There is no other value. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Browse AI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A headless browser for pages that need one, plain requests where they do not, Selectors with several fallback strategies per field, Per-host rate limits, stated rules honoured, honest identification, Normalised comparison ignoring known-volatile regions, TypeScript with PostgreSQL and a worker queue. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. - Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each.
MILESTONES.md
# Delivery milestones · Browse AI Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Being a client sites do not mind Your fetching respects what each host has said and asked. ### Steps 1. Fetch and re-fetch stated rules With an expiry. A permanent cache ignores a site changing its mind. Files: `src/policy.ts` 2. Identify honestly with a contact So an operator with a problem can reach you instead of blocking you. Files: `src/fetch.ts` 3. Honour backoff over your schedule An explicit instruction outranks your plan. Always. Files: `src/ratelimit.ts`, `src/ratelimit.test.ts` ### Done when - [ ] Stated rules are honoured per path and re-fetched - [ ] A newly disallowed path disables the robot - [ ] The user agent identifies the client and a contact - [ ] Backoff instructions are honoured exactly ### Watch out - Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore. ## M2 · Selectors that survive a redesign A robot keeps working after the site changes its markup. ### Steps 1. Capture alternatives at build time Structure, label proximity, attributes. One path is not enough. Files: `src/selectors.ts` 2. Record which strategy worked A fallback succeeding is a warning, not a clean run. Files: `src/extract.ts` 3. Fail rather than recording empty An empty value that looks like data is worse than a failure. Files: `src/extract.ts` ### Done when - [ ] Each field has several strategies - [ ] A changed path falls back successfully - [ ] The strategy used is reported and warned on - [ ] A field matching nothing fails the run ### Watch out - A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run. ## M3 · Running on a schedule without piling up Robots run reliably and a slow site does not stop everything. ### Steps 1. Limit concurrency per host Not globally. Twenty robots on one site is an attack from their side. Files: `src/queue.ts` 2. Isolate failures per robot One dead site must not stop the others. Files: `src/worker.ts` 3. Disable persistent failures And say why. Retrying a dead robot forever helps nobody. Files: `src/health.ts` ### Done when - [ ] Twenty robots on one host run serially at its delay - [ ] A slow host does not block others - [ ] A repeatedly failing robot is disabled with a reason - [ ] Runs are terminated at a duration cap ### Watch out - Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run. ## M4 · Noticing real changes only An alert means something changed that you care about. ### Steps 1. Normalise before comparing Whitespace, ordering, formatting. Raw comparison fires constantly. Files: `src/normalise.ts` 2. Let volatile regions be marked from the alert One click, at the moment of annoyance. Otherwise it never happens. Files: `src/volatile.ts` 3. Report before and after Something changed is not information. Files: `src/changes.ts` ### Done when - [ ] Rotating content does not fire alerts - [ ] Marking volatile from an alert works - [ ] Changes report before and after values - [ ] A genuine change fires reliably ### Watch out - Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself. ## M5 · Delivery, and what you collected (production only) Results reach where you need them and you know what you are holding. ### Steps 1. Deliver with retries and signing A receiving endpoint being down should not lose the run. Files: `src/deliver.ts` 2. Flag likely personal fields at build When someone is looking. Not at scale, six months later. Files: `src/personal.ts` 3. Retain on a stated policy Scraped data accumulates. Decide how long before it does. Files: `src/retention.ts` ### Done when - [ ] Webhook delivery retries and is signed - [ ] Personal-looking fields are flagged at build time - [ ] Retention deletes runs on the stated policy - [ ] Export round trips robots, fields and results ### Watch out - Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each.
OPERATIONS.md
# Operations · Browse AI ## Backup Robot definitions are the work; results are usually reproducible by running again, subject to the site still having the data. Back up definitions and change history, and apply retention to results rather than accumulating them. ## Restore Restore and confirm host policy caches are expired rather than restored stale, since a restored cache may permit a path the site has since disallowed. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Fallback selector usage and per-host error rates are the health metrics. A robot silently running on its third fallback strategy is one redesign from producing wrong data. ## Incident checklist Being blocked by a site is the incident, and the cause is nearly always concurrency or ignoring a stated rule. The run log records what your client actually did, which is what lets you answer an operator rather than guess. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Stated rules are confirmed honoured and re-fetched - [ ] A newly disallowed path is confirmed to disable the robot - [ ] Twenty robots on one host are confirmed to run serially - [ ] A fallback strategy is confirmed reported as a warning - [ ] Rotating content is confirmed not to fire alerts ## Launch constraint Do not market omitted Browse AI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Robots, runs and extracted results, which may include personal data. DATABASE_URL=postgres://localhost:5432/robots # Required. Identifies the client with a contact route. Anonymous clients get blocked. USER_AGENT=MyRobot/1.0 (+https://example.com/robot) # Required. Per host, unless the host states its own. Conservative by default. DEFAULT_CRAWL_DELAY_MS=5000 # Required. Regardless of how many robots target it. Twenty at once looks like an attack. MAX_CONCURRENCY_PER_HOST=1 # Required. A run that will not finish should be stopped rather than left going. RUN_DURATION_CAP_SECONDS=600 # Required. Present as a constant so the answer is visible. There is no other value. EVASION_FEATURES=none
$ choose a build depth, inspect the files, then open the complete pack in your agent
Browse AI: One automation is easy. Subscribers pay for maintained integrations, credentials, retries, observability, and someone else owning breakage.
xOAuth app verification
xdurable execution at scale
xschema drift handling and enterprise controls
xhundreds of maintained connectors
share on X ↗"I just replaced Browse AI ($19/mo) with one prompt"
Don't feel like building it? These folks already made it free.
EasySpiderA desktop scraper whose academic-looking workflow editor still gets paginated data out.open source↗
MaxunPoint, click, schedule a robot, and keep the scraper on your own box.open source↗no votes, no pay-to-list · just what's real
Browse AI pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| free | $0/workspace | $0/workspace | Unlimited robots; 2 websites; 3 users; 90-day data retention; monitoring no more frequent than hourly |
| personal | $48/workspace | $19/workspace | 2,000 credits on monthly billing or 12,000 credits/year paid upfront; 5 websites; 3 users; unlimited robots; 90-day retention; 5-minute monitoring |
| professional | $87/workspace | $69/workspace | 5,000 credits/month or 60,000 credits/year; 10 websites; 10 users; unlimited robots; 90-day retention; 5-minute monitoring |
| premium | — | $500/workspace | Custom credits, websites and users; managed onboarding and higher-scale automation |
free tierUnlimited robots, 2 websites, 3 users, 90-day retention and hourly monitoring; numeric monthly credits were not reliably exposed on the current page
billingfree + monthly or annual Personal/Professional; Premium is annual-only from $6,000/year; credits reset by billing period and do not carry over
hidden costsExtra websites cost $5/month on monthly Personal or $4/month annual, and $3/month monthly or $2.40/month annual on Professional; some premium-site runs consume 2-10+ credits; 1 credit covers only 10 extracted rows or 1 screenshot, and unused credits expire
verified 2026-08-14 · source ↗
Vibecode Browse AI
Kinda. The core of Browse AI is buildable in a weekend with the prompt on this page, but there are real gaps: OAuth app verification, durable execution at scale. Read the honest list above before committing.
How much does Browse AI cost?
Browse AI costs about $19/month (Starter, checked 2026-07-31), which is $228 per year.
What do I lose by replacing Browse AI?
Honestly: OAuth app verification; durable execution at scale; schema drift handling and enterprise controls; hundreds of maintained connectors. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Browse AI?
Yes: EasySpider (A desktop scraper whose academic-looking workflow editor still gets paginated data out.) Maxun (Point, click, schedule a robot, and keep the scraper on your own box.) The prompt is for when you want it exactly your way.