Build Browse AI

KINDAreplaces $19/mosaves $228/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A visual scraper builder where the extraction is the easy half. The hard half is selectors that survive a site redesign, change detection that does not fire on a rotating advertisement, and being a polite enough client that the sites you depend on do not block you.

estimated effort multi-daythe files for this build are in the project pack

BrowserA headless browser for pages that need one, plain requests where they do notSelectionSelectors with several fallback strategies per fieldPolitenessPer-host rate limits, stated rules honoured, honest identificationChange detectionNormalised comparison ignoring known-volatile regionsRuntimeTypeScript with PostgreSQL and a worker queue

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the server, the workers and the headless browser driver.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
installfree locally
Why Runs, results and change history with per-host state.
Get it Docker or your package manager. open ↗
Verify psql connects and creates a table
installfree
Why Some pages render their content in the browser and cannot be fetched as documents, though many can and running a browser for those is waste.
Get it A maintained headless browser driver, plus a way to decide per site whether one is needed.
Verify A page requiring rendering extracts correctly and a static page is fetched without the browser
decidefree to read; advice if this is commercial and contested
Why The site's terms, whether the data is personal, and whether you are behind a login are three separate questions with different answers, and all three belong before the first run.
Get it For each target, read the terms and the stated crawling rules, note whether the data includes personal information, and write down your conclusion.
Verify A written note per target states the terms position, the personal data position and the login position
have readyfree
Why Selector fragility, rendering requirements and volatile regions only appear on real sites, and one of the three should be a site that has been redesigned so you can test fallbacks.
Get it Pick sites you have a legitimate reason to monitor, ideally including one whose old markup you have saved.
Verify Three targets exist, one static, one requiring rendering, one with rotating content

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- robots(id, name, start_url, host, schedule, enabled, created_at, disabled_reason)
- fields(id, robot_id, name, selector_strategies_json, required, last_successful_strategy)
- runs(id, robot_id, started_at, ended_at, status, pages_fetched, bytes, outcome, error)
- results(id, run_id, row_ordinal, data_json, content_hash)
- changes(id, robot_id, field, before, after, detected_at, significant) · significant excludes known-volatile regions
- host_policy(host, crawl_delay_ms, rules_fetched_at, disallowed_paths_json, backoff_until)
- A robot records which selector strategy last worked, so a redesign is visible as a fallback being used rather than as silent success.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_URL`secret	required	`postgres://localhost:5432/robots`	Robots, runs and extracted results, which may include personal data.
`USER_AGENT`	required	`MyRobot/1.0 (+https://example.com/robot)`	Identifies the client with a contact route. Anonymous clients get blocked.
`DEFAULT_CRAWL_DELAY_MS`	required	`5000`	Per host, unless the host states its own. Conservative by default.
`MAX_CONCURRENCY_PER_HOST`	required	`1`	Regardless of how many robots target it. Twenty at once looks like an attack.
`RUN_DURATION_CAP_SECONDS`	required	`600`	A run that will not finish should be stopped rather than left going.
`EVASION_FEATURES`	required	`none`	Present as a constant so the answer is visible. There is no other value.

The build, in order

1
Being a client sites do not mind
Your fetching respects what each host has said and asked.
1. With an expiry. A permanent cache ignores a site changing its mind.
  Files src/policy.ts
2. So an operator with a problem can reach you instead of blocking you.
  Files src/fetch.ts
3. An explicit instruction outranks your plan. Always.
  Files src/ratelimit.tssrc/ratelimit.test.ts
done when · tick each as it passes
watch out
- Fetching the rules once and caching them forever means a site that later disallows your path keeps being crawled, and the operator's only remaining option is to block you. Re-fetch periodically, and treat a newly disallowed path as a reason to disable the robot and tell the user rather than a rule to ignore.
next phase
2
Selectors that survive a redesign
A robot keeps working after the site changes its markup.
1. Structure, label proximity, attributes. One path is not enough.
  Files src/selectors.ts
2. A fallback succeeding is a warning, not a clean run.
  Files src/extract.ts
3. An empty value that looks like data is worse than a failure.
  Files src/extract.ts
done when · tick each as it passes
watch out
- A single generated selector path breaks on the first redesign and the robot either fails loudly or, worse, matches the wrong element and reports plausible wrong data indefinitely. Capture alternatives at build time, and treat a fallback strategy succeeding as a warning that the primary broke rather than as a clean run.
next phase
3
Running on a schedule without piling up
Robots run reliably and a slow site does not stop everything.
1. Not globally. Twenty robots on one site is an attack from their side.
  Files src/queue.ts
2. One dead site must not stop the others.
  Files src/worker.ts
3. And say why. Retrying a dead robot forever helps nobody.
  Files src/health.ts
done when · tick each as it passes
watch out
- Global concurrency without a per-host limit means twenty robots pointed at one site hit it twenty times at once, which is indistinguishable from an attack from the operator's side. Limit concurrency per host regardless of how many robots want to run.
next phase
4
Noticing real changes only
An alert means something changed that you care about.
1. Whitespace, ordering, formatting. Raw comparison fires constantly.
  Files src/normalise.ts
2. One click, at the moment of annoyance. Otherwise it never happens.
  Files src/volatile.ts
3. Something changed is not information.
  Files src/changes.ts
done when · tick each as it passes
watch out
- Comparing raw page content fires on every rotating advertisement, timestamp and session identifier, and the user turns off alerts within two days, at which point the monitoring is decorative. Normalise, exclude volatile regions, and make marking something volatile a one-click action from the alert itself.
5
Delivery, and what you collectedproduct builder
Results reach where you need them and you know what you are holding.
1. A receiving endpoint being down should not lose the run.
  Files src/deliver.ts
2. When someone is looking. Not at scale, six months later.
  Files src/personal.ts
3. Scraped data accumulates. Decide how long before it does.
  Files src/retention.ts
done when · tick each as it passes
watch out
- Scraping personal data without noticing you have done so creates obligations you did not know you had, and at scale the first you hear about it is a complaint. Flag likely personal fields when the robot is built, when someone is actually looking, and record what was decided about each.

what this build does not replace

Evasion of any kind.
Scraping behind a login.
Breadth of site coverage.
Legal advice about a site's terms.
OAuth app verification
durable execution at scale
schema drift handling and enterprise controls
hundreds of maintained connectors

after v1, if you want it

A diff view across runs, which is what makes change history worth keeping
Robot templates for common page shapes, once selectors have proven durable
A shared host policy cache, so several robots on one site cooperate rather than compete

Need the files? The project pack on the verdict page hands your agent the whole brief · more automation.