Vibecode Checkly
track this build5 phases, 15 steps, beginner friendly0%KINDA · weekend project
The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Checkly, run scheduled API and browser checks from one independent worker. The hard boundary is global browser fleet, playwright operations, alerting, ci integration, and support, plus independent infrastructure and reliable alerting.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Checkly. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Checkly · indie build A checker running outside your infrastructure that performs API and browser checks on a schedule, distinguishes a real failure from a blip before alerting, sends one alert per incident rather than one per check, and publishes a status page whose uptime figure is computed honestly including its own gaps. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with PostgreSQL | checks, results and incidents are relational and the incident logic needs transactions | | Browser checks | Playwright, headless | a check that logs in and clicks through a flow catches what an endpoint check cannot | | Location | A server with a different provider and region from what it monitors | this is the whole design constraint, not a deployment preference | | Alerting | A path independent of your own infrastructure | alerting through a service you host means no alert during the outage that matters | | Status page | Static, generated and served from somewhere separate | a status page hosted on the failing server is the oldest mistake in this category | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the checker and Playwright. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A server with a different provider from what you monitor** · 5 to 10 US dollars a month - Why: This is the entire design constraint. A monitor in the same failure domain cannot report the outage you built it for. - Get it: A small instance with a provider you do not otherwise use. It needs very little capacity; independence is the requirement rather than power. - Verify: A written list of shared dependencies exists and is short - [ ] **PostgreSQL 16 or newer** · free self-hosted - Why: Results, incidents and alerts, with the incident logic needing transactions. - Get it: On the checker server, or a managed instance from a third provider. - Verify: psql connects and reports 16 or higher - [ ] **An alerting channel independent of your infrastructure** · free tiers cover this volume - Why: An alert routed through something you host is an alert you will not receive during the outage that matters. - Get it: A transactional email provider or an external chat service. Not a webhook into your own application. - Verify: An alert arrives with your monitored infrastructure fully down - [ ] **Somewhere independent to serve the status page** · free on most static hosts - Why: A status page hosted on the failing infrastructure is the oldest mistake in this category and it is made repeatedly. - Get it: Static hosting with a different provider, or the checker server itself if that is genuinely separate. - Verify: The page serves while the monitored system is down ## Quick start ```sh npx playwright install chromium ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A global probe network. - Phone and text message delivery. - Managed browser check operations at scale. - Anyone to call when the monitoring breaks. - global browser fleet, Playwright operations, alerting, CI integration, and support - global probe network - phone and SMS delivery - massive retention - advanced incident response and support If one of those is essential to you, that is the reason to keep paying for Checkly, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Checkly The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled checks that run from outside your infrastructure, alert once, and feed an honest status page to replace Checkly. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The monitor must not share a failure domain with what it monitors. A check running on the same server, in the same region or behind the same provider will go down with the thing it is watching, and the outage you most need to know about is the one it cannot report. That single constraint decides the architecture: the checker runs somewhere else, the alerting path does not depend on your application, and the status page is served from somewhere that survives your infrastructure being gone. Everything else here is straightforward. ### Stack (fixed, do not substitute) - Runtime: Node 22 with PostgreSQL · checks, results and incidents are relational and the incident logic needs transactions - Browser checks: Playwright, headless · a check that logs in and clicks through a flow catches what an endpoint check cannot - Location: A server with a different provider and region from what it monitors · this is the whole design constraint, not a deployment preference - Alerting: A path independent of your own infrastructure · alerting through a service you host means no alert during the outage that matters - Status page: Static, generated and served from somewhere separate · a status page hosted on the failing server is the oldest mistake in this category ### Data model (create this before Phase 1) - checks(id, name, kind, target, interval_seconds, timeout_ms, expected_json, enabled, degraded_after_ms) · kind is http or browser - results(check_id, at, ok, duration_ms, status_code, error, screenshot_key) - incidents(id, check_id, started_at, confirmed_at, resolved_at, state, cause_summary) · confirmed_at is when it passed the confirmation threshold - alerts(incident_id, at, channel, delivered, error) - maintenance(id, check_id, starts_at, ends_at, reason) · declared downtime excluded from uptime, and labelled on the status page - checker_gaps(started_at, ended_at, reason) · when the checker itself was not running, so uptime can be honest about it - An outage is a confirmed incident, not a single failed result. The confirmation threshold is what separates a monitor from a noise generator. ### Phase 1 · A checker somewhere else Build: Deploy the checker to a different provider and region from your application. Run a simple HTTP check on a schedule with a timeout. Record every result. Then confirm the separation by listing what the two share. Trap: A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. Done when: Checks run on a schedule from a different provider and region. Every result is recorded with duration and status. A written note lists what the checker and target share and why each is acceptable. Stopping the target produces failed results within one interval. Do not build yet: No alerting, incidents or status page. ### Phase 2 · Checks that assert something real Build: Assertions beyond a status code: response body content, a JSON field, and a response time threshold that marks degraded rather than failed. Then add a browser check that completes a real user flow. Trap: A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. Done when: A check asserts a body value and fails when the application returns an error page with a two hundred status. A response slower than the degraded threshold is marked degraded rather than failed. A browser check logs in and reaches an authenticated page, and captures a screenshot on failure. A browser check failing on a selector reports which selector. Do not build yet: No incident logic or alerting yet. ### Phase 3 · Incidents, not results Build: Require several consecutive failures before confirming an incident, and several consecutive successes before resolving it. Group failures into an incident with a start and an end. Record the first error as the cause summary. Trap: Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. Done when: A single failed check followed by a success produces no incident. Three consecutive failures open one incident. A flapping service produces one incident rather than a sequence, verified by alternating failure and success at the boundary. The incident records the first error as its cause summary. Do not build yet: No alerting yet. ### Phase 4 · Alerting that survives the outage Build: Send one alert when an incident is confirmed and one when it resolves. Use a channel that does not route through your own infrastructure. Retry delivery and record failures. Support a maintenance window that suppresses alerts without hiding the downtime. Trap: Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. Done when: An incident produces one alert and its resolution produces one more. Taking the monitored infrastructure down entirely still delivers the alert, tested deliberately. A failed delivery is retried and then recorded. A maintenance window suppresses the alert while the downtime still appears on the status page. Do not build yet: No status page yet. ### Phase 5 · A status page that tells the truth Build: Generate a static status page from incidents, served from somewhere independent. Compute uptime from confirmed incidents, exclude declared maintenance and label it, and account for periods when the checker itself was not running rather than counting them as up. Trap: Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. Done when: The status page is served from infrastructure independent of the monitored system, verified by taking the latter down. Uptime excludes declared maintenance and labels it. A period when the checker was not running appears as unknown rather than as up. The uptime figure matches a hand calculation over a known window. Do not build yet: Operations and multiple locations come last. ### Phase 6 · More locations, and knowing what you cannot see Build: Run checkers from a second location and require agreement before confirming a global incident. Report a single-location failure as regional. Then write down honestly what this setup cannot detect. Trap: One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is. Done when: Two locations run independently and a failure at one is reported as regional rather than global. Agreement across locations confirms a global incident. A written note records what this cannot detect, including problems affecting only regions with no checker. Each location's results are visible separately. Do not build yet: Nothing after this. ### Out of scope (and why) - A global probe network. Two locations is meaningfully better than one and a long way from checking every region your users are in. - Phone and text message delivery, which needs a provider relationship and is what you want at three in the morning. - Managed browser check operations at scale, including keeping Playwright and its browsers current across a fleet. - Someone to call when the monitoring itself breaks, which is a real gap since a broken monitor is silent by nature. ### README must contain - That the checker runs outside the monitored failure domain, and the written list of what they still share - That checks assert content rather than status codes, and why - That an incident requires consecutive failures, with the resulting detection time stated - That the status page and alerting are independent of the monitored infrastructure - That uptime accounts for checker gaps as unknown rather than counting them as up ===== AGENTS.md ===== # Agent instructions · Checkly indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with PostgreSQL, Playwright, headless, A server with a different provider and region from what it monitors, A path independent of your own infrastructure, Static, generated and served from somewhere separate. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. ===== BUILD_PLAN.md ===== # Build plan · Checkly A checker running outside your infrastructure that performs API and browser checks on a schedule, distinguishes a real failure from a blip before alerting, sends one alert per incident rather than one per check, and publishes a status page whose uptime figure is computed honestly including its own gaps. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A checker somewhere else Checks run from outside the failure domain of what they watch, and you have verified that. ### Steps 1. Deploy outside the failure domain Different provider and region. Not a different instance in the same account. Files: `docs/separation.md` 2. Run scheduled HTTP checks With a timeout, recording duration, status and error for every run. Files: `src/http-check.js` 3. Write down the shared dependencies Provider, region, DNS, certificate authority. Each one is a way the monitor can fail with the target. Files: `docs/separation.md` ### Done when - [ ] Checks run from a different provider and region - [ ] Every result records duration, status and error - [ ] The shared dependency list is written and reviewed - [ ] Stopping the target produces failures within one interval ### Watch out - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. ## Phase 2 · Checks that assert something real A check fails when the application is broken, not only when it is down. ### Steps 1. Assert content, not just status A value only a working application produces. Status codes lie routinely. Files: `src/assertions.js` 2. Separate degraded from failed A slow response is information; treating it as an outage produces noise. Files: `src/assertions.js` 3. Add a browser check with a screenshot A real flow, with a screenshot and the failing selector captured on failure. Files: `src/browser-check.js` ```sh npx playwright install chromium ``` ### Done when - [ ] A content assertion catches an error page returning two hundred - [ ] A slow response is marked degraded rather than failed - [ ] A browser check completes an authenticated flow - [ ] A browser failure captures a screenshot and names the selector ### Watch out - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. ## Phase 3 · Incidents, not results One outage produces one incident, and a blip produces nothing. ### Steps 1. Confirm before opening Consecutive failures past a threshold. The threshold and interval together define your detection time; state it. Files: `src/incidents.js` 2. Require stability to resolve Consecutive successes, so flapping does not close and reopen. Files: `src/incidents.js` 3. Test the flapping case Alternate failure and success around the threshold. This is where naive logic produces a storm. Files: `test/incidents.test.js` ### Done when - [ ] A single failure followed by success opens no incident - [ ] Consecutive failures open exactly one incident - [ ] Flapping produces one incident rather than a sequence - [ ] The incident records the first error ### Watch out - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. ## Phase 4 · Alerting that survives the outage You are told once, through a path that does not depend on what is broken. ### Steps 1. Alert once per state change Confirmed and resolved. Not per failed check. Files: `src/alerts.js` 2. Use an independent delivery path External provider, not your own infrastructure. Test by taking everything down. Files: `src/channels.js` 3. Add maintenance windows Suppress the alert, keep the downtime visible. Hiding it makes uptime figures dishonest. Files: `src/maintenance.js` ### Done when - [ ] One alert on confirmation and one on resolution - [ ] The alert arrives with the monitored infrastructure fully down - [ ] A failed delivery is retried and recorded - [ ] Maintenance suppresses alerts while downtime stays visible ### Watch out - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. ## Phase 5 · A status page that tells the truth The uptime figure is honest, including about the times the monitor was not looking. ### Steps 1. Generate a static page From incidents, published somewhere separate. Static because it must serve when everything else is gone. Files: `src/status.js` 2. Record checker gaps Whenever the checker was not running. These become unknown periods rather than uptime. Files: `src/gaps.js` 3. Verify against a hand calculation One month, computed by hand from the incident list. Uptime arithmetic is easy to get subtly wrong. ### Done when - [ ] The status page serves with the monitored system fully down - [ ] Maintenance is excluded and labelled - [ ] Checker gaps appear as unknown rather than up - [ ] Uptime matches a hand calculation ### Watch out - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. ## Not in this build - A global probe network. - Phone and text message delivery. - Managed browser check operations at scale. - Anyone to call when the monitoring breaks. ## After v1, if you want it - A third location, once two have disagreed usefully at least once - Response time percentiles per check, which reveal degradation before failure - Public incident write-ups attached to the status page history ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. On the checker, not on the monitored infrastructure. DATABASE_URL=postgres://user:pass@localhost:5432/checks # Required. Consecutive failures before an incident is confirmed. With the interval, this defines your detection time. CONFIRM_FAILURES=3 # Required. Consecutive successes before resolving. Higher values suppress flapping. RESOLVE_SUCCESSES=3 # Required. How often checks run. Shorter detects faster and costs more requests. DEFAULT_INTERVAL_SECONDS=60 # Required · secret. Independent delivery path. Never a webhook into the monitored system. ALERT_CHANNEL_URL=https://api.provider.example/send # Required. Where the generated status page is published. Independent hosting. STATUS_PAGE_DEST=s3://status-bucket # Required. How long without a checker run before the period is recorded as a gap rather than uptime. CHECKER_HEARTBEAT_SECONDS=120
You are building a lean indie version of Checkly. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Checkly · indie build A checker running outside your infrastructure that performs API and browser checks on a schedule, distinguishes a real failure from a blip before alerting, sends one alert per incident rather than one per check, and publishes a status page whose uptime figure is computed honestly including its own gaps. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with PostgreSQL | checks, results and incidents are relational and the incident logic needs transactions | | Browser checks | Playwright, headless | a check that logs in and clicks through a flow catches what an endpoint check cannot | | Location | A server with a different provider and region from what it monitors | this is the whole design constraint, not a deployment preference | | Alerting | A path independent of your own infrastructure | alerting through a service you host means no alert during the outage that matters | | Status page | Static, generated and served from somewhere separate | a status page hosted on the failing server is the oldest mistake in this category | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the checker and Playwright. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A server with a different provider from what you monitor** · 5 to 10 US dollars a month - Why: This is the entire design constraint. A monitor in the same failure domain cannot report the outage you built it for. - Get it: A small instance with a provider you do not otherwise use. It needs very little capacity; independence is the requirement rather than power. - Verify: A written list of shared dependencies exists and is short - [ ] **PostgreSQL 16 or newer** · free self-hosted - Why: Results, incidents and alerts, with the incident logic needing transactions. - Get it: On the checker server, or a managed instance from a third provider. - Verify: psql connects and reports 16 or higher - [ ] **An alerting channel independent of your infrastructure** · free tiers cover this volume - Why: An alert routed through something you host is an alert you will not receive during the outage that matters. - Get it: A transactional email provider or an external chat service. Not a webhook into your own application. - Verify: An alert arrives with your monitored infrastructure fully down - [ ] **Somewhere independent to serve the status page** · free on most static hosts - Why: A status page hosted on the failing infrastructure is the oldest mistake in this category and it is made repeatedly. - Get it: Static hosting with a different provider, or the checker server itself if that is genuinely separate. - Verify: The page serves while the monitored system is down ## Quick start ```sh npx playwright install chromium ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A global probe network. - Phone and text message delivery. - Managed browser check operations at scale. - Anyone to call when the monitoring breaks. - global browser fleet, Playwright operations, alerting, CI integration, and support - global probe network - phone and SMS delivery - massive retention - advanced incident response and support If one of those is essential to you, that is the reason to keep paying for Checkly, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Checkly The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled checks that run from outside your infrastructure, alert once, and feed an honest status page to replace Checkly. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The monitor must not share a failure domain with what it monitors. A check running on the same server, in the same region or behind the same provider will go down with the thing it is watching, and the outage you most need to know about is the one it cannot report. That single constraint decides the architecture: the checker runs somewhere else, the alerting path does not depend on your application, and the status page is served from somewhere that survives your infrastructure being gone. Everything else here is straightforward. ### Stack (fixed, do not substitute) - Runtime: Node 22 with PostgreSQL · checks, results and incidents are relational and the incident logic needs transactions - Browser checks: Playwright, headless · a check that logs in and clicks through a flow catches what an endpoint check cannot - Location: A server with a different provider and region from what it monitors · this is the whole design constraint, not a deployment preference - Alerting: A path independent of your own infrastructure · alerting through a service you host means no alert during the outage that matters - Status page: Static, generated and served from somewhere separate · a status page hosted on the failing server is the oldest mistake in this category ### Data model (create this before Phase 1) - checks(id, name, kind, target, interval_seconds, timeout_ms, expected_json, enabled, degraded_after_ms) · kind is http or browser - results(check_id, at, ok, duration_ms, status_code, error, screenshot_key) - incidents(id, check_id, started_at, confirmed_at, resolved_at, state, cause_summary) · confirmed_at is when it passed the confirmation threshold - alerts(incident_id, at, channel, delivered, error) - maintenance(id, check_id, starts_at, ends_at, reason) · declared downtime excluded from uptime, and labelled on the status page - checker_gaps(started_at, ended_at, reason) · when the checker itself was not running, so uptime can be honest about it - An outage is a confirmed incident, not a single failed result. The confirmation threshold is what separates a monitor from a noise generator. ### Phase 1 · A checker somewhere else Build: Deploy the checker to a different provider and region from your application. Run a simple HTTP check on a schedule with a timeout. Record every result. Then confirm the separation by listing what the two share. Trap: A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. Done when: Checks run on a schedule from a different provider and region. Every result is recorded with duration and status. A written note lists what the checker and target share and why each is acceptable. Stopping the target produces failed results within one interval. Do not build yet: No alerting, incidents or status page. ### Phase 2 · Checks that assert something real Build: Assertions beyond a status code: response body content, a JSON field, and a response time threshold that marks degraded rather than failed. Then add a browser check that completes a real user flow. Trap: A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. Done when: A check asserts a body value and fails when the application returns an error page with a two hundred status. A response slower than the degraded threshold is marked degraded rather than failed. A browser check logs in and reaches an authenticated page, and captures a screenshot on failure. A browser check failing on a selector reports which selector. Do not build yet: No incident logic or alerting yet. ### Phase 3 · Incidents, not results Build: Require several consecutive failures before confirming an incident, and several consecutive successes before resolving it. Group failures into an incident with a start and an end. Record the first error as the cause summary. Trap: Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. Done when: A single failed check followed by a success produces no incident. Three consecutive failures open one incident. A flapping service produces one incident rather than a sequence, verified by alternating failure and success at the boundary. The incident records the first error as its cause summary. Do not build yet: No alerting yet. ### Phase 4 · Alerting that survives the outage Build: Send one alert when an incident is confirmed and one when it resolves. Use a channel that does not route through your own infrastructure. Retry delivery and record failures. Support a maintenance window that suppresses alerts without hiding the downtime. Trap: Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. Done when: An incident produces one alert and its resolution produces one more. Taking the monitored infrastructure down entirely still delivers the alert, tested deliberately. A failed delivery is retried and then recorded. A maintenance window suppresses the alert while the downtime still appears on the status page. Do not build yet: No status page yet. ### Phase 5 · A status page that tells the truth Build: Generate a static status page from incidents, served from somewhere independent. Compute uptime from confirmed incidents, exclude declared maintenance and label it, and account for periods when the checker itself was not running rather than counting them as up. Trap: Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. Done when: The status page is served from infrastructure independent of the monitored system, verified by taking the latter down. Uptime excludes declared maintenance and labels it. A period when the checker was not running appears as unknown rather than as up. The uptime figure matches a hand calculation over a known window. Do not build yet: Operations and multiple locations come last. ### Phase 6 · More locations, and knowing what you cannot see Build: Run checkers from a second location and require agreement before confirming a global incident. Report a single-location failure as regional. Then write down honestly what this setup cannot detect. Trap: One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is. Done when: Two locations run independently and a failure at one is reported as regional rather than global. Agreement across locations confirms a global incident. A written note records what this cannot detect, including problems affecting only regions with no checker. Each location's results are visible separately. Do not build yet: Nothing after this. ### Out of scope (and why) - A global probe network. Two locations is meaningfully better than one and a long way from checking every region your users are in. - Phone and text message delivery, which needs a provider relationship and is what you want at three in the morning. - Managed browser check operations at scale, including keeping Playwright and its browsers current across a fleet. - Someone to call when the monitoring itself breaks, which is a real gap since a broken monitor is silent by nature. ### README must contain - That the checker runs outside the monitored failure domain, and the written list of what they still share - That checks assert content rather than status codes, and why - That an incident requires consecutive failures, with the resulting detection time stated - That the status page and alerting are independent of the monitored infrastructure - That uptime accounts for checker gaps as unknown rather than counting them as up ===== AGENTS.md ===== # Agent instructions · Checkly indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with PostgreSQL, Playwright, headless, A server with a different provider and region from what it monitors, A path independent of your own infrastructure, Static, generated and served from somewhere separate. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. ===== BUILD_PLAN.md ===== # Build plan · Checkly A checker running outside your infrastructure that performs API and browser checks on a schedule, distinguishes a real failure from a blip before alerting, sends one alert per incident rather than one per check, and publishes a status page whose uptime figure is computed honestly including its own gaps. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A checker somewhere else Checks run from outside the failure domain of what they watch, and you have verified that. ### Steps 1. Deploy outside the failure domain Different provider and region. Not a different instance in the same account. Files: `docs/separation.md` 2. Run scheduled HTTP checks With a timeout, recording duration, status and error for every run. Files: `src/http-check.js` 3. Write down the shared dependencies Provider, region, DNS, certificate authority. Each one is a way the monitor can fail with the target. Files: `docs/separation.md` ### Done when - [ ] Checks run from a different provider and region - [ ] Every result records duration, status and error - [ ] The shared dependency list is written and reviewed - [ ] Stopping the target produces failures within one interval ### Watch out - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. ## Phase 2 · Checks that assert something real A check fails when the application is broken, not only when it is down. ### Steps 1. Assert content, not just status A value only a working application produces. Status codes lie routinely. Files: `src/assertions.js` 2. Separate degraded from failed A slow response is information; treating it as an outage produces noise. Files: `src/assertions.js` 3. Add a browser check with a screenshot A real flow, with a screenshot and the failing selector captured on failure. Files: `src/browser-check.js` ```sh npx playwright install chromium ``` ### Done when - [ ] A content assertion catches an error page returning two hundred - [ ] A slow response is marked degraded rather than failed - [ ] A browser check completes an authenticated flow - [ ] A browser failure captures a screenshot and names the selector ### Watch out - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. ## Phase 3 · Incidents, not results One outage produces one incident, and a blip produces nothing. ### Steps 1. Confirm before opening Consecutive failures past a threshold. The threshold and interval together define your detection time; state it. Files: `src/incidents.js` 2. Require stability to resolve Consecutive successes, so flapping does not close and reopen. Files: `src/incidents.js` 3. Test the flapping case Alternate failure and success around the threshold. This is where naive logic produces a storm. Files: `test/incidents.test.js` ### Done when - [ ] A single failure followed by success opens no incident - [ ] Consecutive failures open exactly one incident - [ ] Flapping produces one incident rather than a sequence - [ ] The incident records the first error ### Watch out - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. ## Phase 4 · Alerting that survives the outage You are told once, through a path that does not depend on what is broken. ### Steps 1. Alert once per state change Confirmed and resolved. Not per failed check. Files: `src/alerts.js` 2. Use an independent delivery path External provider, not your own infrastructure. Test by taking everything down. Files: `src/channels.js` 3. Add maintenance windows Suppress the alert, keep the downtime visible. Hiding it makes uptime figures dishonest. Files: `src/maintenance.js` ### Done when - [ ] One alert on confirmation and one on resolution - [ ] The alert arrives with the monitored infrastructure fully down - [ ] A failed delivery is retried and recorded - [ ] Maintenance suppresses alerts while downtime stays visible ### Watch out - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. ## Phase 5 · A status page that tells the truth The uptime figure is honest, including about the times the monitor was not looking. ### Steps 1. Generate a static page From incidents, published somewhere separate. Static because it must serve when everything else is gone. Files: `src/status.js` 2. Record checker gaps Whenever the checker was not running. These become unknown periods rather than uptime. Files: `src/gaps.js` 3. Verify against a hand calculation One month, computed by hand from the incident list. Uptime arithmetic is easy to get subtly wrong. ### Done when - [ ] The status page serves with the monitored system fully down - [ ] Maintenance is excluded and labelled - [ ] Checker gaps appear as unknown rather than up - [ ] Uptime matches a hand calculation ### Watch out - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. ## Not in this build - A global probe network. - Phone and text message delivery. - Managed browser check operations at scale. - Anyone to call when the monitoring breaks. ## After v1, if you want it - A third location, once two have disagreed usefully at least once - Response time percentiles per check, which reveal degradation before failure - Public incident write-ups attached to the status page history ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. On the checker, not on the monitored infrastructure. DATABASE_URL=postgres://user:pass@localhost:5432/checks # Required. Consecutive failures before an incident is confirmed. With the interval, this defines your detection time. CONFIRM_FAILURES=3 # Required. Consecutive successes before resolving. Higher values suppress flapping. RESOLVE_SUCCESSES=3 # Required. How often checks run. Shorter detects faster and costs more requests. DEFAULT_INTERVAL_SECONDS=60 # Required · secret. Independent delivery path. Never a webhook into the monitored system. ALERT_CHANNEL_URL=https://api.provider.example/send # Required. Where the generated status page is published. Independent hosting. STATUS_PAGE_DEST=s3://status-bucket # Required. How long without a checker run before the period is recorded as a gap rather than uptime. CHECKER_HEARTBEAT_SECONDS=120
You are building a production product version of Checkly. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Checkly · product brief ## Problem The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Checkly, run scheduled API and browser checks from one independent worker. The hard boundary is global browser fleet, playwright operations, alerting, ci integration, and support, plus independent infrastructure and reliable alerting. ## Product outcome Monitoring that can actually observe your worst outage, alerts once through a path that survives it, and publishes an uptime figure that is honest about its own blind spots. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - server outside the monitored failure domain - PostgreSQL - optional ClickHouse - email or webhook destination - public HTTPS ## Explicit non-goals for v1 - A global probe network. - Phone and text message delivery. - Managed browser check operations at scale. - Anyone to call when the monitoring breaks. - global browser fleet, Playwright operations, alerting, CI integration, and support - global probe network - phone and SMS delivery - massive retention - advanced incident response and support ## Success criteria - The shared dependency list is reviewed and short - A content assertion is confirmed to catch an error page returning two hundred - Flapping is confirmed to produce one incident - An alert is confirmed to arrive with the monitored infrastructure fully down - Uptime is confirmed to match a hand calculation including checker gaps ===== BRIEF.md ===== # Build brief · Checkly The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled checks that run from outside your infrastructure, alert once, and feed an honest status page to replace Checkly. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The monitor must not share a failure domain with what it monitors. A check running on the same server, in the same region or behind the same provider will go down with the thing it is watching, and the outage you most need to know about is the one it cannot report. That single constraint decides the architecture: the checker runs somewhere else, the alerting path does not depend on your application, and the status page is served from somewhere that survives your infrastructure being gone. Everything else here is straightforward. ### Stack (fixed, do not substitute) - Runtime: Node 22 with PostgreSQL · checks, results and incidents are relational and the incident logic needs transactions - Browser checks: Playwright, headless · a check that logs in and clicks through a flow catches what an endpoint check cannot - Location: A server with a different provider and region from what it monitors · this is the whole design constraint, not a deployment preference - Alerting: A path independent of your own infrastructure · alerting through a service you host means no alert during the outage that matters - Status page: Static, generated and served from somewhere separate · a status page hosted on the failing server is the oldest mistake in this category ### Data model (create this before Phase 1) - checks(id, name, kind, target, interval_seconds, timeout_ms, expected_json, enabled, degraded_after_ms) · kind is http or browser - results(check_id, at, ok, duration_ms, status_code, error, screenshot_key) - incidents(id, check_id, started_at, confirmed_at, resolved_at, state, cause_summary) · confirmed_at is when it passed the confirmation threshold - alerts(incident_id, at, channel, delivered, error) - maintenance(id, check_id, starts_at, ends_at, reason) · declared downtime excluded from uptime, and labelled on the status page - checker_gaps(started_at, ended_at, reason) · when the checker itself was not running, so uptime can be honest about it - An outage is a confirmed incident, not a single failed result. The confirmation threshold is what separates a monitor from a noise generator. ### Phase 1 · A checker somewhere else Build: Deploy the checker to a different provider and region from your application. Run a simple HTTP check on a schedule with a timeout. Record every result. Then confirm the separation by listing what the two share. Trap: A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. Done when: Checks run on a schedule from a different provider and region. Every result is recorded with duration and status. A written note lists what the checker and target share and why each is acceptable. Stopping the target produces failed results within one interval. Do not build yet: No alerting, incidents or status page. ### Phase 2 · Checks that assert something real Build: Assertions beyond a status code: response body content, a JSON field, and a response time threshold that marks degraded rather than failed. Then add a browser check that completes a real user flow. Trap: A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. Done when: A check asserts a body value and fails when the application returns an error page with a two hundred status. A response slower than the degraded threshold is marked degraded rather than failed. A browser check logs in and reaches an authenticated page, and captures a screenshot on failure. A browser check failing on a selector reports which selector. Do not build yet: No incident logic or alerting yet. ### Phase 3 · Incidents, not results Build: Require several consecutive failures before confirming an incident, and several consecutive successes before resolving it. Group failures into an incident with a start and an end. Record the first error as the cause summary. Trap: Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. Done when: A single failed check followed by a success produces no incident. Three consecutive failures open one incident. A flapping service produces one incident rather than a sequence, verified by alternating failure and success at the boundary. The incident records the first error as its cause summary. Do not build yet: No alerting yet. ### Phase 4 · Alerting that survives the outage Build: Send one alert when an incident is confirmed and one when it resolves. Use a channel that does not route through your own infrastructure. Retry delivery and record failures. Support a maintenance window that suppresses alerts without hiding the downtime. Trap: Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. Done when: An incident produces one alert and its resolution produces one more. Taking the monitored infrastructure down entirely still delivers the alert, tested deliberately. A failed delivery is retried and then recorded. A maintenance window suppresses the alert while the downtime still appears on the status page. Do not build yet: No status page yet. ### Phase 5 · A status page that tells the truth Build: Generate a static status page from incidents, served from somewhere independent. Compute uptime from confirmed incidents, exclude declared maintenance and label it, and account for periods when the checker itself was not running rather than counting them as up. Trap: Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. Done when: The status page is served from infrastructure independent of the monitored system, verified by taking the latter down. Uptime excludes declared maintenance and labels it. A period when the checker was not running appears as unknown rather than as up. The uptime figure matches a hand calculation over a known window. Do not build yet: Operations and multiple locations come last. ### Phase 6 · More locations, and knowing what you cannot see Build: Run checkers from a second location and require agreement before confirming a global incident. Report a single-location failure as regional. Then write down honestly what this setup cannot detect. Trap: One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is. Done when: Two locations run independently and a failure at one is reported as regional rather than global. Agreement across locations confirms a global incident. A written note records what this cannot detect, including problems affecting only regions with no checker. Each location's results are visible separately. Do not build yet: Nothing after this. ### Out of scope (and why) - A global probe network. Two locations is meaningfully better than one and a long way from checking every region your users are in. - Phone and text message delivery, which needs a provider relationship and is what you want at three in the morning. - Managed browser check operations at scale, including keeping Playwright and its browsers current across a fleet. - Someone to call when the monitoring itself breaks, which is a real gap since a broken monitor is silent by nature. ### README must contain - That the checker runs outside the monitored failure domain, and the written list of what they still share - That checks assert content rather than status codes, and why - That an incident requires consecutive failures, with the resulting detection time stated - That the status page and alerting are independent of the monitored infrastructure - That uptime accounts for checker gaps as unknown rather than counting them as up ===== ARCHITECTURE.md ===== # Architecture · Checkly ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with PostgreSQL | checks, results and incidents are relational and the incident logic needs transactions | | Browser checks | Playwright, headless | a check that logs in and clicks through a flow catches what an endpoint check cannot | | Location | A server with a different provider and region from what it monitors | this is the whole design constraint, not a deployment preference | | Alerting | A path independent of your own infrastructure | alerting through a service you host means no alert during the outage that matters | | Status page | Static, generated and served from somewhere separate | a status page hosted on the failing server is the oldest mistake in this category | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Checker | Running checks from outside the failure domain. | Location independence is the requirement. Any implementation that keeps it is fine; any that loses it defeats the purpose. | | Assertions | Deciding whether a response proves the system worked. | Content assertions rather than status codes is the correctness property. | | Incidents | Turning results into events. | The confirmation thresholds are the difference between a monitor and a noise source and should be tuned rather than removed. | | Alerting | Independent delivery. | Any external provider. Never a path through the monitored system. | | Status | An honest public figure. | Any static host. Accounting for checker gaps is what makes the number honest. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · On the checker, not on the monitored infrastructure. - `CONFIRM_FAILURES` · required · Consecutive failures before an incident is confirmed. With the interval, this defines your detection time. - `RESOLVE_SUCCESSES` · required · Consecutive successes before resolving. Higher values suppress flapping. - `DEFAULT_INTERVAL_SECONDS` · required · How often checks run. Shorter detects faster and costs more requests. - `ALERT_CHANNEL_URL` · required, secret · Independent delivery path. Never a webhook into the monitored system. - `STATUS_PAGE_DEST` · required · Where the generated status page is published. Independent hosting. - `CHECKER_HEARTBEAT_SECONDS` · required · How long without a checker run before the period is recorded as a gap rather than uptime. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Checkly product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with PostgreSQL, Playwright, headless, A server with a different provider and region from what it monitors, A path independent of your own infrastructure, Static, generated and served from somewhere separate. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. - One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is. ===== MILESTONES.md ===== # Delivery milestones · Checkly Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A checker somewhere else Checks run from outside the failure domain of what they watch, and you have verified that. ### Steps 1. Deploy outside the failure domain Different provider and region. Not a different instance in the same account. Files: `docs/separation.md` 2. Run scheduled HTTP checks With a timeout, recording duration, status and error for every run. Files: `src/http-check.js` 3. Write down the shared dependencies Provider, region, DNS, certificate authority. Each one is a way the monitor can fail with the target. Files: `docs/separation.md` ### Done when - [ ] Checks run from a different provider and region - [ ] Every result records duration, status and error - [ ] The shared dependency list is written and reviewed - [ ] Stopping the target produces failures within one interval ### Watch out - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. ## M2 · Checks that assert something real A check fails when the application is broken, not only when it is down. ### Steps 1. Assert content, not just status A value only a working application produces. Status codes lie routinely. Files: `src/assertions.js` 2. Separate degraded from failed A slow response is information; treating it as an outage produces noise. Files: `src/assertions.js` 3. Add a browser check with a screenshot A real flow, with a screenshot and the failing selector captured on failure. Files: `src/browser-check.js` ```sh npx playwright install chromium ``` ### Done when - [ ] A content assertion catches an error page returning two hundred - [ ] A slow response is marked degraded rather than failed - [ ] A browser check completes an authenticated flow - [ ] A browser failure captures a screenshot and names the selector ### Watch out - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. ## M3 · Incidents, not results One outage produces one incident, and a blip produces nothing. ### Steps 1. Confirm before opening Consecutive failures past a threshold. The threshold and interval together define your detection time; state it. Files: `src/incidents.js` 2. Require stability to resolve Consecutive successes, so flapping does not close and reopen. Files: `src/incidents.js` 3. Test the flapping case Alternate failure and success around the threshold. This is where naive logic produces a storm. Files: `test/incidents.test.js` ### Done when - [ ] A single failure followed by success opens no incident - [ ] Consecutive failures open exactly one incident - [ ] Flapping produces one incident rather than a sequence - [ ] The incident records the first error ### Watch out - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. ## M4 · Alerting that survives the outage You are told once, through a path that does not depend on what is broken. ### Steps 1. Alert once per state change Confirmed and resolved. Not per failed check. Files: `src/alerts.js` 2. Use an independent delivery path External provider, not your own infrastructure. Test by taking everything down. Files: `src/channels.js` 3. Add maintenance windows Suppress the alert, keep the downtime visible. Hiding it makes uptime figures dishonest. Files: `src/maintenance.js` ### Done when - [ ] One alert on confirmation and one on resolution - [ ] The alert arrives with the monitored infrastructure fully down - [ ] A failed delivery is retried and recorded - [ ] Maintenance suppresses alerts while downtime stays visible ### Watch out - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. ## M5 · A status page that tells the truth The uptime figure is honest, including about the times the monitor was not looking. ### Steps 1. Generate a static page From incidents, published somewhere separate. Static because it must serve when everything else is gone. Files: `src/status.js` 2. Record checker gaps Whenever the checker was not running. These become unknown periods rather than uptime. Files: `src/gaps.js` 3. Verify against a hand calculation One month, computed by hand from the incident list. Uptime arithmetic is easy to get subtly wrong. ### Done when - [ ] The status page serves with the monitored system fully down - [ ] Maintenance is excluded and labelled - [ ] Checker gaps appear as unknown rather than up - [ ] Uptime matches a hand calculation ### Watch out - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. ## M6 · More locations, and knowing what you cannot see (production only) A regional problem is distinguishable from a global one, and the blind spots are documented. ### Steps 1. Add a second location Different provider again. Two locations that share a provider tell you less than they appear to. Files: `docs/separation.md` 2. Require agreement for global One location failing is regional or a checker problem. Both failing is an outage. Files: `src/incidents.js` 3. Document the blind spots Regions with no checker, times with no coverage, and failure modes no check asserts. Files: `docs/coverage.md` ### Done when - [ ] A single-location failure is reported as regional - [ ] Agreement across locations confirms a global incident - [ ] Per-location results are visible - [ ] The blind spot document is written and current ### Watch out - One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is. ===== OPERATIONS.md ===== # Operations · Checkly ## Backup The incident history is the record of your reliability and is small. Results are large and mostly disposable beyond a retention window, so back up incidents and keep results only as long as you need them. ## Restore Restore incidents and regenerate the status page. Missing results affect detail rather than the uptime figure, which is computed from incidents. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring This is the monitoring, so the question is who monitors it. A heartbeat to an external service is the usual answer, and without it a stopped checker is silent by definition. ## Incident checklist A checker that stops is the failure mode with no natural alarm, which is what the heartbeat and the gap recording exist for. If uptime looks suspiciously high, check the gap log before believing it. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The shared dependency list is reviewed and short - [ ] A content assertion is confirmed to catch an error page returning two hundred - [ ] Flapping is confirmed to produce one incident - [ ] An alert is confirmed to arrive with the monitored infrastructure fully down - [ ] Uptime is confirmed to match a hand calculation including checker gaps ## Launch constraint Do not market omitted Checkly capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. On the checker, not on the monitored infrastructure. DATABASE_URL=postgres://user:pass@localhost:5432/checks # Required. Consecutive failures before an incident is confirmed. With the interval, this defines your detection time. CONFIRM_FAILURES=3 # Required. Consecutive successes before resolving. Higher values suppress flapping. RESOLVE_SUCCESSES=3 # Required. How often checks run. Shorter detects faster and costs more requests. DEFAULT_INTERVAL_SECONDS=60 # Required · secret. Independent delivery path. Never a webhook into the monitored system. ALERT_CHANNEL_URL=https://api.provider.example/send # Required. Where the generated status page is published. Independent hosting. STATUS_PAGE_DEST=s3://status-bucket # Required. How long without a checker run before the period is recorded as a gap rather than uptime. CHECKER_HEARTBEAT_SECONDS=120
README.md
# Checkly · indie build A checker running outside your infrastructure that performs API and browser checks on a schedule, distinguishes a real failure from a blip before alerting, sends one alert per incident rather than one per check, and publishes a status page whose uptime figure is computed honestly including its own gaps. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with PostgreSQL | checks, results and incidents are relational and the incident logic needs transactions | | Browser checks | Playwright, headless | a check that logs in and clicks through a flow catches what an endpoint check cannot | | Location | A server with a different provider and region from what it monitors | this is the whole design constraint, not a deployment preference | | Alerting | A path independent of your own infrastructure | alerting through a service you host means no alert during the outage that matters | | Status page | Static, generated and served from somewhere separate | a status page hosted on the failing server is the oldest mistake in this category | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the checker and Playwright. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **A server with a different provider from what you monitor** · 5 to 10 US dollars a month - Why: This is the entire design constraint. A monitor in the same failure domain cannot report the outage you built it for. - Get it: A small instance with a provider you do not otherwise use. It needs very little capacity; independence is the requirement rather than power. - Verify: A written list of shared dependencies exists and is short - [ ] **PostgreSQL 16 or newer** · free self-hosted - Why: Results, incidents and alerts, with the incident logic needing transactions. - Get it: On the checker server, or a managed instance from a third provider. - Verify: psql connects and reports 16 or higher - [ ] **An alerting channel independent of your infrastructure** · free tiers cover this volume - Why: An alert routed through something you host is an alert you will not receive during the outage that matters. - Get it: A transactional email provider or an external chat service. Not a webhook into your own application. - Verify: An alert arrives with your monitored infrastructure fully down - [ ] **Somewhere independent to serve the status page** · free on most static hosts - Why: A status page hosted on the failing infrastructure is the oldest mistake in this category and it is made repeatedly. - Get it: Static hosting with a different provider, or the checker server itself if that is genuinely separate. - Verify: The page serves while the monitored system is down ## Quick start ```sh npx playwright install chromium ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A global probe network. - Phone and text message delivery. - Managed browser check operations at scale. - Anyone to call when the monitoring breaks. - global browser fleet, Playwright operations, alerting, CI integration, and support - global probe network - phone and SMS delivery - massive retention - advanced incident response and support If one of those is essential to you, that is the reason to keep paying for Checkly, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Checkly The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled checks that run from outside your infrastructure, alert once, and feed an honest status page to replace Checkly. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The monitor must not share a failure domain with what it monitors. A check running on the same server, in the same region or behind the same provider will go down with the thing it is watching, and the outage you most need to know about is the one it cannot report. That single constraint decides the architecture: the checker runs somewhere else, the alerting path does not depend on your application, and the status page is served from somewhere that survives your infrastructure being gone. Everything else here is straightforward. ### Stack (fixed, do not substitute) - Runtime: Node 22 with PostgreSQL · checks, results and incidents are relational and the incident logic needs transactions - Browser checks: Playwright, headless · a check that logs in and clicks through a flow catches what an endpoint check cannot - Location: A server with a different provider and region from what it monitors · this is the whole design constraint, not a deployment preference - Alerting: A path independent of your own infrastructure · alerting through a service you host means no alert during the outage that matters - Status page: Static, generated and served from somewhere separate · a status page hosted on the failing server is the oldest mistake in this category ### Data model (create this before Phase 1) - checks(id, name, kind, target, interval_seconds, timeout_ms, expected_json, enabled, degraded_after_ms) · kind is http or browser - results(check_id, at, ok, duration_ms, status_code, error, screenshot_key) - incidents(id, check_id, started_at, confirmed_at, resolved_at, state, cause_summary) · confirmed_at is when it passed the confirmation threshold - alerts(incident_id, at, channel, delivered, error) - maintenance(id, check_id, starts_at, ends_at, reason) · declared downtime excluded from uptime, and labelled on the status page - checker_gaps(started_at, ended_at, reason) · when the checker itself was not running, so uptime can be honest about it - An outage is a confirmed incident, not a single failed result. The confirmation threshold is what separates a monitor from a noise generator. ### Phase 1 · A checker somewhere else Build: Deploy the checker to a different provider and region from your application. Run a simple HTTP check on a schedule with a timeout. Record every result. Then confirm the separation by listing what the two share. Trap: A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. Done when: Checks run on a schedule from a different provider and region. Every result is recorded with duration and status. A written note lists what the checker and target share and why each is acceptable. Stopping the target produces failed results within one interval. Do not build yet: No alerting, incidents or status page. ### Phase 2 · Checks that assert something real Build: Assertions beyond a status code: response body content, a JSON field, and a response time threshold that marks degraded rather than failed. Then add a browser check that completes a real user flow. Trap: A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. Done when: A check asserts a body value and fails when the application returns an error page with a two hundred status. A response slower than the degraded threshold is marked degraded rather than failed. A browser check logs in and reaches an authenticated page, and captures a screenshot on failure. A browser check failing on a selector reports which selector. Do not build yet: No incident logic or alerting yet. ### Phase 3 · Incidents, not results Build: Require several consecutive failures before confirming an incident, and several consecutive successes before resolving it. Group failures into an incident with a start and an end. Record the first error as the cause summary. Trap: Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. Done when: A single failed check followed by a success produces no incident. Three consecutive failures open one incident. A flapping service produces one incident rather than a sequence, verified by alternating failure and success at the boundary. The incident records the first error as its cause summary. Do not build yet: No alerting yet. ### Phase 4 · Alerting that survives the outage Build: Send one alert when an incident is confirmed and one when it resolves. Use a channel that does not route through your own infrastructure. Retry delivery and record failures. Support a maintenance window that suppresses alerts without hiding the downtime. Trap: Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. Done when: An incident produces one alert and its resolution produces one more. Taking the monitored infrastructure down entirely still delivers the alert, tested deliberately. A failed delivery is retried and then recorded. A maintenance window suppresses the alert while the downtime still appears on the status page. Do not build yet: No status page yet. ### Phase 5 · A status page that tells the truth Build: Generate a static status page from incidents, served from somewhere independent. Compute uptime from confirmed incidents, exclude declared maintenance and label it, and account for periods when the checker itself was not running rather than counting them as up. Trap: Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. Done when: The status page is served from infrastructure independent of the monitored system, verified by taking the latter down. Uptime excludes declared maintenance and labels it. A period when the checker was not running appears as unknown rather than as up. The uptime figure matches a hand calculation over a known window. Do not build yet: Operations and multiple locations come last. ### Phase 6 · More locations, and knowing what you cannot see Build: Run checkers from a second location and require agreement before confirming a global incident. Report a single-location failure as regional. Then write down honestly what this setup cannot detect. Trap: One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is. Done when: Two locations run independently and a failure at one is reported as regional rather than global. Agreement across locations confirms a global incident. A written note records what this cannot detect, including problems affecting only regions with no checker. Each location's results are visible separately. Do not build yet: Nothing after this. ### Out of scope (and why) - A global probe network. Two locations is meaningfully better than one and a long way from checking every region your users are in. - Phone and text message delivery, which needs a provider relationship and is what you want at three in the morning. - Managed browser check operations at scale, including keeping Playwright and its browsers current across a fleet. - Someone to call when the monitoring itself breaks, which is a real gap since a broken monitor is silent by nature. ### README must contain - That the checker runs outside the monitored failure domain, and the written list of what they still share - That checks assert content rather than status codes, and why - That an incident requires consecutive failures, with the resulting detection time stated - That the status page and alerting are independent of the monitored infrastructure - That uptime accounts for checker gaps as unknown rather than counting them as up
AGENTS.md
# Agent instructions · Checkly indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22 with PostgreSQL, Playwright, headless, A server with a different provider and region from what it monitors, A path independent of your own infrastructure, Static, generated and served from somewhere separate. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong.
BUILD_PLAN.md
# Build plan · Checkly A checker running outside your infrastructure that performs API and browser checks on a schedule, distinguishes a real failure from a blip before alerting, sends one alert per incident rather than one per check, and publishes a status page whose uptime figure is computed honestly including its own gaps. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A checker somewhere else Checks run from outside the failure domain of what they watch, and you have verified that. ### Steps 1. Deploy outside the failure domain Different provider and region. Not a different instance in the same account. Files: `docs/separation.md` 2. Run scheduled HTTP checks With a timeout, recording duration, status and error for every run. Files: `src/http-check.js` 3. Write down the shared dependencies Provider, region, DNS, certificate authority. Each one is a way the monitor can fail with the target. Files: `docs/separation.md` ### Done when - [ ] Checks run from a different provider and region - [ ] Every result records duration, status and error - [ ] The shared dependency list is written and reviewed - [ ] Stopping the target produces failures within one interval ### Watch out - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. ## Phase 2 · Checks that assert something real A check fails when the application is broken, not only when it is down. ### Steps 1. Assert content, not just status A value only a working application produces. Status codes lie routinely. Files: `src/assertions.js` 2. Separate degraded from failed A slow response is information; treating it as an outage produces noise. Files: `src/assertions.js` 3. Add a browser check with a screenshot A real flow, with a screenshot and the failing selector captured on failure. Files: `src/browser-check.js` ```sh npx playwright install chromium ``` ### Done when - [ ] A content assertion catches an error page returning two hundred - [ ] A slow response is marked degraded rather than failed - [ ] A browser check completes an authenticated flow - [ ] A browser failure captures a screenshot and names the selector ### Watch out - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. ## Phase 3 · Incidents, not results One outage produces one incident, and a blip produces nothing. ### Steps 1. Confirm before opening Consecutive failures past a threshold. The threshold and interval together define your detection time; state it. Files: `src/incidents.js` 2. Require stability to resolve Consecutive successes, so flapping does not close and reopen. Files: `src/incidents.js` 3. Test the flapping case Alternate failure and success around the threshold. This is where naive logic produces a storm. Files: `test/incidents.test.js` ### Done when - [ ] A single failure followed by success opens no incident - [ ] Consecutive failures open exactly one incident - [ ] Flapping produces one incident rather than a sequence - [ ] The incident records the first error ### Watch out - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. ## Phase 4 · Alerting that survives the outage You are told once, through a path that does not depend on what is broken. ### Steps 1. Alert once per state change Confirmed and resolved. Not per failed check. Files: `src/alerts.js` 2. Use an independent delivery path External provider, not your own infrastructure. Test by taking everything down. Files: `src/channels.js` 3. Add maintenance windows Suppress the alert, keep the downtime visible. Hiding it makes uptime figures dishonest. Files: `src/maintenance.js` ### Done when - [ ] One alert on confirmation and one on resolution - [ ] The alert arrives with the monitored infrastructure fully down - [ ] A failed delivery is retried and recorded - [ ] Maintenance suppresses alerts while downtime stays visible ### Watch out - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. ## Phase 5 · A status page that tells the truth The uptime figure is honest, including about the times the monitor was not looking. ### Steps 1. Generate a static page From incidents, published somewhere separate. Static because it must serve when everything else is gone. Files: `src/status.js` 2. Record checker gaps Whenever the checker was not running. These become unknown periods rather than uptime. Files: `src/gaps.js` 3. Verify against a hand calculation One month, computed by hand from the incident list. Uptime arithmetic is easy to get subtly wrong. ### Done when - [ ] The status page serves with the monitored system fully down - [ ] Maintenance is excluded and labelled - [ ] Checker gaps appear as unknown rather than up - [ ] Uptime matches a hand calculation ### Watch out - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. ## Not in this build - A global probe network. - Phone and text message delivery. - Managed browser check operations at scale. - Anyone to call when the monitoring breaks. ## After v1, if you want it - A third location, once two have disagreed usefully at least once - Response time percentiles per check, which reveal degradation before failure - Public incident write-ups attached to the status page history
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. On the checker, not on the monitored infrastructure. DATABASE_URL=postgres://user:pass@localhost:5432/checks # Required. Consecutive failures before an incident is confirmed. With the interval, this defines your detection time. CONFIRM_FAILURES=3 # Required. Consecutive successes before resolving. Higher values suppress flapping. RESOLVE_SUCCESSES=3 # Required. How often checks run. Shorter detects faster and costs more requests. DEFAULT_INTERVAL_SECONDS=60 # Required · secret. Independent delivery path. Never a webhook into the monitored system. ALERT_CHANNEL_URL=https://api.provider.example/send # Required. Where the generated status page is published. Independent hosting. STATUS_PAGE_DEST=s3://status-bucket # Required. How long without a checker run before the period is recorded as a gap rather than uptime. CHECKER_HEARTBEAT_SECONDS=120
PRODUCT.md
# Checkly · product brief ## Problem The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Checkly, run scheduled API and browser checks from one independent worker. The hard boundary is global browser fleet, playwright operations, alerting, ci integration, and support, plus independent infrastructure and reliable alerting. ## Product outcome Monitoring that can actually observe your worst outage, alerts once through a path that survives it, and publishes an uptime figure that is honest about its own blind spots. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - server outside the monitored failure domain - PostgreSQL - optional ClickHouse - email or webhook destination - public HTTPS ## Explicit non-goals for v1 - A global probe network. - Phone and text message delivery. - Managed browser check operations at scale. - Anyone to call when the monitoring breaks. - global browser fleet, Playwright operations, alerting, CI integration, and support - global probe network - phone and SMS delivery - massive retention - advanced incident response and support ## Success criteria - The shared dependency list is reviewed and short - A content assertion is confirmed to catch an error page returning two hundred - Flapping is confirmed to produce one incident - An alert is confirmed to arrive with the monitored infrastructure fully down - Uptime is confirmed to match a hand calculation including checker gaps
BRIEF.md
# Build brief · Checkly The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me scheduled checks that run from outside your infrastructure, alert once, and feed an honest status page to replace Checkly. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The monitor must not share a failure domain with what it monitors. A check running on the same server, in the same region or behind the same provider will go down with the thing it is watching, and the outage you most need to know about is the one it cannot report. That single constraint decides the architecture: the checker runs somewhere else, the alerting path does not depend on your application, and the status page is served from somewhere that survives your infrastructure being gone. Everything else here is straightforward. ### Stack (fixed, do not substitute) - Runtime: Node 22 with PostgreSQL · checks, results and incidents are relational and the incident logic needs transactions - Browser checks: Playwright, headless · a check that logs in and clicks through a flow catches what an endpoint check cannot - Location: A server with a different provider and region from what it monitors · this is the whole design constraint, not a deployment preference - Alerting: A path independent of your own infrastructure · alerting through a service you host means no alert during the outage that matters - Status page: Static, generated and served from somewhere separate · a status page hosted on the failing server is the oldest mistake in this category ### Data model (create this before Phase 1) - checks(id, name, kind, target, interval_seconds, timeout_ms, expected_json, enabled, degraded_after_ms) · kind is http or browser - results(check_id, at, ok, duration_ms, status_code, error, screenshot_key) - incidents(id, check_id, started_at, confirmed_at, resolved_at, state, cause_summary) · confirmed_at is when it passed the confirmation threshold - alerts(incident_id, at, channel, delivered, error) - maintenance(id, check_id, starts_at, ends_at, reason) · declared downtime excluded from uptime, and labelled on the status page - checker_gaps(started_at, ended_at, reason) · when the checker itself was not running, so uptime can be honest about it - An outage is a confirmed incident, not a single failed result. The confirmation threshold is what separates a monitor from a noise generator. ### Phase 1 · A checker somewhere else Build: Deploy the checker to a different provider and region from your application. Run a simple HTTP check on a schedule with a timeout. Record every result. Then confirm the separation by listing what the two share. Trap: A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. Done when: Checks run on a schedule from a different provider and region. Every result is recorded with duration and status. A written note lists what the checker and target share and why each is acceptable. Stopping the target produces failed results within one interval. Do not build yet: No alerting, incidents or status page. ### Phase 2 · Checks that assert something real Build: Assertions beyond a status code: response body content, a JSON field, and a response time threshold that marks degraded rather than failed. Then add a browser check that completes a real user flow. Trap: A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. Done when: A check asserts a body value and fails when the application returns an error page with a two hundred status. A response slower than the degraded threshold is marked degraded rather than failed. A browser check logs in and reaches an authenticated page, and captures a screenshot on failure. A browser check failing on a selector reports which selector. Do not build yet: No incident logic or alerting yet. ### Phase 3 · Incidents, not results Build: Require several consecutive failures before confirming an incident, and several consecutive successes before resolving it. Group failures into an incident with a start and an end. Record the first error as the cause summary. Trap: Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. Done when: A single failed check followed by a success produces no incident. Three consecutive failures open one incident. A flapping service produces one incident rather than a sequence, verified by alternating failure and success at the boundary. The incident records the first error as its cause summary. Do not build yet: No alerting yet. ### Phase 4 · Alerting that survives the outage Build: Send one alert when an incident is confirmed and one when it resolves. Use a channel that does not route through your own infrastructure. Retry delivery and record failures. Support a maintenance window that suppresses alerts without hiding the downtime. Trap: Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. Done when: An incident produces one alert and its resolution produces one more. Taking the monitored infrastructure down entirely still delivers the alert, tested deliberately. A failed delivery is retried and then recorded. A maintenance window suppresses the alert while the downtime still appears on the status page. Do not build yet: No status page yet. ### Phase 5 · A status page that tells the truth Build: Generate a static status page from incidents, served from somewhere independent. Compute uptime from confirmed incidents, exclude declared maintenance and label it, and account for periods when the checker itself was not running rather than counting them as up. Trap: Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. Done when: The status page is served from infrastructure independent of the monitored system, verified by taking the latter down. Uptime excludes declared maintenance and labels it. A period when the checker was not running appears as unknown rather than as up. The uptime figure matches a hand calculation over a known window. Do not build yet: Operations and multiple locations come last. ### Phase 6 · More locations, and knowing what you cannot see Build: Run checkers from a second location and require agreement before confirming a global incident. Report a single-location failure as regional. Then write down honestly what this setup cannot detect. Trap: One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is. Done when: Two locations run independently and a failure at one is reported as regional rather than global. Agreement across locations confirms a global incident. A written note records what this cannot detect, including problems affecting only regions with no checker. Each location's results are visible separately. Do not build yet: Nothing after this. ### Out of scope (and why) - A global probe network. Two locations is meaningfully better than one and a long way from checking every region your users are in. - Phone and text message delivery, which needs a provider relationship and is what you want at three in the morning. - Managed browser check operations at scale, including keeping Playwright and its browsers current across a fleet. - Someone to call when the monitoring itself breaks, which is a real gap since a broken monitor is silent by nature. ### README must contain - That the checker runs outside the monitored failure domain, and the written list of what they still share - That checks assert content rather than status codes, and why - That an incident requires consecutive failures, with the resulting detection time stated - That the status page and alerting are independent of the monitored infrastructure - That uptime accounts for checker gaps as unknown rather than counting them as up
ARCHITECTURE.md
# Architecture · Checkly ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22 with PostgreSQL | checks, results and incidents are relational and the incident logic needs transactions | | Browser checks | Playwright, headless | a check that logs in and clicks through a flow catches what an endpoint check cannot | | Location | A server with a different provider and region from what it monitors | this is the whole design constraint, not a deployment preference | | Alerting | A path independent of your own infrastructure | alerting through a service you host means no alert during the outage that matters | | Status page | Static, generated and served from somewhere separate | a status page hosted on the failing server is the oldest mistake in this category | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Checker | Running checks from outside the failure domain. | Location independence is the requirement. Any implementation that keeps it is fine; any that loses it defeats the purpose. | | Assertions | Deciding whether a response proves the system worked. | Content assertions rather than status codes is the correctness property. | | Incidents | Turning results into events. | The confirmation thresholds are the difference between a monitor and a noise source and should be tuned rather than removed. | | Alerting | Independent delivery. | Any external provider. Never a path through the monitored system. | | Status | An honest public figure. | Any static host. Accounting for checker gaps is what makes the number honest. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · On the checker, not on the monitored infrastructure. - `CONFIRM_FAILURES` · required · Consecutive failures before an incident is confirmed. With the interval, this defines your detection time. - `RESOLVE_SUCCESSES` · required · Consecutive successes before resolving. Higher values suppress flapping. - `DEFAULT_INTERVAL_SECONDS` · required · How often checks run. Shorter detects faster and costs more requests. - `ALERT_CHANNEL_URL` · required, secret · Independent delivery path. Never a webhook into the monitored system. - `STATUS_PAGE_DEST` · required · Where the generated status page is published. Independent hosting. - `CHECKER_HEARTBEAT_SECONDS` · required · How long without a checker run before the period is recorded as a gap rather than uptime. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Checkly product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22 with PostgreSQL, Playwright, headless, A server with a different provider and region from what it monitors, A path independent of your own infrastructure, Static, generated and served from somewhere separate. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. - One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is.
MILESTONES.md
# Delivery milestones · Checkly Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A checker somewhere else Checks run from outside the failure domain of what they watch, and you have verified that. ### Steps 1. Deploy outside the failure domain Different provider and region. Not a different instance in the same account. Files: `docs/separation.md` 2. Run scheduled HTTP checks With a timeout, recording duration, status and error for every run. Files: `src/http-check.js` 3. Write down the shared dependencies Provider, region, DNS, certificate authority. Each one is a way the monitor can fail with the target. Files: `docs/separation.md` ### Done when - [ ] Checks run from a different provider and region - [ ] Every result records duration, status and error - [ ] The shared dependency list is written and reviewed - [ ] Stopping the target produces failures within one interval ### Watch out - A monitor in the same cloud region as the application will miss a regional outage entirely, and that is precisely the event you built it for. Write down what the monitor and the target share, including provider, region, DNS and certificate authority, and be deliberate about each. ## M2 · Checks that assert something real A check fails when the application is broken, not only when it is down. ### Steps 1. Assert content, not just status A value only a working application produces. Status codes lie routinely. Files: `src/assertions.js` 2. Separate degraded from failed A slow response is information; treating it as an outage produces noise. Files: `src/assertions.js` 3. Add a browser check with a screenshot A real flow, with a screenshot and the failing selector captured on failure. Files: `src/browser-check.js` ```sh npx playwright install chromium ``` ### Done when - [ ] A content assertion catches an error page returning two hundred - [ ] A slow response is marked degraded rather than failed - [ ] A browser check completes an authenticated flow - [ ] A browser failure captures a screenshot and names the selector ### Watch out - A check asserting only a two hundred status code passes while the application returns an error page, an empty result set or a login screen. Assert something that proves the application actually worked, which usually means a value that could only be produced by a working system. ## M3 · Incidents, not results One outage produces one incident, and a blip produces nothing. ### Steps 1. Confirm before opening Consecutive failures past a threshold. The threshold and interval together define your detection time; state it. Files: `src/incidents.js` 2. Require stability to resolve Consecutive successes, so flapping does not close and reopen. Files: `src/incidents.js` 3. Test the flapping case Alternate failure and success around the threshold. This is where naive logic produces a storm. Files: `test/incidents.test.js` ### Done when - [ ] A single failure followed by success opens no incident - [ ] Consecutive failures open exactly one incident - [ ] Flapping produces one incident rather than a sequence - [ ] The incident records the first error ### Watch out - Alerting on every failed result turns a thirty second network blip into a page at three in the morning, and after two of those the alerts get muted. Confirm with consecutive failures, and require consecutive successes to resolve so a flapping service does not open and close incidents repeatedly. ## M4 · Alerting that survives the outage You are told once, through a path that does not depend on what is broken. ### Steps 1. Alert once per state change Confirmed and resolved. Not per failed check. Files: `src/alerts.js` 2. Use an independent delivery path External provider, not your own infrastructure. Test by taking everything down. Files: `src/channels.js` 3. Add maintenance windows Suppress the alert, keep the downtime visible. Hiding it makes uptime figures dishonest. Files: `src/maintenance.js` ### Done when - [ ] One alert on confirmation and one on resolution - [ ] The alert arrives with the monitored infrastructure fully down - [ ] A failed delivery is retried and recorded - [ ] Maintenance suppresses alerts while downtime stays visible ### Watch out - Alerting through a webhook into a service you also host means the outage takes the alert with it. Use an external provider for delivery, and test the whole path by taking your infrastructure down deliberately rather than by taking one service down. ## M5 · A status page that tells the truth The uptime figure is honest, including about the times the monitor was not looking. ### Steps 1. Generate a static page From incidents, published somewhere separate. Static because it must serve when everything else is gone. Files: `src/status.js` 2. Record checker gaps Whenever the checker was not running. These become unknown periods rather than uptime. Files: `src/gaps.js` 3. Verify against a hand calculation One month, computed by hand from the incident list. Uptime arithmetic is easy to get subtly wrong. ### Done when - [ ] The status page serves with the monitored system fully down - [ ] Maintenance is excluded and labelled - [ ] Checker gaps appear as unknown rather than up - [ ] Uptime matches a hand calculation ### Watch out - Computing uptime as the proportion of successful checks counts a period when the checker was down as no data and therefore, silently, as up. Record checker gaps explicitly and show them, because an uptime figure that quietly excludes its own blind spots is the number every status page gets wrong. ## M6 · More locations, and knowing what you cannot see (production only) A regional problem is distinguishable from a global one, and the blind spots are documented. ### Steps 1. Add a second location Different provider again. Two locations that share a provider tell you less than they appear to. Files: `docs/separation.md` 2. Require agreement for global One location failing is regional or a checker problem. Both failing is an outage. Files: `src/incidents.js` 3. Document the blind spots Regions with no checker, times with no coverage, and failure modes no check asserts. Files: `docs/coverage.md` ### Done when - [ ] A single-location failure is reported as regional - [ ] Agreement across locations confirms a global incident - [ ] Per-location results are visible - [ ] The blind spot document is written and current ### Watch out - One checker cannot tell the difference between your service being down and its own network path being broken, and it will report the second as the first. A second location that disagrees turns a false alarm into useful information about where the problem is.
OPERATIONS.md
# Operations · Checkly ## Backup The incident history is the record of your reliability and is small. Results are large and mostly disposable beyond a retention window, so back up incidents and keep results only as long as you need them. ## Restore Restore incidents and regenerate the status page. Missing results affect detail rather than the uptime figure, which is computed from incidents. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring This is the monitoring, so the question is who monitors it. A heartbeat to an external service is the usual answer, and without it a stopped checker is silent by definition. ## Incident checklist A checker that stops is the failure mode with no natural alarm, which is what the heartbeat and the gap recording exist for. If uptime looks suspiciously high, check the gap log before believing it. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The shared dependency list is reviewed and short - [ ] A content assertion is confirmed to catch an error page returning two hundred - [ ] Flapping is confirmed to produce one incident - [ ] An alert is confirmed to arrive with the monitored infrastructure fully down - [ ] Uptime is confirmed to match a hand calculation including checker gaps ## Launch constraint Do not market omitted Checkly capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. On the checker, not on the monitored infrastructure. DATABASE_URL=postgres://user:pass@localhost:5432/checks # Required. Consecutive failures before an incident is confirmed. With the interval, this defines your detection time. CONFIRM_FAILURES=3 # Required. Consecutive successes before resolving. Higher values suppress flapping. RESOLVE_SUCCESSES=3 # Required. How often checks run. Shorter detects faster and costs more requests. DEFAULT_INTERVAL_SECONDS=60 # Required · secret. Independent delivery path. Never a webhook into the monitored system. ALERT_CHANNEL_URL=https://api.provider.example/send # Required. Where the generated status page is published. Independent hosting. STATUS_PAGE_DEST=s3://status-bucket # Required. How long without a checker run before the period is recorded as a gap rather than uptime. CHECKER_HEARTBEAT_SECONDS=120
$ choose a build depth, inspect the files, then open the complete pack in your agent
People still pay for Checkly because monitoring must continue working during the exact outage it reports, which makes independent infrastructure and alert delivery the real product. The recurring cost buys probe geography, clocks, retries, deduplication, sampling, storage, paging, notification delivery, on-call rules, and its own uptime, not just the visible interface.
xglobal browser fleet, Playwright operations, alerting, CI integration, and support
xglobal probe network
xphone and SMS delivery
xmassive retention
xadvanced incident response and support
share on X ↗"I just replaced Checkly ($24/mo) with one prompt"
Don't feel like building it? These folks already made it free.
OneUptimeScheduled Playwright and API checks, plus an incident system you did not ask for.open source↗no votes, no pay-to-list · just what's real
Checkly pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| hobby | $0/workspace | $0/workspace | 1 user; 10 uptime monitors; 2-minute minimum interval; 1,000 browser/Playwright runs; 10,000 API runs; 7-day raw retention |
| starter | — | $24/workspace | 3 users; 50 uptime monitors; 1-minute interval; 3,000 browser runs; 25,000 API runs; 100 SMS alerts; 7-day raw retention |
| team | — | $64/workspace | 10 users; 75 uptime monitors; 30-second interval; 12,000 browser runs; 100,000 API runs; 200 SMS + 200 phone alerts; 30-day raw retention |
| enterprise | custom | — | Custom users and monitor volumes; 1-second scheduling available; 180-day raw and 25-month aggregate retention |
free tier1 user; 10 uptime monitors; 1,000 browser/Playwright runs/month; 10,000 API runs/month; 6 public locations; 0 SMS and 0 phone alerts; 7-day raw and 30-day aggregate retention
billingmonthly + annual (up to 20% off); paid plans are prepaid and overages are invoiced in the next cycle
hidden costsStarter/Team charge $8 per extra block of 25 uptime monitors; browser overages are $6.50/$6.25 per 1,000 and API overages $2.60/$2.50 per 10,000; every parallel location and retry counts, and each 30 seconds of a Playwright suite consumes 1 run
verified 2026-08-14 · source ↗
Vibecode Checkly
Kinda. The core of Checkly is buildable in a weekend with the prompt on this page, but there are real gaps: global browser fleet, Playwright operations, alerting, CI integration, and support, global probe network. Read the honest list above before committing.
How much does Checkly cost?
Checkly costs about $24/month (Starter, checked 2026-08-14), which is $288 per year.
What do I lose by replacing Checkly?
Honestly: global browser fleet, Playwright operations, alerting, CI integration, and support; global probe network; phone and SMS delivery; massive retention; advanced incident response and support. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Checkly?
Yes: OneUptime (Scheduled Playwright and API checks, plus an incident system you did not ask for.) The prompt is for when you want it exactly your way.