KINDA · weekend project
The visible B2B product analytics loop is buildable, but a credible replacement needs more than the first screen. June earns its keep through ingestion, storage, reliability, so expect a weekend or multi-day build and a narrower personal scope.
the project pack5 files · written for this build, step by step
You are building a lean indie version of June. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # June · indie build A first-party analytics collector whose hard problems are a browser flush that actually fires on mobile, an account rollup that does not invent companies out of email domains, and reports that return the same figure twice. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small Node 22 endpoint that validates, truncates and batches, never blocking the page | an analytics endpoint that is slow becomes an analytics endpoint people remove | | Event store | ClickHouse, with events append-only and never edited | a product analytics query is a column scan and everything else is a slow surprise later | | Configuration | SQLite for accounts, memberships, report definitions and exclusion rules | these are small, relational, and you will edit them by hand at two in the morning | | Client | A tiny first-party script that queues locally and flushes on the visibility change | the tail of every mobile session is lost if you flush on unload | | Reports | Definitions stored as data, with one shared sessionisation function | two reports that disagree by ten percent cost a day each time | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the collector, the report server and the jobs. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **A ClickHouse instance** · free locally; a small managed instance is a few units of currency a month - Why: Product analytics queries scan columns over millions of rows, and a row store makes every report slow enough that you stop opening them. - Get it: Run it in a container locally, or use a small managed instance. - Verify: A ten million row table answers a group-by in under a second - [ ] **Measurement mode or identified mode** · free - Why: It decides whether you need consent, whether the account rollup exists at all, and what the interface has to say, and deciding it after launch means deciding it from whatever you already collected. - Get it: Read the current regulatory guidance for where your users are, pick a mode, and write down the lawful basis if it is the second. - Verify: The mode is set in configuration, stated in the README and shown in the interface - [ ] **A list of public mail providers** · free - Why: Without it the account rollup invents a very large fictional customer and you will believe it. - Get it: Any maintained public list, checked into the repository so the behaviour is reproducible. - Verify: The rollup test refuses to group two users from the same public provider - [ ] **A real product with real traffic** · free - Why: Every hard problem here appears only with real browsers: closed tabs, a proxy stripping headers, a customer's monitoring hitting you every minute. - Get it: Point the client at something you already run, even at low volume. - Verify: Events arrive from at least two browsers and one mobile device ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Session replay. - A managed ingestion pipeline at scale. - Licensed bot and device identification lists. - Cross-site or cross-product identity. - enterprise governance and integrations - bank, payment, tax, and reconciliation integrations - high-volume ingestion and retention - bot and identity resolution - session replay privacy tooling If one of those is essential to you, that is the reason to keep paying for June, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · June The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party product analytics collector for one product that rolls events up to accounts and is explicit about which of its numbers you can rely on to replace June. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Analytics is regulated where your users are, and the exemption that lets a first-party measurement tool skip a consent banner is narrower than the product you are about to build. The French regulator's guidance on using analytics sets out what an exempt tool must do: inform users and let them object, restrict the purpose to audience measurement and A/B testing, not cross-check the data with other processing such as a customer file or statistics about visits to other sites, keep the identifier scoped to a single publisher, truncate the last byte of the IP address, and limit the tracker lifetime to thirteen months. It also states that most large audience measurement offerings fall outside the exemption whatever their configuration. Read the current text yourself, and read the rules where your users actually are rather than only these. Now read that list against what account-level product analytics is. Rolling users up to companies by joining events to your customer records is cross-checking with another processing operation. So this build has two modes and makes you choose. Measurement mode never joins to a customer file, never leaves the product, truncates at ingest and expires the identifier. Identified mode does the account rollup and needs a lawful basis, a consent record per person and a sentence in the interface saying so. Measurement mode is the default and the README says which one you switched on. And the second rule, which is about arithmetic rather than law: a number nobody can reproduce is not a number. Every figure states its window, its session rule and its exclusions, and clicks through to the rows underneath it. ### Stack (fixed, do not substitute) - Collector: A small Node 22 endpoint that validates, truncates and batches, never blocking the page · an analytics endpoint that is slow becomes an analytics endpoint people remove - Event store: ClickHouse, with events append-only and never edited · a product analytics query is a column scan and everything else is a slow surprise later - Configuration: SQLite for accounts, memberships, report definitions and exclusion rules · these are small, relational, and you will edit them by hand at two in the morning - Client: A tiny first-party script that queues locally and flushes on the visibility change · the tail of every mobile session is lost if you flush on unload - Reports: Definitions stored as data, with one shared sessionisation function · two reports that disagree by ten percent cost a day each time ### Data model (create this before Phase 1) - events(at, event_name, anon_id, user_id, account_id, props_json, url_path, referrer_host, ua_family, ip_prefix, ingest_at, schema_version) · ip_prefix is truncated at ingest and the full address is written nowhere - identities(anon_id, user_id, first_seen_at, linked_at) · append only, so a wrong link can be seen rather than only its result - accounts(id, name, domain, source, created_at) · source records how the mapping was made, which is the only way to audit it later - account_members(user_id, account_id, from_at, to_at) · ranged, because a report over last quarter must use the membership as it was then - event_schema(event_name, prop, type, first_seen_at, last_seen_at, sample_count) · discovered rather than declared, so schema drift shows up as a row - exclusions(id, rule_kind, pattern, reason, effective_from) · bots, internal users and load tests, each with a stated reason and a date - reports(id, name, definition_json, created_at) · the definition is the report, and it is versioned rather than edited in place - There is no column anywhere holding a full IP address or a raw user agent string. ### Phase 1 · A collector that does not lose the end of the session Build: A batch endpoint that validates against the discovered schema, truncates the address, stamps ingest time and writes in batches. A client that queues events locally and flushes when the page becomes hidden, with the queue surviving into the next page load. Split batches below the queue limit and treat a refused flush as something to retry rather than something to drop. Trap: Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. Done when: The client flushes on the visibility change to hidden and the request completes after the page is gone, checked in a browser with the network log open. A refused flush leaves the events in the local queue and they arrive on the next page load, tested by forcing a refusal. The full IP address appears nowhere in storage or logs, asserted by a test that scans both. A batch above the configured size is split before sending, tested at the boundary. Do not build yet: No identity, accounts, reports or exclusions yet. ### Phase 2 · Accounts that exist, and joins that are named Build: Anonymous identifiers first, a user identifier on identify, and account membership as an explicit mapping with a source and a validity range. Keep a deny list of public mail providers. Implement the two modes: in measurement mode no code path reads an external customer file, and in identified mode attribution requires a consent record for that person. Trap: Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. Done when: Two users on the same public mail provider are never placed in the same account, tested against the deny list. A membership change is recorded with a range and a report over an earlier window uses the membership as it was then, tested. In measurement mode there is no code path that reads a customer file, asserted by a test over the module graph. In identified mode an event for a person with no consent record is stored without the account attribution rather than with it. Do not build yet: No report definitions, exclusions, retention or export yet. ### Phase 3 · The same question, the same answer, twice Build: Store report definitions as data. Put the sessionisation rule in one function that every report calls. Keep unique counts exact at this volume. Print the window, the session rule and the active exclusions on the report itself, and make every figure click through to the events behind it. Trap: Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. Done when: Running a report over a closed past window twice returns byte-identical results, tested. Every report renders its window, sessionisation parameters and the exclusions applied. Clicking any figure lists the underlying events and the count of that list matches the figure, tested. Unique counts are exact and no approximating function appears in any report query, checked by scanning the generated SQL. Do not build yet: No exclusion management, retention, deletion or export yet. ### Phase 4 · Bots, staff and the traffic you should not have counted Build: Exclusion rules carrying a reason and an effective date, applied at query time rather than at ingest. A view showing how much volume each rule removes over the current window. Rules can be disabled and the affected reports recompute. Keep the excluded rows. Trap: Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. Done when: No exclusion rule deletes or edits a row, asserted by checking that the event store has no delete path. A report over a window before a rule's effective date is unchanged by that rule, tested. Every rule shows the volume it removes over the current window. Disabling a rule and re-running a report changes the figure and the report says which rules were active, tested. Do not build yet: No retention, deletion or export yet. ### Phase 5 · Retention, deletion and saying when a number is an estimate Build: Scheduled jobs enforcing the identifier lifetime and the event retention, writing what they did. A command that deletes every row for a person or an account and produces a receipt. Export of definitions and aggregates. If volume forces sampling, mark every affected figure as sampled wherever it appears, including in exports. Trap: A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page. Done when: The retention job deletes events past the configured age and writes a row recording the count and range, tested. Deleting a person removes their events, identities and memberships and produces a receipt listing the counts, tested. A sampled figure is marked as sampled in the interface and in the export, tested by forcing sampling on. Export of report definitions and aggregates reimports into an empty installation and reproduces the same figures. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay, which turns an analytics store into a recording of people using your product and is a different consent conversation entirely. - A managed ingestion pipeline at scale, which is where the subscription's money goes and is an operations job rather than a build. - Bought bot and device identification lists, which are licensed data you cannot redistribute and would have to keep paying for. - Cross-site or cross-product identity, which is the exact thing the measurement exemption forbids and the reason it exists. ### README must contain - Which mode the installation is running in, measurement or identified, and what changes between them - The conditions the regulator sets for consent-exempt audience measurement, where that text is published, and which of them this build implements - That the client flushes on the visibility change and what happens to events when a flush is refused - How accounts are mapped to users, that the mapping is explicit, and why email domains are not used - The retention periods, how a deletion request is executed, and what the receipt contains ===== AGENTS.md ===== # Agent instructions · June indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A small Node 22 endpoint that validates, truncates and batches, never blocking the page, ClickHouse, with events append-only and never edited, SQLite for accounts, memberships, report definitions and exclusion rules, A tiny first-party script that queues locally and flushes on the visibility change, Definitions stored as data, with one shared sessionisation function. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. ===== BUILD_PLAN.md ===== # Build plan · June A first-party analytics collector whose hard problems are a browser flush that actually fires on mobile, an account rollup that does not invent companies out of email domains, and reports that return the same figure twice. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A collector that does not lose the end of the session Events reach the store from real browsers, including the ones that close without warning. ### Steps 1. Flush on the visibility change, not on unload Unload is the event mobile browsers are allowed to skip. Files: `client/collect.js`, `docs/client.md` 2. Keep a local queue that survives a page load A refused flush should be a delay, not a loss. Files: `client/queue.js`, `tests/queue.test.ts` 3. Truncate the address at ingest, before anything is written The only place a full address cannot leak from is the place it never reached. Files: `src/collect.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] The hidden-visibility flush completes after the page is gone - [ ] A refused flush is retried on the next load - [ ] No full IP address exists in storage or logs - [ ] Oversized batches are split before sending ### Watch out - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. ## Phase 2 · Accounts that exist, and joins that are named A user belongs to a company because somebody said so, not because of the text after the at sign. ### Steps 1. Make membership explicit and ranged People move between companies and last quarter did not change. Files: `src/accounts.ts`, `tests/membership.test.ts` 2. Keep a deny list of public mail providers Otherwise your largest customer is a free mail service. Files: `config/public-domains.txt`, `tests/rollup.test.ts` 3. Make the two modes a build-time boundary Measurement mode should be unable to reach the customer file. Files: `src/modes.ts`, `tests/mode-isolation.test.ts` ### Done when - [ ] Public mail domains never form an account - [ ] Past windows use the membership of that time - [ ] Measurement mode cannot reach a customer file - [ ] Identified mode requires a consent record to attribute ### Watch out - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. ## Phase 3 · The same question, the same answer, twice A report over a closed past window returns identical numbers every time it runs. ### Steps 1. One sessionisation function, called by everything Two definitions of a session is two products. Files: `src/reports/session.ts`, `tests/session.test.ts` 2. Print the parameters on the report A figure without its window is a rumour. Files: `src/reports/render.tsx` 3. Make every figure click through to rows The fastest way to find out a number is wrong. Files: `src/reports/drill.ts`, `tests/drill.test.ts` ### Done when - [ ] A closed window re-runs identically - [ ] Reports display window, session rule and exclusions - [ ] Every figure drills through to matching rows - [ ] No approximating count appears in report queries ### Watch out - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. ## Phase 4 · Bots, staff and the traffic you should not have counted Exclusions are revisable because nothing was deleted to make them. ### Steps 1. Exclude at query time, never at ingest You cannot un-drop an event. Files: `src/exclusions.ts`, `tests/exclusions.test.ts` 2. Give every rule a reason and an effective date A rule with no date rewrites history when it is edited. Files: `src/db/exclusions.sql` 3. Show the volume each rule removes So an over-broad rule is visible rather than believed. Files: `src/reports/exclusions.tsx` ### Done when - [ ] No exclusion deletes a row - [ ] Rules do not affect windows before their date - [ ] Removed volume per rule is visible - [ ] Reports name the rules that were active ### Watch out - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. ## Not in this build - Session replay. - A managed ingestion pipeline at scale. - Licensed bot and device identification lists. - Cross-site or cross-product identity. ## After v1, if you want it - A weekly digest of schema drift, built from the discovered schema table rather than from memory - An annotation layer so a release or an outage sits on the chart next to the dip it caused - A second opinion query for one headline figure, written independently, to catch the day the first one drifts ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. measurement or identified. Decides whether the account rollup exists at all. COLLECT_MODE=measurement # Required. The event store. Append only, never edited by the application. CLICKHOUSE_URL=http://localhost:8123 # Required. Accounts, memberships, report definitions and exclusion rules. CONFIG_DB_PATH=./data/config.db # Required. Tracker lifetime. Check the current guidance for the figure that applies to you. IDENTIFIER_LIFETIME_DAYS=395 # Required. Enforced by a scheduled job that logs what it deleted. EVENT_RETENTION_DAYS=760 # Required. Client batch size, kept under the browser queue limit. Verify the current limit. BATCH_MAX_BYTES=32768 # Required. The single sessionisation parameter, printed on every report that uses it. SESSION_GAP_MINUTES=30
You are building a lean indie version of June. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # June · indie build A first-party analytics collector whose hard problems are a browser flush that actually fires on mobile, an account rollup that does not invent companies out of email domains, and reports that return the same figure twice. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small Node 22 endpoint that validates, truncates and batches, never blocking the page | an analytics endpoint that is slow becomes an analytics endpoint people remove | | Event store | ClickHouse, with events append-only and never edited | a product analytics query is a column scan and everything else is a slow surprise later | | Configuration | SQLite for accounts, memberships, report definitions and exclusion rules | these are small, relational, and you will edit them by hand at two in the morning | | Client | A tiny first-party script that queues locally and flushes on the visibility change | the tail of every mobile session is lost if you flush on unload | | Reports | Definitions stored as data, with one shared sessionisation function | two reports that disagree by ten percent cost a day each time | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the collector, the report server and the jobs. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **A ClickHouse instance** · free locally; a small managed instance is a few units of currency a month - Why: Product analytics queries scan columns over millions of rows, and a row store makes every report slow enough that you stop opening them. - Get it: Run it in a container locally, or use a small managed instance. - Verify: A ten million row table answers a group-by in under a second - [ ] **Measurement mode or identified mode** · free - Why: It decides whether you need consent, whether the account rollup exists at all, and what the interface has to say, and deciding it after launch means deciding it from whatever you already collected. - Get it: Read the current regulatory guidance for where your users are, pick a mode, and write down the lawful basis if it is the second. - Verify: The mode is set in configuration, stated in the README and shown in the interface - [ ] **A list of public mail providers** · free - Why: Without it the account rollup invents a very large fictional customer and you will believe it. - Get it: Any maintained public list, checked into the repository so the behaviour is reproducible. - Verify: The rollup test refuses to group two users from the same public provider - [ ] **A real product with real traffic** · free - Why: Every hard problem here appears only with real browsers: closed tabs, a proxy stripping headers, a customer's monitoring hitting you every minute. - Get it: Point the client at something you already run, even at low volume. - Verify: Events arrive from at least two browsers and one mobile device ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Session replay. - A managed ingestion pipeline at scale. - Licensed bot and device identification lists. - Cross-site or cross-product identity. - enterprise governance and integrations - bank, payment, tax, and reconciliation integrations - high-volume ingestion and retention - bot and identity resolution - session replay privacy tooling If one of those is essential to you, that is the reason to keep paying for June, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · June The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party product analytics collector for one product that rolls events up to accounts and is explicit about which of its numbers you can rely on to replace June. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Analytics is regulated where your users are, and the exemption that lets a first-party measurement tool skip a consent banner is narrower than the product you are about to build. The French regulator's guidance on using analytics sets out what an exempt tool must do: inform users and let them object, restrict the purpose to audience measurement and A/B testing, not cross-check the data with other processing such as a customer file or statistics about visits to other sites, keep the identifier scoped to a single publisher, truncate the last byte of the IP address, and limit the tracker lifetime to thirteen months. It also states that most large audience measurement offerings fall outside the exemption whatever their configuration. Read the current text yourself, and read the rules where your users actually are rather than only these. Now read that list against what account-level product analytics is. Rolling users up to companies by joining events to your customer records is cross-checking with another processing operation. So this build has two modes and makes you choose. Measurement mode never joins to a customer file, never leaves the product, truncates at ingest and expires the identifier. Identified mode does the account rollup and needs a lawful basis, a consent record per person and a sentence in the interface saying so. Measurement mode is the default and the README says which one you switched on. And the second rule, which is about arithmetic rather than law: a number nobody can reproduce is not a number. Every figure states its window, its session rule and its exclusions, and clicks through to the rows underneath it. ### Stack (fixed, do not substitute) - Collector: A small Node 22 endpoint that validates, truncates and batches, never blocking the page · an analytics endpoint that is slow becomes an analytics endpoint people remove - Event store: ClickHouse, with events append-only and never edited · a product analytics query is a column scan and everything else is a slow surprise later - Configuration: SQLite for accounts, memberships, report definitions and exclusion rules · these are small, relational, and you will edit them by hand at two in the morning - Client: A tiny first-party script that queues locally and flushes on the visibility change · the tail of every mobile session is lost if you flush on unload - Reports: Definitions stored as data, with one shared sessionisation function · two reports that disagree by ten percent cost a day each time ### Data model (create this before Phase 1) - events(at, event_name, anon_id, user_id, account_id, props_json, url_path, referrer_host, ua_family, ip_prefix, ingest_at, schema_version) · ip_prefix is truncated at ingest and the full address is written nowhere - identities(anon_id, user_id, first_seen_at, linked_at) · append only, so a wrong link can be seen rather than only its result - accounts(id, name, domain, source, created_at) · source records how the mapping was made, which is the only way to audit it later - account_members(user_id, account_id, from_at, to_at) · ranged, because a report over last quarter must use the membership as it was then - event_schema(event_name, prop, type, first_seen_at, last_seen_at, sample_count) · discovered rather than declared, so schema drift shows up as a row - exclusions(id, rule_kind, pattern, reason, effective_from) · bots, internal users and load tests, each with a stated reason and a date - reports(id, name, definition_json, created_at) · the definition is the report, and it is versioned rather than edited in place - There is no column anywhere holding a full IP address or a raw user agent string. ### Phase 1 · A collector that does not lose the end of the session Build: A batch endpoint that validates against the discovered schema, truncates the address, stamps ingest time and writes in batches. A client that queues events locally and flushes when the page becomes hidden, with the queue surviving into the next page load. Split batches below the queue limit and treat a refused flush as something to retry rather than something to drop. Trap: Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. Done when: The client flushes on the visibility change to hidden and the request completes after the page is gone, checked in a browser with the network log open. A refused flush leaves the events in the local queue and they arrive on the next page load, tested by forcing a refusal. The full IP address appears nowhere in storage or logs, asserted by a test that scans both. A batch above the configured size is split before sending, tested at the boundary. Do not build yet: No identity, accounts, reports or exclusions yet. ### Phase 2 · Accounts that exist, and joins that are named Build: Anonymous identifiers first, a user identifier on identify, and account membership as an explicit mapping with a source and a validity range. Keep a deny list of public mail providers. Implement the two modes: in measurement mode no code path reads an external customer file, and in identified mode attribution requires a consent record for that person. Trap: Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. Done when: Two users on the same public mail provider are never placed in the same account, tested against the deny list. A membership change is recorded with a range and a report over an earlier window uses the membership as it was then, tested. In measurement mode there is no code path that reads a customer file, asserted by a test over the module graph. In identified mode an event for a person with no consent record is stored without the account attribution rather than with it. Do not build yet: No report definitions, exclusions, retention or export yet. ### Phase 3 · The same question, the same answer, twice Build: Store report definitions as data. Put the sessionisation rule in one function that every report calls. Keep unique counts exact at this volume. Print the window, the session rule and the active exclusions on the report itself, and make every figure click through to the events behind it. Trap: Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. Done when: Running a report over a closed past window twice returns byte-identical results, tested. Every report renders its window, sessionisation parameters and the exclusions applied. Clicking any figure lists the underlying events and the count of that list matches the figure, tested. Unique counts are exact and no approximating function appears in any report query, checked by scanning the generated SQL. Do not build yet: No exclusion management, retention, deletion or export yet. ### Phase 4 · Bots, staff and the traffic you should not have counted Build: Exclusion rules carrying a reason and an effective date, applied at query time rather than at ingest. A view showing how much volume each rule removes over the current window. Rules can be disabled and the affected reports recompute. Keep the excluded rows. Trap: Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. Done when: No exclusion rule deletes or edits a row, asserted by checking that the event store has no delete path. A report over a window before a rule's effective date is unchanged by that rule, tested. Every rule shows the volume it removes over the current window. Disabling a rule and re-running a report changes the figure and the report says which rules were active, tested. Do not build yet: No retention, deletion or export yet. ### Phase 5 · Retention, deletion and saying when a number is an estimate Build: Scheduled jobs enforcing the identifier lifetime and the event retention, writing what they did. A command that deletes every row for a person or an account and produces a receipt. Export of definitions and aggregates. If volume forces sampling, mark every affected figure as sampled wherever it appears, including in exports. Trap: A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page. Done when: The retention job deletes events past the configured age and writes a row recording the count and range, tested. Deleting a person removes their events, identities and memberships and produces a receipt listing the counts, tested. A sampled figure is marked as sampled in the interface and in the export, tested by forcing sampling on. Export of report definitions and aggregates reimports into an empty installation and reproduces the same figures. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay, which turns an analytics store into a recording of people using your product and is a different consent conversation entirely. - A managed ingestion pipeline at scale, which is where the subscription's money goes and is an operations job rather than a build. - Bought bot and device identification lists, which are licensed data you cannot redistribute and would have to keep paying for. - Cross-site or cross-product identity, which is the exact thing the measurement exemption forbids and the reason it exists. ### README must contain - Which mode the installation is running in, measurement or identified, and what changes between them - The conditions the regulator sets for consent-exempt audience measurement, where that text is published, and which of them this build implements - That the client flushes on the visibility change and what happens to events when a flush is refused - How accounts are mapped to users, that the mapping is explicit, and why email domains are not used - The retention periods, how a deletion request is executed, and what the receipt contains ===== AGENTS.md ===== # Agent instructions · June indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A small Node 22 endpoint that validates, truncates and batches, never blocking the page, ClickHouse, with events append-only and never edited, SQLite for accounts, memberships, report definitions and exclusion rules, A tiny first-party script that queues locally and flushes on the visibility change, Definitions stored as data, with one shared sessionisation function. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. ===== BUILD_PLAN.md ===== # Build plan · June A first-party analytics collector whose hard problems are a browser flush that actually fires on mobile, an account rollup that does not invent companies out of email domains, and reports that return the same figure twice. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A collector that does not lose the end of the session Events reach the store from real browsers, including the ones that close without warning. ### Steps 1. Flush on the visibility change, not on unload Unload is the event mobile browsers are allowed to skip. Files: `client/collect.js`, `docs/client.md` 2. Keep a local queue that survives a page load A refused flush should be a delay, not a loss. Files: `client/queue.js`, `tests/queue.test.ts` 3. Truncate the address at ingest, before anything is written The only place a full address cannot leak from is the place it never reached. Files: `src/collect.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] The hidden-visibility flush completes after the page is gone - [ ] A refused flush is retried on the next load - [ ] No full IP address exists in storage or logs - [ ] Oversized batches are split before sending ### Watch out - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. ## Phase 2 · Accounts that exist, and joins that are named A user belongs to a company because somebody said so, not because of the text after the at sign. ### Steps 1. Make membership explicit and ranged People move between companies and last quarter did not change. Files: `src/accounts.ts`, `tests/membership.test.ts` 2. Keep a deny list of public mail providers Otherwise your largest customer is a free mail service. Files: `config/public-domains.txt`, `tests/rollup.test.ts` 3. Make the two modes a build-time boundary Measurement mode should be unable to reach the customer file. Files: `src/modes.ts`, `tests/mode-isolation.test.ts` ### Done when - [ ] Public mail domains never form an account - [ ] Past windows use the membership of that time - [ ] Measurement mode cannot reach a customer file - [ ] Identified mode requires a consent record to attribute ### Watch out - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. ## Phase 3 · The same question, the same answer, twice A report over a closed past window returns identical numbers every time it runs. ### Steps 1. One sessionisation function, called by everything Two definitions of a session is two products. Files: `src/reports/session.ts`, `tests/session.test.ts` 2. Print the parameters on the report A figure without its window is a rumour. Files: `src/reports/render.tsx` 3. Make every figure click through to rows The fastest way to find out a number is wrong. Files: `src/reports/drill.ts`, `tests/drill.test.ts` ### Done when - [ ] A closed window re-runs identically - [ ] Reports display window, session rule and exclusions - [ ] Every figure drills through to matching rows - [ ] No approximating count appears in report queries ### Watch out - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. ## Phase 4 · Bots, staff and the traffic you should not have counted Exclusions are revisable because nothing was deleted to make them. ### Steps 1. Exclude at query time, never at ingest You cannot un-drop an event. Files: `src/exclusions.ts`, `tests/exclusions.test.ts` 2. Give every rule a reason and an effective date A rule with no date rewrites history when it is edited. Files: `src/db/exclusions.sql` 3. Show the volume each rule removes So an over-broad rule is visible rather than believed. Files: `src/reports/exclusions.tsx` ### Done when - [ ] No exclusion deletes a row - [ ] Rules do not affect windows before their date - [ ] Removed volume per rule is visible - [ ] Reports name the rules that were active ### Watch out - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. ## Not in this build - Session replay. - A managed ingestion pipeline at scale. - Licensed bot and device identification lists. - Cross-site or cross-product identity. ## After v1, if you want it - A weekly digest of schema drift, built from the discovered schema table rather than from memory - An annotation layer so a release or an outage sits on the chart next to the dip it caused - A second opinion query for one headline figure, written independently, to catch the day the first one drifts ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. measurement or identified. Decides whether the account rollup exists at all. COLLECT_MODE=measurement # Required. The event store. Append only, never edited by the application. CLICKHOUSE_URL=http://localhost:8123 # Required. Accounts, memberships, report definitions and exclusion rules. CONFIG_DB_PATH=./data/config.db # Required. Tracker lifetime. Check the current guidance for the figure that applies to you. IDENTIFIER_LIFETIME_DAYS=395 # Required. Enforced by a scheduled job that logs what it deleted. EVENT_RETENTION_DAYS=760 # Required. Client batch size, kept under the browser queue limit. Verify the current limit. BATCH_MAX_BYTES=32768 # Required. The single sessionisation parameter, printed on every report that uses it. SESSION_GAP_MINUTES=30
You are building a production product version of June. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # June · product brief ## Problem The visible B2B product analytics loop is buildable, but a credible replacement needs more than the first screen. June earns its keep through ingestion, storage, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A collector that keeps the end of mobile sessions, an account rollup that never invented a company, reports that reproduce, exclusions that can be revised because nothing was deleted, and retention that runs as a job. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - deploy target - ClickHouse or SQLite for low volume - first-party tracking script ## Explicit non-goals for v1 - Session replay. - A managed ingestion pipeline at scale. - Licensed bot and device identification lists. - Cross-site or cross-product identity. - enterprise governance and integrations - bank, payment, tax, and reconciliation integrations - high-volume ingestion and retention - bot and identity resolution - session replay privacy tooling ## Success criteria - The visibility flush is confirmed to deliver after the page is gone - No full IP address is confirmed present in storage or logs - Public mail domains are confirmed never to form an account - A closed past window is confirmed to re-run identically - Retention and deletion are confirmed to run as logged jobs ===== BRIEF.md ===== # Build brief · June The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party product analytics collector for one product that rolls events up to accounts and is explicit about which of its numbers you can rely on to replace June. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Analytics is regulated where your users are, and the exemption that lets a first-party measurement tool skip a consent banner is narrower than the product you are about to build. The French regulator's guidance on using analytics sets out what an exempt tool must do: inform users and let them object, restrict the purpose to audience measurement and A/B testing, not cross-check the data with other processing such as a customer file or statistics about visits to other sites, keep the identifier scoped to a single publisher, truncate the last byte of the IP address, and limit the tracker lifetime to thirteen months. It also states that most large audience measurement offerings fall outside the exemption whatever their configuration. Read the current text yourself, and read the rules where your users actually are rather than only these. Now read that list against what account-level product analytics is. Rolling users up to companies by joining events to your customer records is cross-checking with another processing operation. So this build has two modes and makes you choose. Measurement mode never joins to a customer file, never leaves the product, truncates at ingest and expires the identifier. Identified mode does the account rollup and needs a lawful basis, a consent record per person and a sentence in the interface saying so. Measurement mode is the default and the README says which one you switched on. And the second rule, which is about arithmetic rather than law: a number nobody can reproduce is not a number. Every figure states its window, its session rule and its exclusions, and clicks through to the rows underneath it. ### Stack (fixed, do not substitute) - Collector: A small Node 22 endpoint that validates, truncates and batches, never blocking the page · an analytics endpoint that is slow becomes an analytics endpoint people remove - Event store: ClickHouse, with events append-only and never edited · a product analytics query is a column scan and everything else is a slow surprise later - Configuration: SQLite for accounts, memberships, report definitions and exclusion rules · these are small, relational, and you will edit them by hand at two in the morning - Client: A tiny first-party script that queues locally and flushes on the visibility change · the tail of every mobile session is lost if you flush on unload - Reports: Definitions stored as data, with one shared sessionisation function · two reports that disagree by ten percent cost a day each time ### Data model (create this before Phase 1) - events(at, event_name, anon_id, user_id, account_id, props_json, url_path, referrer_host, ua_family, ip_prefix, ingest_at, schema_version) · ip_prefix is truncated at ingest and the full address is written nowhere - identities(anon_id, user_id, first_seen_at, linked_at) · append only, so a wrong link can be seen rather than only its result - accounts(id, name, domain, source, created_at) · source records how the mapping was made, which is the only way to audit it later - account_members(user_id, account_id, from_at, to_at) · ranged, because a report over last quarter must use the membership as it was then - event_schema(event_name, prop, type, first_seen_at, last_seen_at, sample_count) · discovered rather than declared, so schema drift shows up as a row - exclusions(id, rule_kind, pattern, reason, effective_from) · bots, internal users and load tests, each with a stated reason and a date - reports(id, name, definition_json, created_at) · the definition is the report, and it is versioned rather than edited in place - There is no column anywhere holding a full IP address or a raw user agent string. ### Phase 1 · A collector that does not lose the end of the session Build: A batch endpoint that validates against the discovered schema, truncates the address, stamps ingest time and writes in batches. A client that queues events locally and flushes when the page becomes hidden, with the queue surviving into the next page load. Split batches below the queue limit and treat a refused flush as something to retry rather than something to drop. Trap: Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. Done when: The client flushes on the visibility change to hidden and the request completes after the page is gone, checked in a browser with the network log open. A refused flush leaves the events in the local queue and they arrive on the next page load, tested by forcing a refusal. The full IP address appears nowhere in storage or logs, asserted by a test that scans both. A batch above the configured size is split before sending, tested at the boundary. Do not build yet: No identity, accounts, reports or exclusions yet. ### Phase 2 · Accounts that exist, and joins that are named Build: Anonymous identifiers first, a user identifier on identify, and account membership as an explicit mapping with a source and a validity range. Keep a deny list of public mail providers. Implement the two modes: in measurement mode no code path reads an external customer file, and in identified mode attribution requires a consent record for that person. Trap: Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. Done when: Two users on the same public mail provider are never placed in the same account, tested against the deny list. A membership change is recorded with a range and a report over an earlier window uses the membership as it was then, tested. In measurement mode there is no code path that reads a customer file, asserted by a test over the module graph. In identified mode an event for a person with no consent record is stored without the account attribution rather than with it. Do not build yet: No report definitions, exclusions, retention or export yet. ### Phase 3 · The same question, the same answer, twice Build: Store report definitions as data. Put the sessionisation rule in one function that every report calls. Keep unique counts exact at this volume. Print the window, the session rule and the active exclusions on the report itself, and make every figure click through to the events behind it. Trap: Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. Done when: Running a report over a closed past window twice returns byte-identical results, tested. Every report renders its window, sessionisation parameters and the exclusions applied. Clicking any figure lists the underlying events and the count of that list matches the figure, tested. Unique counts are exact and no approximating function appears in any report query, checked by scanning the generated SQL. Do not build yet: No exclusion management, retention, deletion or export yet. ### Phase 4 · Bots, staff and the traffic you should not have counted Build: Exclusion rules carrying a reason and an effective date, applied at query time rather than at ingest. A view showing how much volume each rule removes over the current window. Rules can be disabled and the affected reports recompute. Keep the excluded rows. Trap: Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. Done when: No exclusion rule deletes or edits a row, asserted by checking that the event store has no delete path. A report over a window before a rule's effective date is unchanged by that rule, tested. Every rule shows the volume it removes over the current window. Disabling a rule and re-running a report changes the figure and the report says which rules were active, tested. Do not build yet: No retention, deletion or export yet. ### Phase 5 · Retention, deletion and saying when a number is an estimate Build: Scheduled jobs enforcing the identifier lifetime and the event retention, writing what they did. A command that deletes every row for a person or an account and produces a receipt. Export of definitions and aggregates. If volume forces sampling, mark every affected figure as sampled wherever it appears, including in exports. Trap: A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page. Done when: The retention job deletes events past the configured age and writes a row recording the count and range, tested. Deleting a person removes their events, identities and memberships and produces a receipt listing the counts, tested. A sampled figure is marked as sampled in the interface and in the export, tested by forcing sampling on. Export of report definitions and aggregates reimports into an empty installation and reproduces the same figures. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay, which turns an analytics store into a recording of people using your product and is a different consent conversation entirely. - A managed ingestion pipeline at scale, which is where the subscription's money goes and is an operations job rather than a build. - Bought bot and device identification lists, which are licensed data you cannot redistribute and would have to keep paying for. - Cross-site or cross-product identity, which is the exact thing the measurement exemption forbids and the reason it exists. ### README must contain - Which mode the installation is running in, measurement or identified, and what changes between them - The conditions the regulator sets for consent-exempt audience measurement, where that text is published, and which of them this build implements - That the client flushes on the visibility change and what happens to events when a flush is refused - How accounts are mapped to users, that the mapping is explicit, and why email domains are not used - The retention periods, how a deletion request is executed, and what the receipt contains ===== ARCHITECTURE.md ===== # Architecture · June ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small Node 22 endpoint that validates, truncates and batches, never blocking the page | an analytics endpoint that is slow becomes an analytics endpoint people remove | | Event store | ClickHouse, with events append-only and never edited | a product analytics query is a column scan and everything else is a slow surprise later | | Configuration | SQLite for accounts, memberships, report definitions and exclusion rules | these are small, relational, and you will edit them by hand at two in the morning | | Client | A tiny first-party script that queues locally and flushes on the visibility change | the tail of every mobile session is lost if you flush on unload | | Reports | Definitions stored as data, with one shared sessionisation function | two reports that disagree by ten percent cost a day each time | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Client | Local queue, batching and the visibility flush. | Any client, provided the flush happens on visibility change. | | Collector | Validation, truncation, schema discovery and batched writes. | Any runtime, provided truncation happens before the first write. | | Identity | Anonymous to user linking and ranged account membership. | Any mapping, provided it is explicit and has a recorded source. | | Reports | Definitions, sessionisation and drill-through. | Any query layer, provided one session function serves all of them. | | Governance | Exclusions, retention, deletion and sampling labels. | Not swappable for documentation, which is what it usually becomes. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `COLLECT_MODE` · required · measurement or identified. Decides whether the account rollup exists at all. - `CLICKHOUSE_URL` · required · The event store. Append only, never edited by the application. - `CONFIG_DB_PATH` · required · Accounts, memberships, report definitions and exclusion rules. - `IDENTIFIER_LIFETIME_DAYS` · required · Tracker lifetime. Check the current guidance for the figure that applies to you. - `EVENT_RETENTION_DAYS` · required · Enforced by a scheduled job that logs what it deleted. - `BATCH_MAX_BYTES` · required · Client batch size, kept under the browser queue limit. Verify the current limit. - `SESSION_GAP_MINUTES` · required · The single sessionisation parameter, printed on every report that uses it. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · June product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A small Node 22 endpoint that validates, truncates and batches, never blocking the page, ClickHouse, with events append-only and never edited, SQLite for accounts, memberships, report definitions and exclusion rules, A tiny first-party script that queues locally and flushes on the visibility change, Definitions stored as data, with one shared sessionisation function. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. - A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page. ===== MILESTONES.md ===== # Delivery milestones · June Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A collector that does not lose the end of the session Events reach the store from real browsers, including the ones that close without warning. ### Steps 1. Flush on the visibility change, not on unload Unload is the event mobile browsers are allowed to skip. Files: `client/collect.js`, `docs/client.md` 2. Keep a local queue that survives a page load A refused flush should be a delay, not a loss. Files: `client/queue.js`, `tests/queue.test.ts` 3. Truncate the address at ingest, before anything is written The only place a full address cannot leak from is the place it never reached. Files: `src/collect.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] The hidden-visibility flush completes after the page is gone - [ ] A refused flush is retried on the next load - [ ] No full IP address exists in storage or logs - [ ] Oversized batches are split before sending ### Watch out - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. ## M2 · Accounts that exist, and joins that are named A user belongs to a company because somebody said so, not because of the text after the at sign. ### Steps 1. Make membership explicit and ranged People move between companies and last quarter did not change. Files: `src/accounts.ts`, `tests/membership.test.ts` 2. Keep a deny list of public mail providers Otherwise your largest customer is a free mail service. Files: `config/public-domains.txt`, `tests/rollup.test.ts` 3. Make the two modes a build-time boundary Measurement mode should be unable to reach the customer file. Files: `src/modes.ts`, `tests/mode-isolation.test.ts` ### Done when - [ ] Public mail domains never form an account - [ ] Past windows use the membership of that time - [ ] Measurement mode cannot reach a customer file - [ ] Identified mode requires a consent record to attribute ### Watch out - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. ## M3 · The same question, the same answer, twice A report over a closed past window returns identical numbers every time it runs. ### Steps 1. One sessionisation function, called by everything Two definitions of a session is two products. Files: `src/reports/session.ts`, `tests/session.test.ts` 2. Print the parameters on the report A figure without its window is a rumour. Files: `src/reports/render.tsx` 3. Make every figure click through to rows The fastest way to find out a number is wrong. Files: `src/reports/drill.ts`, `tests/drill.test.ts` ### Done when - [ ] A closed window re-runs identically - [ ] Reports display window, session rule and exclusions - [ ] Every figure drills through to matching rows - [ ] No approximating count appears in report queries ### Watch out - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. ## M4 · Bots, staff and the traffic you should not have counted Exclusions are revisable because nothing was deleted to make them. ### Steps 1. Exclude at query time, never at ingest You cannot un-drop an event. Files: `src/exclusions.ts`, `tests/exclusions.test.ts` 2. Give every rule a reason and an effective date A rule with no date rewrites history when it is edited. Files: `src/db/exclusions.sql` 3. Show the volume each rule removes So an over-broad rule is visible rather than believed. Files: `src/reports/exclusions.tsx` ### Done when - [ ] No exclusion deletes a row - [ ] Rules do not affect windows before their date - [ ] Removed volume per rule is visible - [ ] Reports name the rules that were active ### Watch out - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. ## M5 · Retention, deletion and saying when a number is an estimate (production only) The policies run as jobs, a deletion request is a command, and a sampled figure says it is sampled. ### Steps 1. Make retention a job with a log A policy nobody can prove ran did not run. Files: `src/jobs/retention.ts`, `tests/retention.test.ts` 2. Make deletion a command with a receipt Counts per table, so the answer is a document. Files: `src/jobs/erase.ts`, `tests/erase.test.ts` 3. Mark sampled figures at the figure Not in a footnote nobody reads. Files: `src/reports/render.tsx` ### Done when - [ ] Retention runs as a job and logs what it deleted - [ ] Deletion covers events, identities and memberships - [ ] A sampled figure is labelled everywhere it appears - [ ] Definitions and aggregates export and reimport ### Watch out - A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page. ===== OPERATIONS.md ===== # Operations · June ## Backup The configuration database first: report definitions, account mappings and exclusion rules are the parts you authored and cannot regenerate. The event store second, because it is large, append only and a gap in it is visible rather than silent. ## Restore Restore the configuration, then the events, then re-run one report over a closed past window and compare it against a figure you recorded before. If the number moved, the exclusions or the memberships did not come back the way they went in. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Ingest rate against its usual shape, refused flush rate from the client, the proportion of events with no account mapping, excluded volume per rule, and whether any report is running in sampled mode. A rising unmapped proportion means new customers are arriving faster than the mapping is being maintained. ## Incident checklist The incident is a figure somebody acted on. Because nothing is deleted and exclusions carry dates, any report can be recomputed as it stood on the day, which is the difference between explaining a discrepancy and arguing about it. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The visibility flush is confirmed to deliver after the page is gone - [ ] No full IP address is confirmed present in storage or logs - [ ] Public mail domains are confirmed never to form an account - [ ] A closed past window is confirmed to re-run identically - [ ] Retention and deletion are confirmed to run as logged jobs ## Launch constraint Do not market omitted June capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. measurement or identified. Decides whether the account rollup exists at all. COLLECT_MODE=measurement # Required. The event store. Append only, never edited by the application. CLICKHOUSE_URL=http://localhost:8123 # Required. Accounts, memberships, report definitions and exclusion rules. CONFIG_DB_PATH=./data/config.db # Required. Tracker lifetime. Check the current guidance for the figure that applies to you. IDENTIFIER_LIFETIME_DAYS=395 # Required. Enforced by a scheduled job that logs what it deleted. EVENT_RETENTION_DAYS=760 # Required. Client batch size, kept under the browser queue limit. Verify the current limit. BATCH_MAX_BYTES=32768 # Required. The single sessionisation parameter, printed on every report that uses it. SESSION_GAP_MINUTES=30
README.md
# June · indie build A first-party analytics collector whose hard problems are a browser flush that actually fires on mobile, an account rollup that does not invent companies out of email domains, and reports that return the same figure twice. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small Node 22 endpoint that validates, truncates and batches, never blocking the page | an analytics endpoint that is slow becomes an analytics endpoint people remove | | Event store | ClickHouse, with events append-only and never edited | a product analytics query is a column scan and everything else is a slow surprise later | | Configuration | SQLite for accounts, memberships, report definitions and exclusion rules | these are small, relational, and you will edit them by hand at two in the morning | | Client | A tiny first-party script that queues locally and flushes on the visibility change | the tail of every mobile session is lost if you flush on unload | | Reports | Definitions stored as data, with one shared sessionisation function | two reports that disagree by ten percent cost a day each time | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the collector, the report server and the jobs. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **A ClickHouse instance** · free locally; a small managed instance is a few units of currency a month - Why: Product analytics queries scan columns over millions of rows, and a row store makes every report slow enough that you stop opening them. - Get it: Run it in a container locally, or use a small managed instance. - Verify: A ten million row table answers a group-by in under a second - [ ] **Measurement mode or identified mode** · free - Why: It decides whether you need consent, whether the account rollup exists at all, and what the interface has to say, and deciding it after launch means deciding it from whatever you already collected. - Get it: Read the current regulatory guidance for where your users are, pick a mode, and write down the lawful basis if it is the second. - Verify: The mode is set in configuration, stated in the README and shown in the interface - [ ] **A list of public mail providers** · free - Why: Without it the account rollup invents a very large fictional customer and you will believe it. - Get it: Any maintained public list, checked into the repository so the behaviour is reproducible. - Verify: The rollup test refuses to group two users from the same public provider - [ ] **A real product with real traffic** · free - Why: Every hard problem here appears only with real browsers: closed tabs, a proxy stripping headers, a customer's monitoring hitting you every minute. - Get it: Point the client at something you already run, even at low volume. - Verify: Events arrive from at least two browsers and one mobile device ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Session replay. - A managed ingestion pipeline at scale. - Licensed bot and device identification lists. - Cross-site or cross-product identity. - enterprise governance and integrations - bank, payment, tax, and reconciliation integrations - high-volume ingestion and retention - bot and identity resolution - session replay privacy tooling If one of those is essential to you, that is the reason to keep paying for June, and the README should say so rather than pretend.
BRIEF.md
# Build brief · June The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party product analytics collector for one product that rolls events up to accounts and is explicit about which of its numbers you can rely on to replace June. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Analytics is regulated where your users are, and the exemption that lets a first-party measurement tool skip a consent banner is narrower than the product you are about to build. The French regulator's guidance on using analytics sets out what an exempt tool must do: inform users and let them object, restrict the purpose to audience measurement and A/B testing, not cross-check the data with other processing such as a customer file or statistics about visits to other sites, keep the identifier scoped to a single publisher, truncate the last byte of the IP address, and limit the tracker lifetime to thirteen months. It also states that most large audience measurement offerings fall outside the exemption whatever their configuration. Read the current text yourself, and read the rules where your users actually are rather than only these. Now read that list against what account-level product analytics is. Rolling users up to companies by joining events to your customer records is cross-checking with another processing operation. So this build has two modes and makes you choose. Measurement mode never joins to a customer file, never leaves the product, truncates at ingest and expires the identifier. Identified mode does the account rollup and needs a lawful basis, a consent record per person and a sentence in the interface saying so. Measurement mode is the default and the README says which one you switched on. And the second rule, which is about arithmetic rather than law: a number nobody can reproduce is not a number. Every figure states its window, its session rule and its exclusions, and clicks through to the rows underneath it. ### Stack (fixed, do not substitute) - Collector: A small Node 22 endpoint that validates, truncates and batches, never blocking the page · an analytics endpoint that is slow becomes an analytics endpoint people remove - Event store: ClickHouse, with events append-only and never edited · a product analytics query is a column scan and everything else is a slow surprise later - Configuration: SQLite for accounts, memberships, report definitions and exclusion rules · these are small, relational, and you will edit them by hand at two in the morning - Client: A tiny first-party script that queues locally and flushes on the visibility change · the tail of every mobile session is lost if you flush on unload - Reports: Definitions stored as data, with one shared sessionisation function · two reports that disagree by ten percent cost a day each time ### Data model (create this before Phase 1) - events(at, event_name, anon_id, user_id, account_id, props_json, url_path, referrer_host, ua_family, ip_prefix, ingest_at, schema_version) · ip_prefix is truncated at ingest and the full address is written nowhere - identities(anon_id, user_id, first_seen_at, linked_at) · append only, so a wrong link can be seen rather than only its result - accounts(id, name, domain, source, created_at) · source records how the mapping was made, which is the only way to audit it later - account_members(user_id, account_id, from_at, to_at) · ranged, because a report over last quarter must use the membership as it was then - event_schema(event_name, prop, type, first_seen_at, last_seen_at, sample_count) · discovered rather than declared, so schema drift shows up as a row - exclusions(id, rule_kind, pattern, reason, effective_from) · bots, internal users and load tests, each with a stated reason and a date - reports(id, name, definition_json, created_at) · the definition is the report, and it is versioned rather than edited in place - There is no column anywhere holding a full IP address or a raw user agent string. ### Phase 1 · A collector that does not lose the end of the session Build: A batch endpoint that validates against the discovered schema, truncates the address, stamps ingest time and writes in batches. A client that queues events locally and flushes when the page becomes hidden, with the queue surviving into the next page load. Split batches below the queue limit and treat a refused flush as something to retry rather than something to drop. Trap: Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. Done when: The client flushes on the visibility change to hidden and the request completes after the page is gone, checked in a browser with the network log open. A refused flush leaves the events in the local queue and they arrive on the next page load, tested by forcing a refusal. The full IP address appears nowhere in storage or logs, asserted by a test that scans both. A batch above the configured size is split before sending, tested at the boundary. Do not build yet: No identity, accounts, reports or exclusions yet. ### Phase 2 · Accounts that exist, and joins that are named Build: Anonymous identifiers first, a user identifier on identify, and account membership as an explicit mapping with a source and a validity range. Keep a deny list of public mail providers. Implement the two modes: in measurement mode no code path reads an external customer file, and in identified mode attribution requires a consent record for that person. Trap: Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. Done when: Two users on the same public mail provider are never placed in the same account, tested against the deny list. A membership change is recorded with a range and a report over an earlier window uses the membership as it was then, tested. In measurement mode there is no code path that reads a customer file, asserted by a test over the module graph. In identified mode an event for a person with no consent record is stored without the account attribution rather than with it. Do not build yet: No report definitions, exclusions, retention or export yet. ### Phase 3 · The same question, the same answer, twice Build: Store report definitions as data. Put the sessionisation rule in one function that every report calls. Keep unique counts exact at this volume. Print the window, the session rule and the active exclusions on the report itself, and make every figure click through to the events behind it. Trap: Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. Done when: Running a report over a closed past window twice returns byte-identical results, tested. Every report renders its window, sessionisation parameters and the exclusions applied. Clicking any figure lists the underlying events and the count of that list matches the figure, tested. Unique counts are exact and no approximating function appears in any report query, checked by scanning the generated SQL. Do not build yet: No exclusion management, retention, deletion or export yet. ### Phase 4 · Bots, staff and the traffic you should not have counted Build: Exclusion rules carrying a reason and an effective date, applied at query time rather than at ingest. A view showing how much volume each rule removes over the current window. Rules can be disabled and the affected reports recompute. Keep the excluded rows. Trap: Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. Done when: No exclusion rule deletes or edits a row, asserted by checking that the event store has no delete path. A report over a window before a rule's effective date is unchanged by that rule, tested. Every rule shows the volume it removes over the current window. Disabling a rule and re-running a report changes the figure and the report says which rules were active, tested. Do not build yet: No retention, deletion or export yet. ### Phase 5 · Retention, deletion and saying when a number is an estimate Build: Scheduled jobs enforcing the identifier lifetime and the event retention, writing what they did. A command that deletes every row for a person or an account and produces a receipt. Export of definitions and aggregates. If volume forces sampling, mark every affected figure as sampled wherever it appears, including in exports. Trap: A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page. Done when: The retention job deletes events past the configured age and writes a row recording the count and range, tested. Deleting a person removes their events, identities and memberships and produces a receipt listing the counts, tested. A sampled figure is marked as sampled in the interface and in the export, tested by forcing sampling on. Export of report definitions and aggregates reimports into an empty installation and reproduces the same figures. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay, which turns an analytics store into a recording of people using your product and is a different consent conversation entirely. - A managed ingestion pipeline at scale, which is where the subscription's money goes and is an operations job rather than a build. - Bought bot and device identification lists, which are licensed data you cannot redistribute and would have to keep paying for. - Cross-site or cross-product identity, which is the exact thing the measurement exemption forbids and the reason it exists. ### README must contain - Which mode the installation is running in, measurement or identified, and what changes between them - The conditions the regulator sets for consent-exempt audience measurement, where that text is published, and which of them this build implements - That the client flushes on the visibility change and what happens to events when a flush is refused - How accounts are mapped to users, that the mapping is explicit, and why email domains are not used - The retention periods, how a deletion request is executed, and what the receipt contains
AGENTS.md
# Agent instructions · June indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A small Node 22 endpoint that validates, truncates and batches, never blocking the page, ClickHouse, with events append-only and never edited, SQLite for accounts, memberships, report definitions and exclusion rules, A tiny first-party script that queues locally and flushes on the visibility change, Definitions stored as data, with one shared sessionisation function. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts.
BUILD_PLAN.md
# Build plan · June A first-party analytics collector whose hard problems are a browser flush that actually fires on mobile, an account rollup that does not invent companies out of email domains, and reports that return the same figure twice. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A collector that does not lose the end of the session Events reach the store from real browsers, including the ones that close without warning. ### Steps 1. Flush on the visibility change, not on unload Unload is the event mobile browsers are allowed to skip. Files: `client/collect.js`, `docs/client.md` 2. Keep a local queue that survives a page load A refused flush should be a delay, not a loss. Files: `client/queue.js`, `tests/queue.test.ts` 3. Truncate the address at ingest, before anything is written The only place a full address cannot leak from is the place it never reached. Files: `src/collect.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] The hidden-visibility flush completes after the page is gone - [ ] A refused flush is retried on the next load - [ ] No full IP address exists in storage or logs - [ ] Oversized batches are split before sending ### Watch out - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. ## Phase 2 · Accounts that exist, and joins that are named A user belongs to a company because somebody said so, not because of the text after the at sign. ### Steps 1. Make membership explicit and ranged People move between companies and last quarter did not change. Files: `src/accounts.ts`, `tests/membership.test.ts` 2. Keep a deny list of public mail providers Otherwise your largest customer is a free mail service. Files: `config/public-domains.txt`, `tests/rollup.test.ts` 3. Make the two modes a build-time boundary Measurement mode should be unable to reach the customer file. Files: `src/modes.ts`, `tests/mode-isolation.test.ts` ### Done when - [ ] Public mail domains never form an account - [ ] Past windows use the membership of that time - [ ] Measurement mode cannot reach a customer file - [ ] Identified mode requires a consent record to attribute ### Watch out - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. ## Phase 3 · The same question, the same answer, twice A report over a closed past window returns identical numbers every time it runs. ### Steps 1. One sessionisation function, called by everything Two definitions of a session is two products. Files: `src/reports/session.ts`, `tests/session.test.ts` 2. Print the parameters on the report A figure without its window is a rumour. Files: `src/reports/render.tsx` 3. Make every figure click through to rows The fastest way to find out a number is wrong. Files: `src/reports/drill.ts`, `tests/drill.test.ts` ### Done when - [ ] A closed window re-runs identically - [ ] Reports display window, session rule and exclusions - [ ] Every figure drills through to matching rows - [ ] No approximating count appears in report queries ### Watch out - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. ## Phase 4 · Bots, staff and the traffic you should not have counted Exclusions are revisable because nothing was deleted to make them. ### Steps 1. Exclude at query time, never at ingest You cannot un-drop an event. Files: `src/exclusions.ts`, `tests/exclusions.test.ts` 2. Give every rule a reason and an effective date A rule with no date rewrites history when it is edited. Files: `src/db/exclusions.sql` 3. Show the volume each rule removes So an over-broad rule is visible rather than believed. Files: `src/reports/exclusions.tsx` ### Done when - [ ] No exclusion deletes a row - [ ] Rules do not affect windows before their date - [ ] Removed volume per rule is visible - [ ] Reports name the rules that were active ### Watch out - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. ## Not in this build - Session replay. - A managed ingestion pipeline at scale. - Licensed bot and device identification lists. - Cross-site or cross-product identity. ## After v1, if you want it - A weekly digest of schema drift, built from the discovered schema table rather than from memory - An annotation layer so a release or an outage sits on the chart next to the dip it caused - A second opinion query for one headline figure, written independently, to catch the day the first one drifts
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. measurement or identified. Decides whether the account rollup exists at all. COLLECT_MODE=measurement # Required. The event store. Append only, never edited by the application. CLICKHOUSE_URL=http://localhost:8123 # Required. Accounts, memberships, report definitions and exclusion rules. CONFIG_DB_PATH=./data/config.db # Required. Tracker lifetime. Check the current guidance for the figure that applies to you. IDENTIFIER_LIFETIME_DAYS=395 # Required. Enforced by a scheduled job that logs what it deleted. EVENT_RETENTION_DAYS=760 # Required. Client batch size, kept under the browser queue limit. Verify the current limit. BATCH_MAX_BYTES=32768 # Required. The single sessionisation parameter, printed on every report that uses it. SESSION_GAP_MINUTES=30
PRODUCT.md
# June · product brief ## Problem The visible B2B product analytics loop is buildable, but a credible replacement needs more than the first screen. June earns its keep through ingestion, storage, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A collector that keeps the end of mobile sessions, an account rollup that never invented a company, reports that reproduce, exclusions that can be revised because nothing was deleted, and retention that runs as a job. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - deploy target - ClickHouse or SQLite for low volume - first-party tracking script ## Explicit non-goals for v1 - Session replay. - A managed ingestion pipeline at scale. - Licensed bot and device identification lists. - Cross-site or cross-product identity. - enterprise governance and integrations - bank, payment, tax, and reconciliation integrations - high-volume ingestion and retention - bot and identity resolution - session replay privacy tooling ## Success criteria - The visibility flush is confirmed to deliver after the page is gone - No full IP address is confirmed present in storage or logs - Public mail domains are confirmed never to form an account - A closed past window is confirmed to re-run identically - Retention and deletion are confirmed to run as logged jobs
BRIEF.md
# Build brief · June The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party product analytics collector for one product that rolls events up to accounts and is explicit about which of its numbers you can rely on to replace June. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Analytics is regulated where your users are, and the exemption that lets a first-party measurement tool skip a consent banner is narrower than the product you are about to build. The French regulator's guidance on using analytics sets out what an exempt tool must do: inform users and let them object, restrict the purpose to audience measurement and A/B testing, not cross-check the data with other processing such as a customer file or statistics about visits to other sites, keep the identifier scoped to a single publisher, truncate the last byte of the IP address, and limit the tracker lifetime to thirteen months. It also states that most large audience measurement offerings fall outside the exemption whatever their configuration. Read the current text yourself, and read the rules where your users actually are rather than only these. Now read that list against what account-level product analytics is. Rolling users up to companies by joining events to your customer records is cross-checking with another processing operation. So this build has two modes and makes you choose. Measurement mode never joins to a customer file, never leaves the product, truncates at ingest and expires the identifier. Identified mode does the account rollup and needs a lawful basis, a consent record per person and a sentence in the interface saying so. Measurement mode is the default and the README says which one you switched on. And the second rule, which is about arithmetic rather than law: a number nobody can reproduce is not a number. Every figure states its window, its session rule and its exclusions, and clicks through to the rows underneath it. ### Stack (fixed, do not substitute) - Collector: A small Node 22 endpoint that validates, truncates and batches, never blocking the page · an analytics endpoint that is slow becomes an analytics endpoint people remove - Event store: ClickHouse, with events append-only and never edited · a product analytics query is a column scan and everything else is a slow surprise later - Configuration: SQLite for accounts, memberships, report definitions and exclusion rules · these are small, relational, and you will edit them by hand at two in the morning - Client: A tiny first-party script that queues locally and flushes on the visibility change · the tail of every mobile session is lost if you flush on unload - Reports: Definitions stored as data, with one shared sessionisation function · two reports that disagree by ten percent cost a day each time ### Data model (create this before Phase 1) - events(at, event_name, anon_id, user_id, account_id, props_json, url_path, referrer_host, ua_family, ip_prefix, ingest_at, schema_version) · ip_prefix is truncated at ingest and the full address is written nowhere - identities(anon_id, user_id, first_seen_at, linked_at) · append only, so a wrong link can be seen rather than only its result - accounts(id, name, domain, source, created_at) · source records how the mapping was made, which is the only way to audit it later - account_members(user_id, account_id, from_at, to_at) · ranged, because a report over last quarter must use the membership as it was then - event_schema(event_name, prop, type, first_seen_at, last_seen_at, sample_count) · discovered rather than declared, so schema drift shows up as a row - exclusions(id, rule_kind, pattern, reason, effective_from) · bots, internal users and load tests, each with a stated reason and a date - reports(id, name, definition_json, created_at) · the definition is the report, and it is versioned rather than edited in place - There is no column anywhere holding a full IP address or a raw user agent string. ### Phase 1 · A collector that does not lose the end of the session Build: A batch endpoint that validates against the discovered schema, truncates the address, stamps ingest time and writes in batches. A client that queues events locally and flushes when the page becomes hidden, with the queue surviving into the next page load. Split batches below the queue limit and treat a refused flush as something to retry rather than something to drop. Trap: Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. Done when: The client flushes on the visibility change to hidden and the request completes after the page is gone, checked in a browser with the network log open. A refused flush leaves the events in the local queue and they arrive on the next page load, tested by forcing a refusal. The full IP address appears nowhere in storage or logs, asserted by a test that scans both. A batch above the configured size is split before sending, tested at the boundary. Do not build yet: No identity, accounts, reports or exclusions yet. ### Phase 2 · Accounts that exist, and joins that are named Build: Anonymous identifiers first, a user identifier on identify, and account membership as an explicit mapping with a source and a validity range. Keep a deny list of public mail providers. Implement the two modes: in measurement mode no code path reads an external customer file, and in identified mode attribution requires a consent record for that person. Trap: Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. Done when: Two users on the same public mail provider are never placed in the same account, tested against the deny list. A membership change is recorded with a range and a report over an earlier window uses the membership as it was then, tested. In measurement mode there is no code path that reads a customer file, asserted by a test over the module graph. In identified mode an event for a person with no consent record is stored without the account attribution rather than with it. Do not build yet: No report definitions, exclusions, retention or export yet. ### Phase 3 · The same question, the same answer, twice Build: Store report definitions as data. Put the sessionisation rule in one function that every report calls. Keep unique counts exact at this volume. Print the window, the session rule and the active exclusions on the report itself, and make every figure click through to the events behind it. Trap: Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. Done when: Running a report over a closed past window twice returns byte-identical results, tested. Every report renders its window, sessionisation parameters and the exclusions applied. Clicking any figure lists the underlying events and the count of that list matches the figure, tested. Unique counts are exact and no approximating function appears in any report query, checked by scanning the generated SQL. Do not build yet: No exclusion management, retention, deletion or export yet. ### Phase 4 · Bots, staff and the traffic you should not have counted Build: Exclusion rules carrying a reason and an effective date, applied at query time rather than at ingest. A view showing how much volume each rule removes over the current window. Rules can be disabled and the affected reports recompute. Keep the excluded rows. Trap: Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. Done when: No exclusion rule deletes or edits a row, asserted by checking that the event store has no delete path. A report over a window before a rule's effective date is unchanged by that rule, tested. Every rule shows the volume it removes over the current window. Disabling a rule and re-running a report changes the figure and the report says which rules were active, tested. Do not build yet: No retention, deletion or export yet. ### Phase 5 · Retention, deletion and saying when a number is an estimate Build: Scheduled jobs enforcing the identifier lifetime and the event retention, writing what they did. A command that deletes every row for a person or an account and produces a receipt. Export of definitions and aggregates. If volume forces sampling, mark every affected figure as sampled wherever it appears, including in exports. Trap: A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page. Done when: The retention job deletes events past the configured age and writes a row recording the count and range, tested. Deleting a person removes their events, identities and memberships and produces a receipt listing the counts, tested. A sampled figure is marked as sampled in the interface and in the export, tested by forcing sampling on. Export of report definitions and aggregates reimports into an empty installation and reproduces the same figures. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay, which turns an analytics store into a recording of people using your product and is a different consent conversation entirely. - A managed ingestion pipeline at scale, which is where the subscription's money goes and is an operations job rather than a build. - Bought bot and device identification lists, which are licensed data you cannot redistribute and would have to keep paying for. - Cross-site or cross-product identity, which is the exact thing the measurement exemption forbids and the reason it exists. ### README must contain - Which mode the installation is running in, measurement or identified, and what changes between them - The conditions the regulator sets for consent-exempt audience measurement, where that text is published, and which of them this build implements - That the client flushes on the visibility change and what happens to events when a flush is refused - How accounts are mapped to users, that the mapping is explicit, and why email domains are not used - The retention periods, how a deletion request is executed, and what the receipt contains
ARCHITECTURE.md
# Architecture · June ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small Node 22 endpoint that validates, truncates and batches, never blocking the page | an analytics endpoint that is slow becomes an analytics endpoint people remove | | Event store | ClickHouse, with events append-only and never edited | a product analytics query is a column scan and everything else is a slow surprise later | | Configuration | SQLite for accounts, memberships, report definitions and exclusion rules | these are small, relational, and you will edit them by hand at two in the morning | | Client | A tiny first-party script that queues locally and flushes on the visibility change | the tail of every mobile session is lost if you flush on unload | | Reports | Definitions stored as data, with one shared sessionisation function | two reports that disagree by ten percent cost a day each time | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Client | Local queue, batching and the visibility flush. | Any client, provided the flush happens on visibility change. | | Collector | Validation, truncation, schema discovery and batched writes. | Any runtime, provided truncation happens before the first write. | | Identity | Anonymous to user linking and ranged account membership. | Any mapping, provided it is explicit and has a recorded source. | | Reports | Definitions, sessionisation and drill-through. | Any query layer, provided one session function serves all of them. | | Governance | Exclusions, retention, deletion and sampling labels. | Not swappable for documentation, which is what it usually becomes. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `COLLECT_MODE` · required · measurement or identified. Decides whether the account rollup exists at all. - `CLICKHOUSE_URL` · required · The event store. Append only, never edited by the application. - `CONFIG_DB_PATH` · required · Accounts, memberships, report definitions and exclusion rules. - `IDENTIFIER_LIFETIME_DAYS` · required · Tracker lifetime. Check the current guidance for the figure that applies to you. - `EVENT_RETENTION_DAYS` · required · Enforced by a scheduled job that logs what it deleted. - `BATCH_MAX_BYTES` · required · Client batch size, kept under the browser queue limit. Verify the current limit. - `SESSION_GAP_MINUTES` · required · The single sessionisation parameter, printed on every report that uses it. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · June product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A small Node 22 endpoint that validates, truncates and batches, never blocking the page, ClickHouse, with events append-only and never edited, SQLite for accounts, memberships, report definitions and exclusion rules, A tiny first-party script that queues locally and flushes on the visibility change, Definitions stored as data, with one shared sessionisation function. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. - A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page.
MILESTONES.md
# Delivery milestones · June Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A collector that does not lose the end of the session Events reach the store from real browsers, including the ones that close without warning. ### Steps 1. Flush on the visibility change, not on unload Unload is the event mobile browsers are allowed to skip. Files: `client/collect.js`, `docs/client.md` 2. Keep a local queue that survives a page load A refused flush should be a delay, not a loss. Files: `client/queue.js`, `tests/queue.test.ts` 3. Truncate the address at ingest, before anything is written The only place a full address cannot leak from is the place it never reached. Files: `src/collect.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] The hidden-visibility flush completes after the page is gone - [ ] A refused flush is retried on the next load - [ ] No full IP address exists in storage or logs - [ ] Oversized batches are split before sending ### Watch out - Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month. ## M2 · Accounts that exist, and joins that are named A user belongs to a company because somebody said so, not because of the text after the at sign. ### Steps 1. Make membership explicit and ranged People move between companies and last quarter did not change. Files: `src/accounts.ts`, `tests/membership.test.ts` 2. Keep a deny list of public mail providers Otherwise your largest customer is a free mail service. Files: `config/public-domains.txt`, `tests/rollup.test.ts` 3. Make the two modes a build-time boundary Measurement mode should be unable to reach the customer file. Files: `src/modes.ts`, `tests/mode-isolation.test.ts` ### Done when - [ ] Public mail domains never form an account - [ ] Past windows use the membership of that time - [ ] Measurement mode cannot reach a customer file - [ ] Identified mode requires a consent record to attribute ### Watch out - Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one. ## M3 · The same question, the same answer, twice A report over a closed past window returns identical numbers every time it runs. ### Steps 1. One sessionisation function, called by everything Two definitions of a session is two products. Files: `src/reports/session.ts`, `tests/session.test.ts` 2. Print the parameters on the report A figure without its window is a rumour. Files: `src/reports/render.tsx` 3. Make every figure click through to rows The fastest way to find out a number is wrong. Files: `src/reports/drill.ts`, `tests/drill.test.ts` ### Done when - [ ] A closed window re-runs identically - [ ] Reports display window, session rule and exclusions - [ ] Every figure drills through to matching rows - [ ] No approximating count appears in report queries ### Watch out - Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report. ## M4 · Bots, staff and the traffic you should not have counted Exclusions are revisable because nothing was deleted to make them. ### Steps 1. Exclude at query time, never at ingest You cannot un-drop an event. Files: `src/exclusions.ts`, `tests/exclusions.test.ts` 2. Give every rule a reason and an effective date A rule with no date rewrites history when it is edited. Files: `src/db/exclusions.sql` 3. Show the volume each rule removes So an over-broad rule is visible rather than believed. Files: `src/reports/exclusions.tsx` ### Done when - [ ] No exclusion deletes a row - [ ] Rules do not affect windows before their date - [ ] Removed volume per rule is visible - [ ] Reports name the rules that were active ### Watch out - Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts. ## M5 · Retention, deletion and saying when a number is an estimate (production only) The policies run as jobs, a deletion request is a command, and a sampled figure says it is sampled. ### Steps 1. Make retention a job with a log A policy nobody can prove ran did not run. Files: `src/jobs/retention.ts`, `tests/retention.test.ts` 2. Make deletion a command with a receipt Counts per table, so the answer is a document. Files: `src/jobs/erase.ts`, `tests/erase.test.ts` 3. Mark sampled figures at the figure Not in a footnote nobody reads. Files: `src/reports/render.tsx` ### Done when - [ ] Retention runs as a job and logs what it deleted - [ ] Deletion covers events, identities and memberships - [ ] A sampled figure is labelled everywhere it appears - [ ] Definitions and aggregates export and reimport ### Watch out - A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page.
OPERATIONS.md
# Operations · June ## Backup The configuration database first: report definitions, account mappings and exclusion rules are the parts you authored and cannot regenerate. The event store second, because it is large, append only and a gap in it is visible rather than silent. ## Restore Restore the configuration, then the events, then re-run one report over a closed past window and compare it against a figure you recorded before. If the number moved, the exclusions or the memberships did not come back the way they went in. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Ingest rate against its usual shape, refused flush rate from the client, the proportion of events with no account mapping, excluded volume per rule, and whether any report is running in sampled mode. A rising unmapped proportion means new customers are arriving faster than the mapping is being maintained. ## Incident checklist The incident is a figure somebody acted on. Because nothing is deleted and exclusions carry dates, any report can be recomputed as it stood on the day, which is the difference between explaining a discrepancy and arguing about it. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The visibility flush is confirmed to deliver after the page is gone - [ ] No full IP address is confirmed present in storage or logs - [ ] Public mail domains are confirmed never to form an account - [ ] A closed past window is confirmed to re-run identically - [ ] Retention and deletion are confirmed to run as logged jobs ## Launch constraint Do not market omitted June capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. measurement or identified. Decides whether the account rollup exists at all. COLLECT_MODE=measurement # Required. The event store. Append only, never edited by the application. CLICKHOUSE_URL=http://localhost:8123 # Required. Accounts, memberships, report definitions and exclusion rules. CONFIG_DB_PATH=./data/config.db # Required. Tracker lifetime. Check the current guidance for the figure that applies to you. IDENTIFIER_LIFETIME_DAYS=395 # Required. Enforced by a scheduled job that logs what it deleted. EVENT_RETENTION_DAYS=760 # Required. Client batch size, kept under the browser queue limit. Verify the current limit. BATCH_MAX_BYTES=32768 # Required. The single sessionisation parameter, printed on every report that uses it. SESSION_GAP_MINUTES=30
$ choose a build depth, inspect the files, then open the complete pack in your agent
June: Customers pay for trusted numbers, retention, privacy controls, and a pipeline that remains accurate while traffic and schemas change.
xenterprise governance and integrations
xbank, payment, tax, and reconciliation integrations
xhigh-volume ingestion and retention
xbot and identity resolution
xsession replay privacy tooling
share on X ↗"I just replaced June with one prompt"
Don't feel like building it? These folks already made it free.
PostHogGroup analytics turns companies into first-class entities; the dashboards require more assembly than June.open source↗no votes, no pay-to-list · just what's real
June pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| all-in-one | custom | — | No numeric event, seat, report, or retention limits are published; positioned for companies at or above $1M ARR |
free tierno free tier
billingcustom contract; monthly and annual prices are not published; free trial available
hidden costsImplementation setup fees vary with complexity; June explicitly says it has no startup plan and is intended for companies with at least $1M ARR.
verified 2026-08-12 · source ↗
Vibecode June
Kinda. The core of June is buildable in a weekend with the prompt on this page, but there are real gaps: enterprise governance and integrations, bank, payment, tax, and reconciliation integrations. Read the honest list above before committing.
How much does June cost?
June's pricing is usage-based or varies by plan · The official pricing URL is recorded, but a stable current amount was not safely recoverable in this pass. Do not publish a number until rechecked..
What do I lose by replacing June?
Honestly: enterprise governance and integrations; bank, payment, tax, and reconciliation integrations; high-volume ingestion and retention; bot and identity resolution; session replay privacy tooling. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to June?
Yes: PostHog (Group analytics turns companies into first-class entities; the dashboards require more assembly than June.) The prompt is for when you want it exactly your way.