Build June

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A first-party analytics collector whose hard problems are a browser flush that actually fires on mobile, an account rollup that does not invent companies out of email domains, and reports that return the same figure twice.

estimated effort multi-daythe files for this build are in the project pack

CollectorA small Node 22 endpoint that validates, truncates and batches, never blocking the pageEvent storeClickHouse, with events append-only and never editedConfigurationSQLite for accounts, memberships, report definitions and exclusion rulesClientA tiny first-party script that queues locally and flushes on the visibility changeReportsDefinitions stored as data, with one shared sessionisation function

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the collector, the report server and the jobs.
Get it nodejs.org or your version manager. open ↗
Verify node --version prints v22 or higher
installfree locally; a small managed instance is a few units of currency a month
Why Product analytics queries scan columns over millions of rows, and a row store makes every report slow enough that you stop opening them.
Get it Run it in a container locally, or use a small managed instance.
Verify A ten million row table answers a group-by in under a second
decidefree
Why It decides whether you need consent, whether the account rollup exists at all, and what the interface has to say, and deciding it after launch means deciding it from whatever you already collected.
Get it Read the current regulatory guidance for where your users are, pick a mode, and write down the lawful basis if it is the second.
Verify The mode is set in configuration, stated in the README and shown in the interface
have readyfree
Why Without it the account rollup invents a very large fictional customer and you will believe it.
Get it Any maintained public list, checked into the repository so the behaviour is reproducible.
Verify The rollup test refuses to group two users from the same public provider
have readyfree
Why Every hard problem here appears only with real browsers: closed tabs, a proxy stripping headers, a customer's monitoring hitting you every minute.
Get it Point the client at something you already run, even at low volume.
Verify Events arrive from at least two browsers and one mobile device

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- events(at, event_name, anon_id, user_id, account_id, props_json, url_path, referrer_host, ua_family, ip_prefix, ingest_at, schema_version) · ip_prefix is truncated at ingest and the full address is written nowhere
- identities(anon_id, user_id, first_seen_at, linked_at) · append only, so a wrong link can be seen rather than only its result
- accounts(id, name, domain, source, created_at) · source records how the mapping was made, which is the only way to audit it later
- account_members(user_id, account_id, from_at, to_at) · ranged, because a report over last quarter must use the membership as it was then
- event_schema(event_name, prop, type, first_seen_at, last_seen_at, sample_count) · discovered rather than declared, so schema drift shows up as a row
- exclusions(id, rule_kind, pattern, reason, effective_from) · bots, internal users and load tests, each with a stated reason and a date
- reports(id, name, definition_json, created_at) · the definition is the report, and it is versioned rather than edited in place
- There is no column anywhere holding a full IP address or a raw user agent string.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`COLLECT_MODE`	required	`measurement`	measurement or identified. Decides whether the account rollup exists at all.
`CLICKHOUSE_URL`	required	`http://localhost:8123`	The event store. Append only, never edited by the application.
`CONFIG_DB_PATH`	required	`./data/config.db`	Accounts, memberships, report definitions and exclusion rules.
`IDENTIFIER_LIFETIME_DAYS`	required	`395`	Tracker lifetime. Check the current guidance for the figure that applies to you.
`EVENT_RETENTION_DAYS`	required	`760`	Enforced by a scheduled job that logs what it deleted.
`BATCH_MAX_BYTES`	required	`32768`	Client batch size, kept under the browser queue limit. Verify the current limit.
`SESSION_GAP_MINUTES`	required	`30`	The single sessionisation parameter, printed on every report that uses it.

The build, in order

1
A collector that does not lose the end of the session
Events reach the store from real browsers, including the ones that close without warning.
1. Unload is the event mobile browsers are allowed to skip.
  Files client/collect.jsdocs/client.md
2. A refused flush should be a delay, not a loss.
  Files client/queue.jstests/queue.test.ts
3. The only place a full address cannot leak from is the place it never reached.
  Files src/collect.tstests/no-full-ip.test.ts
done when · tick each as it passes
watch out
- Sending each event as an ordinary request and flushing the rest on unload loses the tail of every mobile session, because a page can be discarded without unload ever firing; the Beacon specification requires the user agent to schedule immediate transmission when the visibility state becomes hidden and to let those requests run to completion, which is why the beacon on visibility change is the flush that actually happens. The same specification says the send returns false when the payload exceeds the user agent limit and defers that limit to Fetch, so look up the current number and keep your batches well under it rather than discovering silent drops in a month.
next phase
2
Accounts that exist, and joins that are named
A user belongs to a company because somebody said so, not because of the text after the at sign.
1. People move between companies and last quarter did not change.
  Files src/accounts.tstests/membership.test.ts
2. Otherwise your largest customer is a free mail service.
  Files config/public-domains.txttests/rollup.test.ts
3. Measurement mode should be unable to reach the customer file.
  Files src/modes.tstests/mode-isolation.test.ts
done when · tick each as it passes
watch out
- Deriving the account from the domain of a user's email address puts every customer on a public mail provider into one enormous fictional company and silently merges two real customers that share a parent domain, and because the chart still looks plausible nobody notices for a quarter. Make membership an explicit mapping with a recorded source, keep the public provider deny list, and leave an unmapped user unmapped rather than inventing an account of one.
next phase
3
The same question, the same answer, twice
A report over a closed past window returns identical numbers every time it runs.
1. Two definitions of a session is two products.
  Files src/reports/session.tstests/session.test.ts
2. A figure without its window is a rumour.
  Files src/reports/render.tsx
3. The fastest way to find out a number is wrong.
  Files src/reports/drill.tstests/drill.test.ts
done when · tick each as it passes
watch out
- Computing a session with one inactivity gap in the funnel query and a different gap in the retention query gives two reports that disagree by ten percent, and you will spend a day proving which one is wrong before discovering both are; an approximate distinct count that is sensible at a billion rows is an unnecessary source of doubt at a million. One session function, exact distinct counts, and the parameters printed on the face of the report.
next phase
4
Bots, staff and the traffic you should not have counted
Exclusions are revisable because nothing was deleted to make them.
1. You cannot un-drop an event.
  Files src/exclusions.tstests/exclusions.test.ts
2. A rule with no date rewrites history when it is edited.
  Files src/db/exclusions.sql
3. So an over-broad rule is visible rather than believed.
  Files src/reports/exclusions.tsx
done when · tick each as it passes
watch out
- Dropping suspected bot traffic at ingest feels tidy and destroys the evidence, so when the rule turns out to have been matching a real customer's monitoring you cannot recompute last quarter without it. Store everything, exclude at query time against a rule with an effective date, and show the volume each rule removes so a rule that quietly eats a tenth of your traffic is visible on the day it starts.
5
Retention, deletion and saying when a number is an estimateproduct builder
The policies run as jobs, a deletion request is a command, and a sampled figure says it is sampled.
1. A policy nobody can prove ran did not run.
  Files src/jobs/retention.tstests/retention.test.ts
2. Counts per table, so the answer is a document.
  Files src/jobs/erase.tstests/erase.test.ts
3. Not in a footnote nobody reads.
  Files src/reports/render.tsx
done when · tick each as it passes
watch out
- A retention policy that lives in a README is not a retention policy, and a deletion request answered by hand is one you will get wrong within a year; meanwhile the first time you sample to keep the bill down, every figure downstream becomes an estimate and nothing in the interface says so. Run retention and deletion as jobs that log what they touched, and attach the sampled flag to the figure rather than to a footnote on a settings page.

what this build does not replace

Session replay.
A managed ingestion pipeline at scale.
Licensed bot and device identification lists.
Cross-site or cross-product identity.
enterprise governance and integrations
bank, payment, tax, and reconciliation integrations
high-volume ingestion and retention
bot and identity resolution
session replay privacy tooling

after v1, if you want it

A weekly digest of schema drift, built from the discovered schema table rather than from memory
An annotation layer so a release or an outage sits on the chart next to the dip it caused
A second opinion query for one headline figure, written independently, to catch the day the first one drifts

Need the files? The project pack on the verdict page hands your agent the whole brief · more analytics.