Build Piwik PRO

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A first-party collector for one site whose hard problems are firing at the moment a page actually goes away rather than at an event browsers no longer honour, counting visits without an identifier that outlives a day, and keeping reports answerable from aggregates after the raw rows have been deleted.

estimated effort weekendthe files for this build are in the project pack

CollectorA small first-party endpoint on your own host, POST onlyClientA hand-written script that sends on visibilitychange using the beacon APIStorageSQLite for a small site, with raw events and daily rollups as separate tablesIdentityA daily salted hash of coarse request attributes, with the salt destroyed after its windowReportingPre-aggregated daily rollups only, with no query path into the raw table

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the collector and the reporting application.
Get it nodejs.org or your version manager. open ↗
Verify node --version prints v22 or higher
have readyfree if you already host the site
Why First-party collection is one of the exemption conditions, and a collector on a different domain fails it before any code is written.
Get it A subdomain of the measured site, or a path on the same origin behind your existing server.
Verify The script and the collector are both served from the measured host
have readyfree
Why Rule zero is one regulator's published position and the text moves, so the conditions you build against need reading rather than inheriting.
Get it Read the current guidance for the jurisdiction you operate in and write the conditions and the date into the README. open ↗
Verify The README lists the conditions, the regulator and the date read
have readyfree
Why Hand-written patterns are how real browsers get excluded silently, and starting from a maintained list with a citation is much safer.
Get it Any maintained public list; record the URL and the date in each row.
Verify Every seeded row has a source and a date
decidefree
Why These numbers appear in the privacy text, the retention job and the tests, and deciding them late means a schema built for a longer window than you intend to claim.
Get it Write the raw-event window, the rollup window and the tracker lifetime into configuration.
Verify All three render on the privacy page from configuration

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- sites(id, host, created_at, salt_rotation_hour) · exactly one publisher, because the exemption conditions say so
- events(id, site_id, occurred_at, received_at, name, path, referrer_host, country, device_class, visit_key, props_json) · no full IP address, no cookie identifier, no column that identifies a person
- salts(day, value, created_at, destroyed_at) · one per day, destroyed after its window, which is what makes yesterday's visit keys unlinkable to today's
- rollups_daily(site_id, day, path, name, count, visits) · everything the reports read comes from here
- retention(site_id, raw_days, rollup_days) · enforced by a job that actually deletes, and tested at the boundary
- bots(pattern, kind, source, added_on) · a list you maintain by hand, with where each entry came from
- deletion_requests(id, received_at, scope, completed_at, rows_affected) · because somebody will ask and the answer needs a record
- There is no users table and no identifier that survives the salt rotation. The schema is the privacy claim; the policy page is only a description of it.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_PATH`	required	`./data/analytics.db`	Events, rollups and salts. Back up the rollups; the raw events are transient by design.
`SITE_HOST`	required	`example.com`	The single publisher this deployment measures. The collector rejects other origins.
`SALT_ROTATION_HOUR`	required	`3`	Local hour at which the day's salt is destroyed after uniques are rolled up.
`RAW_RETENTION_DAYS`	required	`30`	Raw events are deleted after this. Reports do not depend on them.
`ROLLUP_RETENTION_DAYS`	required	`395`	Aggregates kept for roughly thirteen months, matching the tracker lifetime you claim.
`SESSION_GAP_MINUTES`	required	`30`	Events sharing a visit key within this gap are one visit. Stated on the reports page.
`ADMIN_TOKEN`secret	required	`redacted`	The only credential for reports and deletion. There are no accounts.

The build, in order

1
A collector that fires when the page actually goes away
An event is recorded when somebody leaves, including on a phone, and the request never carries a full IP address into storage.
1. It is the only page-exit signal that is reliably fired.
  Files src/client/collect.tstests/no-unload.test.ts
2. A false return is a silently lost batch and the cap is 64 KiB.
  Files src/client/collect.tstests/beacon-limit.test.ts
3. Before storage, before logs, before anything keeps a copy.
  Files src/server/ingest.tstests/no-full-ip.test.ts
done when · tick each as it passes
watch out
- Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value.
next phase
2
Counting visits without an identifier
Visits are countable for a day and become uncountable afterwards, by construction.
1. An archived salt is a permanent identifier wearing a disguise.
  Files src/server/salt.tstests/salt-destroyed.test.ts
2. Afterwards the number is genuinely unrecoverable.
  Files src/server/rollup.ts
3. Asserted in a browser test, not in a policy page.
  Files src/client/collect.tstests/no-storage.test.ts
done when · tick each as it passes
watch out
- Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce.
next phase
3
Reports that survive the raw rows being deleted
Every report reads a rollup, so retention can delete raw events without deleting history.
1. Reports that read raw data cannot survive deletion.
  Files src/server/rollup.tssrc/app/reports/page.tsx
2. A rule enforced by grants is a rule that holds.
  Files src/server/db.tstests/reporting-isolation.test.ts
3. Deletion should be boring and visible.
  Files src/jobs/retention.tstests/retention-boundary.test.ts
done when · tick each as it passes
watch out
- A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be.
next phase
4
Traffic that is not people
Automated traffic and your own visits are excluded with a rule you can point at, not a hunch.
1. A pattern nobody can trace is a pattern nobody can fix.
  Files migrations/004_bots.sqlsrc/server/bots.ts
2. An invisible filter is an unfalsifiable number.
  Files src/server/ingest.tssrc/app/reports/page.tsx
3. Overly broad patterns are the failure mode here.
  Files tests/bot-patterns.test.ts
done when · tick each as it passes
watch out
- Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic.
5
Proving the claims, and getting the data outproduct builder
Each exemption condition is asserted by a test, deletion requests are answerable, and everything exports.
1. The test suite is the argument you would actually make.
  Files tests/conditions/*.test.ts
2. Prose and code drift; shared constants do not.
  Files src/app/privacy/page.tsxsrc/config.ts
3. Somebody will ask, and the answer needs a row.
  Files src/server/deletion.tstests/deletion.test.ts
done when · tick each as it passes
watch out
- A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads.

what this build does not replace

Session replay and form field capture.
Cross-site or cross-application identifiers.
Joins against a customer table.
Advertising attribution and conversion modelling.
session replay privacy tooling
enterprise governance and integrations
high-volume ingestion and retention
bot and identity resolution

after v1, if you want it

A second site on the same deployment, which is only safe once the per-publisher isolation is enforced by grants rather than by a where clause
Server-side event ingestion from your own backend, which avoids the browser entirely for events that do not need it
A published methodology page describing exactly how a visit is counted, which is the number people argue about

Need the files? The project pack on the verdict page hands your agent the whole brief · more analytics.