Vibecode Piwik PRO
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The visible privacy analytics suite loop is buildable, but a credible replacement needs more than the first screen. Piwik PRO earns its keep through ingestion, storage, reliability, so expect a weekend or multi-day build and a narrower personal scope.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Piwik PRO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Piwik PRO · indie build A first-party collector for one site whose hard problems are firing at the moment a page actually goes away rather than at an event browsers no longer honour, counting visits without an identifier that outlives a day, and keeping reports answerable from aggregates after the raw rows have been deleted. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small first-party endpoint on your own host, POST only | a third-party endpoint is the thing the exemption conditions rule out | | Client | A hand-written script that sends on visibilitychange using the beacon API | the unload event is not fired reliably and some browsers have stopped firing it at all | | Storage | SQLite for a small site, with raw events and daily rollups as separate tables | the reports read rollups so the raw rows can be deleted on schedule without losing history | | Identity | A daily salted hash of coarse request attributes, with the salt destroyed after its window | you need to count visits, and that is a different requirement from knowing who somebody is | | Reporting | Pre-aggregated daily rollups only, with no query path into the raw table | if a report cannot be answered from an aggregate it is a report this build does not offer | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the collector and the reporting application. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **A host serving the collector on the same domain as the site** · free if you already host the site - Why: First-party collection is one of the exemption conditions, and a collector on a different domain fails it before any code is written. - Get it: A subdomain of the measured site, or a path on the same origin behind your existing server. - Verify: The script and the collector are both served from the measured host - [ ] **Your own regulator's current guidance on analytics trackers** · free - Why: Rule zero is one regulator's published position and the text moves, so the conditions you build against need reading rather than inheriting. - Get it: Read the current guidance for the jurisdiction you operate in and write the conditions and the date into the README. - Verify: The README lists the conditions, the regulator and the date read - [ ] **A published list of automated user agents to seed the bots table** · free - Why: Hand-written patterns are how real browsers get excluded silently, and starting from a maintained list with a citation is much safer. - Get it: Any maintained public list; record the URL and the date in each row. - Verify: Every seeded row has a source and a date - [ ] **The retention windows and the tracker lifetime, decided before the schema** · free - Why: These numbers appear in the privacy text, the retention job and the tests, and deciding them late means a schema built for a longer window than you intend to claim. - Get it: Write the raw-event window, the rollup window and the tracker lifetime into configuration. - Verify: All three render on the privacy page from configuration ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Session replay and form field capture. - Cross-site or cross-application identifiers. - Joins against a customer table. - Advertising attribution and conversion modelling. - session replay privacy tooling - enterprise governance and integrations - high-volume ingestion and retention - bot and identity resolution If one of those is essential to you, that is the reason to keep paying for Piwik PRO, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Piwik PRO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party analytics collector for one site whose privacy claims are enforced by tests rather than asserted in a policy page to replace Piwik PRO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero An analytics tool that sets a tracker without asking is either exempt from consent or unlawful, and the conditions for exemption are published rather than a matter of taste. The French regulator's guidance on using analytics sets them out plainly: inform people the trackers exist, give them the ability to object, keep the purpose to audience measurement and A/B testing only, do not cross-check the data with any other processing such as a customer file or visit statistics from other sites, keep the tracker scoped to a single site or application publisher, truncate the last byte of the IP address, and limit the lifetime of the trackers to thirteen months. The same guidance notes that most large audience measurement offerings do not fall within the exemption whatever their configuration. So build to that list and treat every item as a test rather than a sentence in a policy. Add an identifier that follows somebody between sites, a join against a customer table, or a profile outliving thirteen months, and you have a product that needs a consent banner, which is not something you bolt on afterwards. Regulators differ and the text moves, so read your own regulator's current position before you ship. And no session replay: capturing what somebody typed before they sent it is a different product and a different risk. ### Stack (fixed, do not substitute) - Collector: A small first-party endpoint on your own host, POST only · a third-party endpoint is the thing the exemption conditions rule out - Client: A hand-written script that sends on visibilitychange using the beacon API · the unload event is not fired reliably and some browsers have stopped firing it at all - Storage: SQLite for a small site, with raw events and daily rollups as separate tables · the reports read rollups so the raw rows can be deleted on schedule without losing history - Identity: A daily salted hash of coarse request attributes, with the salt destroyed after its window · you need to count visits, and that is a different requirement from knowing who somebody is - Reporting: Pre-aggregated daily rollups only, with no query path into the raw table · if a report cannot be answered from an aggregate it is a report this build does not offer ### Data model (create this before Phase 1) - sites(id, host, created_at, salt_rotation_hour) · exactly one publisher, because the exemption conditions say so - events(id, site_id, occurred_at, received_at, name, path, referrer_host, country, device_class, visit_key, props_json) · no full IP address, no cookie identifier, no column that identifies a person - salts(day, value, created_at, destroyed_at) · one per day, destroyed after its window, which is what makes yesterday's visit keys unlinkable to today's - rollups_daily(site_id, day, path, name, count, visits) · everything the reports read comes from here - retention(site_id, raw_days, rollup_days) · enforced by a job that actually deletes, and tested at the boundary - bots(pattern, kind, source, added_on) · a list you maintain by hand, with where each entry came from - deletion_requests(id, received_at, scope, completed_at, rows_affected) · because somebody will ask and the answer needs a record - There is no users table and no identifier that survives the salt rotation. The schema is the privacy claim; the policy page is only a description of it. ### Phase 1 · A collector that fires when the page actually goes away Build: Send with the beacon API from a visibilitychange handler when the document becomes hidden, which MDN describes as the last event that is reliably fired and therefore the best place to send analytics data. Do not use unload: MDN states it is not reliably fired especially on mobile, that it blocks the back forward cache, and that Chrome has stopped firing it by default. Truncate the last byte of the address at the edge before anything is written. Trap: Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. Done when: Events are sent from a visibilitychange handler and no unload or pagehide listener exists anywhere in the script, verified by a test that greps the built bundle. The beacon call's return value is checked and a false return falls back to a smaller batch rather than dropping the data silently, tested by exceeding the documented 64 KiB limit. No full IP address is written to storage or to any log, tested by a fixture request and an assertion over both. The script is served first-party from the same host as the site, and the collector rejects requests whose origin is not the configured host. Do not build yet: No visit counting, no reports, no retention, no bot handling yet. ### Phase 2 · Counting visits without an identifier Build: Derive the visit key as a hash of the day's secret salt together with the truncated address, the user agent string and the site identifier. Rotate the salt daily at a configured hour and destroy the previous value rather than archiving it. Sessionise by grouping events sharing a visit key within a configured gap. Store nothing on the visitor's device. Trap: Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. Done when: Destroying a day's salt is irreversible and the previous value is absent from the database, the backups and the logs, tested by inspecting all three after a rotation. Nothing is written to the visitor's device, verified by a browser test asserting empty cookie, local and session storage after a full session. Daily unique counts are computed into the rollup before the salt is destroyed, tested at the rotation boundary. No report offers a cross-day unique visitor figure and the interface explains why in one sentence. Do not build yet: No reports, no retention job, no bot filtering yet. ### Phase 3 · Reports that survive the raw rows being deleted Build: Roll up daily per site, path and event name with counts and visits. Point every report at the rollup table and give the reporting layer no access path to the raw events at all. Run the retention job on a schedule so raw rows older than the configured window are deleted and rollups older than their own window follow. Show the retention windows on the reports page. Trap: A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. Done when: Deleting every raw event older than the window leaves all reports unchanged, tested by running retention and re-rendering. The reporting layer has no query path to the events table, enforced by a separate read-only connection whose grants exclude it and asserted in a test. Retention deletes at the configured boundary and the deletion is counted and logged without row contents, tested. Both retention windows are displayed on the reports page. Do not build yet: No bot filtering, no deletion requests, no export yet. ### Phase 4 · Traffic that is not people Build: Maintain the bots table by hand from a published list and record where each entry came from and when. Exclude matches at ingest but keep a count of what was excluded so the exclusion rate is visible. Exclude your own traffic with an explicit opt-out you set on your own devices. Show excluded counts beside every report figure rather than hiding them. Trap: Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. Done when: Every bots row records its source and the date it was added, enforced by a not-null constraint. Excluded events are counted per rule and the counts appear beside the report figures, checked in the interface test. A pattern matching an ordinary browser user agent fails a test containing a fixture of common real agents. Your own opt-out works without a cookie, tested, since setting a cookie to avoid tracking is its own small joke. Do not build yet: No deletion requests, no export, no operations work yet. ### Phase 5 · Proving the claims, and getting the data out Build: Write one test per condition from rule zero and name each test after the condition, so the suite reads as the compliance argument. Add a deletion path that takes a scope, deletes matching rows, records the count and cannot be made to print row contents. Export rollups and remaining raw events as CSV. Publish the privacy text from the same constants the code uses, so the two cannot drift apart. Trap: A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads. Done when: One named test exists per exemption condition and the suite fails if any is removed, checked by deleting one deliberately. The privacy text renders its numbers from the same configuration the collector and retention job read, tested by changing a value and asserting the page changes. A deletion request by scope deletes the matching rows, records the count, and logs nothing about their contents, tested. Export produces rollups and raw events as CSV and reimports into an empty database with identical report output. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay and form field capture, which record what somebody typed before they chose to send it and are a different product with a different legal footing. - Cross-site or cross-application identifiers of any kind, because the exemption conditions rule them out and adding one turns this into a product that needs a banner. - Joining analytics to a customer table, which the same conditions explicitly name as cross-checking with other processing. - Advertising attribution and conversion modelling, which exist to follow somebody between sites and cannot be built inside these constraints honestly. ### README must contain - The exemption conditions this build is designed against, quoted with the regulator and the date they were read, and the instruction to check them again before shipping - That identity is a daily salted hash whose salt is destroyed, so a cross-day unique visitor number cannot be produced by design - That reports read only daily rollups, which is what lets raw events be deleted on schedule - That the collector sends on visibilitychange and why unload is not used - How to answer a deletion request and what the record of one looks like ===== AGENTS.md ===== # Agent instructions · Piwik PRO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A small first-party endpoint on your own host, POST only, A hand-written script that sends on visibilitychange using the beacon API, SQLite for a small site, with raw events and daily rollups as separate tables, A daily salted hash of coarse request attributes, with the salt destroyed after its window, Pre-aggregated daily rollups only, with no query path into the raw table. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. ===== BUILD_PLAN.md ===== # Build plan · Piwik PRO A first-party collector for one site whose hard problems are firing at the moment a page actually goes away rather than at an event browsers no longer honour, counting visits without an identifier that outlives a day, and keeping reports answerable from aggregates after the raw rows have been deleted. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A collector that fires when the page actually goes away An event is recorded when somebody leaves, including on a phone, and the request never carries a full IP address into storage. ### Steps 1. Send on visibilitychange, never on unload It is the only page-exit signal that is reliably fired. Files: `src/client/collect.ts`, `tests/no-unload.test.ts` 2. Check the beacon return value A false return is a silently lost batch and the cap is 64 KiB. Files: `src/client/collect.ts`, `tests/beacon-limit.test.ts` 3. Truncate the address at the edge Before storage, before logs, before anything keeps a copy. Files: `src/server/ingest.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] No unload or pagehide listener exists in the bundle - [ ] A false beacon return falls back rather than dropping data - [ ] No full IP address reaches storage or logs - [ ] The collector refuses requests from other origins ### Watch out - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. ## Phase 2 · Counting visits without an identifier Visits are countable for a day and become uncountable afterwards, by construction. ### Steps 1. Rotate and destroy the salt on a schedule An archived salt is a permanent identifier wearing a disguise. Files: `src/server/salt.ts`, `tests/salt-destroyed.test.ts` 2. Compute daily uniques before destroying it Afterwards the number is genuinely unrecoverable. Files: `src/server/rollup.ts` 3. Store nothing on the device Asserted in a browser test, not in a policy page. Files: `src/client/collect.ts`, `tests/no-storage.test.ts` ### Done when - [ ] A destroyed salt is absent from database, backups and logs - [ ] Nothing is written to cookie or browser storage - [ ] Daily uniques are computed before rotation - [ ] No cross-day unique figure is offered ### Watch out - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. ## Phase 3 · Reports that survive the raw rows being deleted Every report reads a rollup, so retention can delete raw events without deleting history. ### Steps 1. Roll up daily and report only from rollups Reports that read raw data cannot survive deletion. Files: `src/server/rollup.ts`, `src/app/reports/page.tsx` 2. Give reporting a connection that cannot see raw events A rule enforced by grants is a rule that holds. Files: `src/server/db.ts`, `tests/reporting-isolation.test.ts` 3. Run retention on a schedule and show the windows Deletion should be boring and visible. Files: `src/jobs/retention.ts`, `tests/retention-boundary.test.ts` ### Done when - [ ] Reports are unchanged after raw deletion - [ ] Reporting cannot query the events table - [ ] Retention deletes at the configured boundary - [ ] Retention windows appear on the reports page ### Watch out - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. ## Phase 4 · Traffic that is not people Automated traffic and your own visits are excluded with a rule you can point at, not a hunch. ### Steps 1. Record the source and date for every rule A pattern nobody can trace is a pattern nobody can fix. Files: `migrations/004_bots.sql`, `src/server/bots.ts` 2. Count exclusions per rule and show them An invisible filter is an unfalsifiable number. Files: `src/server/ingest.ts`, `src/app/reports/page.tsx` 3. Test rules against real user agents Overly broad patterns are the failure mode here. Files: `tests/bot-patterns.test.ts` ### Done when - [ ] Every bot rule records its source and date - [ ] Exclusion counts appear beside report figures - [ ] No rule matches a common real user agent - [ ] The self opt-out works without a cookie ### Watch out - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. ## Not in this build - Session replay and form field capture. - Cross-site or cross-application identifiers. - Joins against a customer table. - Advertising attribution and conversion modelling. ## After v1, if you want it - A second site on the same deployment, which is only safe once the per-publisher isolation is enforced by grants rather than by a where clause - Server-side event ingestion from your own backend, which avoids the browser entirely for events that do not need it - A published methodology page describing exactly how a visit is counted, which is the number people argue about ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Events, rollups and salts. Back up the rollups; the raw events are transient by design. DATABASE_PATH=./data/analytics.db # Required. The single publisher this deployment measures. The collector rejects other origins. SITE_HOST=example.com # Required. Local hour at which the day's salt is destroyed after uniques are rolled up. SALT_ROTATION_HOUR=3 # Required. Raw events are deleted after this. Reports do not depend on them. RAW_RETENTION_DAYS=30 # Required. Aggregates kept for roughly thirteen months, matching the tracker lifetime you claim. ROLLUP_RETENTION_DAYS=395 # Required. Events sharing a visit key within this gap are one visit. Stated on the reports page. SESSION_GAP_MINUTES=30 # Required · secret. The only credential for reports and deletion. There are no accounts. ADMIN_TOKEN=redacted
You are building a lean indie version of Piwik PRO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Piwik PRO · indie build A first-party collector for one site whose hard problems are firing at the moment a page actually goes away rather than at an event browsers no longer honour, counting visits without an identifier that outlives a day, and keeping reports answerable from aggregates after the raw rows have been deleted. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small first-party endpoint on your own host, POST only | a third-party endpoint is the thing the exemption conditions rule out | | Client | A hand-written script that sends on visibilitychange using the beacon API | the unload event is not fired reliably and some browsers have stopped firing it at all | | Storage | SQLite for a small site, with raw events and daily rollups as separate tables | the reports read rollups so the raw rows can be deleted on schedule without losing history | | Identity | A daily salted hash of coarse request attributes, with the salt destroyed after its window | you need to count visits, and that is a different requirement from knowing who somebody is | | Reporting | Pre-aggregated daily rollups only, with no query path into the raw table | if a report cannot be answered from an aggregate it is a report this build does not offer | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the collector and the reporting application. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **A host serving the collector on the same domain as the site** · free if you already host the site - Why: First-party collection is one of the exemption conditions, and a collector on a different domain fails it before any code is written. - Get it: A subdomain of the measured site, or a path on the same origin behind your existing server. - Verify: The script and the collector are both served from the measured host - [ ] **Your own regulator's current guidance on analytics trackers** · free - Why: Rule zero is one regulator's published position and the text moves, so the conditions you build against need reading rather than inheriting. - Get it: Read the current guidance for the jurisdiction you operate in and write the conditions and the date into the README. - Verify: The README lists the conditions, the regulator and the date read - [ ] **A published list of automated user agents to seed the bots table** · free - Why: Hand-written patterns are how real browsers get excluded silently, and starting from a maintained list with a citation is much safer. - Get it: Any maintained public list; record the URL and the date in each row. - Verify: Every seeded row has a source and a date - [ ] **The retention windows and the tracker lifetime, decided before the schema** · free - Why: These numbers appear in the privacy text, the retention job and the tests, and deciding them late means a schema built for a longer window than you intend to claim. - Get it: Write the raw-event window, the rollup window and the tracker lifetime into configuration. - Verify: All three render on the privacy page from configuration ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Session replay and form field capture. - Cross-site or cross-application identifiers. - Joins against a customer table. - Advertising attribution and conversion modelling. - session replay privacy tooling - enterprise governance and integrations - high-volume ingestion and retention - bot and identity resolution If one of those is essential to you, that is the reason to keep paying for Piwik PRO, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Piwik PRO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party analytics collector for one site whose privacy claims are enforced by tests rather than asserted in a policy page to replace Piwik PRO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero An analytics tool that sets a tracker without asking is either exempt from consent or unlawful, and the conditions for exemption are published rather than a matter of taste. The French regulator's guidance on using analytics sets them out plainly: inform people the trackers exist, give them the ability to object, keep the purpose to audience measurement and A/B testing only, do not cross-check the data with any other processing such as a customer file or visit statistics from other sites, keep the tracker scoped to a single site or application publisher, truncate the last byte of the IP address, and limit the lifetime of the trackers to thirteen months. The same guidance notes that most large audience measurement offerings do not fall within the exemption whatever their configuration. So build to that list and treat every item as a test rather than a sentence in a policy. Add an identifier that follows somebody between sites, a join against a customer table, or a profile outliving thirteen months, and you have a product that needs a consent banner, which is not something you bolt on afterwards. Regulators differ and the text moves, so read your own regulator's current position before you ship. And no session replay: capturing what somebody typed before they sent it is a different product and a different risk. ### Stack (fixed, do not substitute) - Collector: A small first-party endpoint on your own host, POST only · a third-party endpoint is the thing the exemption conditions rule out - Client: A hand-written script that sends on visibilitychange using the beacon API · the unload event is not fired reliably and some browsers have stopped firing it at all - Storage: SQLite for a small site, with raw events and daily rollups as separate tables · the reports read rollups so the raw rows can be deleted on schedule without losing history - Identity: A daily salted hash of coarse request attributes, with the salt destroyed after its window · you need to count visits, and that is a different requirement from knowing who somebody is - Reporting: Pre-aggregated daily rollups only, with no query path into the raw table · if a report cannot be answered from an aggregate it is a report this build does not offer ### Data model (create this before Phase 1) - sites(id, host, created_at, salt_rotation_hour) · exactly one publisher, because the exemption conditions say so - events(id, site_id, occurred_at, received_at, name, path, referrer_host, country, device_class, visit_key, props_json) · no full IP address, no cookie identifier, no column that identifies a person - salts(day, value, created_at, destroyed_at) · one per day, destroyed after its window, which is what makes yesterday's visit keys unlinkable to today's - rollups_daily(site_id, day, path, name, count, visits) · everything the reports read comes from here - retention(site_id, raw_days, rollup_days) · enforced by a job that actually deletes, and tested at the boundary - bots(pattern, kind, source, added_on) · a list you maintain by hand, with where each entry came from - deletion_requests(id, received_at, scope, completed_at, rows_affected) · because somebody will ask and the answer needs a record - There is no users table and no identifier that survives the salt rotation. The schema is the privacy claim; the policy page is only a description of it. ### Phase 1 · A collector that fires when the page actually goes away Build: Send with the beacon API from a visibilitychange handler when the document becomes hidden, which MDN describes as the last event that is reliably fired and therefore the best place to send analytics data. Do not use unload: MDN states it is not reliably fired especially on mobile, that it blocks the back forward cache, and that Chrome has stopped firing it by default. Truncate the last byte of the address at the edge before anything is written. Trap: Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. Done when: Events are sent from a visibilitychange handler and no unload or pagehide listener exists anywhere in the script, verified by a test that greps the built bundle. The beacon call's return value is checked and a false return falls back to a smaller batch rather than dropping the data silently, tested by exceeding the documented 64 KiB limit. No full IP address is written to storage or to any log, tested by a fixture request and an assertion over both. The script is served first-party from the same host as the site, and the collector rejects requests whose origin is not the configured host. Do not build yet: No visit counting, no reports, no retention, no bot handling yet. ### Phase 2 · Counting visits without an identifier Build: Derive the visit key as a hash of the day's secret salt together with the truncated address, the user agent string and the site identifier. Rotate the salt daily at a configured hour and destroy the previous value rather than archiving it. Sessionise by grouping events sharing a visit key within a configured gap. Store nothing on the visitor's device. Trap: Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. Done when: Destroying a day's salt is irreversible and the previous value is absent from the database, the backups and the logs, tested by inspecting all three after a rotation. Nothing is written to the visitor's device, verified by a browser test asserting empty cookie, local and session storage after a full session. Daily unique counts are computed into the rollup before the salt is destroyed, tested at the rotation boundary. No report offers a cross-day unique visitor figure and the interface explains why in one sentence. Do not build yet: No reports, no retention job, no bot filtering yet. ### Phase 3 · Reports that survive the raw rows being deleted Build: Roll up daily per site, path and event name with counts and visits. Point every report at the rollup table and give the reporting layer no access path to the raw events at all. Run the retention job on a schedule so raw rows older than the configured window are deleted and rollups older than their own window follow. Show the retention windows on the reports page. Trap: A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. Done when: Deleting every raw event older than the window leaves all reports unchanged, tested by running retention and re-rendering. The reporting layer has no query path to the events table, enforced by a separate read-only connection whose grants exclude it and asserted in a test. Retention deletes at the configured boundary and the deletion is counted and logged without row contents, tested. Both retention windows are displayed on the reports page. Do not build yet: No bot filtering, no deletion requests, no export yet. ### Phase 4 · Traffic that is not people Build: Maintain the bots table by hand from a published list and record where each entry came from and when. Exclude matches at ingest but keep a count of what was excluded so the exclusion rate is visible. Exclude your own traffic with an explicit opt-out you set on your own devices. Show excluded counts beside every report figure rather than hiding them. Trap: Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. Done when: Every bots row records its source and the date it was added, enforced by a not-null constraint. Excluded events are counted per rule and the counts appear beside the report figures, checked in the interface test. A pattern matching an ordinary browser user agent fails a test containing a fixture of common real agents. Your own opt-out works without a cookie, tested, since setting a cookie to avoid tracking is its own small joke. Do not build yet: No deletion requests, no export, no operations work yet. ### Phase 5 · Proving the claims, and getting the data out Build: Write one test per condition from rule zero and name each test after the condition, so the suite reads as the compliance argument. Add a deletion path that takes a scope, deletes matching rows, records the count and cannot be made to print row contents. Export rollups and remaining raw events as CSV. Publish the privacy text from the same constants the code uses, so the two cannot drift apart. Trap: A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads. Done when: One named test exists per exemption condition and the suite fails if any is removed, checked by deleting one deliberately. The privacy text renders its numbers from the same configuration the collector and retention job read, tested by changing a value and asserting the page changes. A deletion request by scope deletes the matching rows, records the count, and logs nothing about their contents, tested. Export produces rollups and raw events as CSV and reimports into an empty database with identical report output. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay and form field capture, which record what somebody typed before they chose to send it and are a different product with a different legal footing. - Cross-site or cross-application identifiers of any kind, because the exemption conditions rule them out and adding one turns this into a product that needs a banner. - Joining analytics to a customer table, which the same conditions explicitly name as cross-checking with other processing. - Advertising attribution and conversion modelling, which exist to follow somebody between sites and cannot be built inside these constraints honestly. ### README must contain - The exemption conditions this build is designed against, quoted with the regulator and the date they were read, and the instruction to check them again before shipping - That identity is a daily salted hash whose salt is destroyed, so a cross-day unique visitor number cannot be produced by design - That reports read only daily rollups, which is what lets raw events be deleted on schedule - That the collector sends on visibilitychange and why unload is not used - How to answer a deletion request and what the record of one looks like ===== AGENTS.md ===== # Agent instructions · Piwik PRO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A small first-party endpoint on your own host, POST only, A hand-written script that sends on visibilitychange using the beacon API, SQLite for a small site, with raw events and daily rollups as separate tables, A daily salted hash of coarse request attributes, with the salt destroyed after its window, Pre-aggregated daily rollups only, with no query path into the raw table. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. ===== BUILD_PLAN.md ===== # Build plan · Piwik PRO A first-party collector for one site whose hard problems are firing at the moment a page actually goes away rather than at an event browsers no longer honour, counting visits without an identifier that outlives a day, and keeping reports answerable from aggregates after the raw rows have been deleted. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A collector that fires when the page actually goes away An event is recorded when somebody leaves, including on a phone, and the request never carries a full IP address into storage. ### Steps 1. Send on visibilitychange, never on unload It is the only page-exit signal that is reliably fired. Files: `src/client/collect.ts`, `tests/no-unload.test.ts` 2. Check the beacon return value A false return is a silently lost batch and the cap is 64 KiB. Files: `src/client/collect.ts`, `tests/beacon-limit.test.ts` 3. Truncate the address at the edge Before storage, before logs, before anything keeps a copy. Files: `src/server/ingest.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] No unload or pagehide listener exists in the bundle - [ ] A false beacon return falls back rather than dropping data - [ ] No full IP address reaches storage or logs - [ ] The collector refuses requests from other origins ### Watch out - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. ## Phase 2 · Counting visits without an identifier Visits are countable for a day and become uncountable afterwards, by construction. ### Steps 1. Rotate and destroy the salt on a schedule An archived salt is a permanent identifier wearing a disguise. Files: `src/server/salt.ts`, `tests/salt-destroyed.test.ts` 2. Compute daily uniques before destroying it Afterwards the number is genuinely unrecoverable. Files: `src/server/rollup.ts` 3. Store nothing on the device Asserted in a browser test, not in a policy page. Files: `src/client/collect.ts`, `tests/no-storage.test.ts` ### Done when - [ ] A destroyed salt is absent from database, backups and logs - [ ] Nothing is written to cookie or browser storage - [ ] Daily uniques are computed before rotation - [ ] No cross-day unique figure is offered ### Watch out - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. ## Phase 3 · Reports that survive the raw rows being deleted Every report reads a rollup, so retention can delete raw events without deleting history. ### Steps 1. Roll up daily and report only from rollups Reports that read raw data cannot survive deletion. Files: `src/server/rollup.ts`, `src/app/reports/page.tsx` 2. Give reporting a connection that cannot see raw events A rule enforced by grants is a rule that holds. Files: `src/server/db.ts`, `tests/reporting-isolation.test.ts` 3. Run retention on a schedule and show the windows Deletion should be boring and visible. Files: `src/jobs/retention.ts`, `tests/retention-boundary.test.ts` ### Done when - [ ] Reports are unchanged after raw deletion - [ ] Reporting cannot query the events table - [ ] Retention deletes at the configured boundary - [ ] Retention windows appear on the reports page ### Watch out - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. ## Phase 4 · Traffic that is not people Automated traffic and your own visits are excluded with a rule you can point at, not a hunch. ### Steps 1. Record the source and date for every rule A pattern nobody can trace is a pattern nobody can fix. Files: `migrations/004_bots.sql`, `src/server/bots.ts` 2. Count exclusions per rule and show them An invisible filter is an unfalsifiable number. Files: `src/server/ingest.ts`, `src/app/reports/page.tsx` 3. Test rules against real user agents Overly broad patterns are the failure mode here. Files: `tests/bot-patterns.test.ts` ### Done when - [ ] Every bot rule records its source and date - [ ] Exclusion counts appear beside report figures - [ ] No rule matches a common real user agent - [ ] The self opt-out works without a cookie ### Watch out - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. ## Not in this build - Session replay and form field capture. - Cross-site or cross-application identifiers. - Joins against a customer table. - Advertising attribution and conversion modelling. ## After v1, if you want it - A second site on the same deployment, which is only safe once the per-publisher isolation is enforced by grants rather than by a where clause - Server-side event ingestion from your own backend, which avoids the browser entirely for events that do not need it - A published methodology page describing exactly how a visit is counted, which is the number people argue about ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Events, rollups and salts. Back up the rollups; the raw events are transient by design. DATABASE_PATH=./data/analytics.db # Required. The single publisher this deployment measures. The collector rejects other origins. SITE_HOST=example.com # Required. Local hour at which the day's salt is destroyed after uniques are rolled up. SALT_ROTATION_HOUR=3 # Required. Raw events are deleted after this. Reports do not depend on them. RAW_RETENTION_DAYS=30 # Required. Aggregates kept for roughly thirteen months, matching the tracker lifetime you claim. ROLLUP_RETENTION_DAYS=395 # Required. Events sharing a visit key within this gap are one visit. Stated on the reports page. SESSION_GAP_MINUTES=30 # Required · secret. The only credential for reports and deletion. There are no accounts. ADMIN_TOKEN=redacted
You are building a production product version of Piwik PRO. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Piwik PRO · product brief ## Problem The visible privacy analytics suite loop is buildable, but a credible replacement needs more than the first screen. Piwik PRO earns its keep through ingestion, storage, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A collector for one site that records the page exits browsers actually report, counts visits with a key that stops existing at the end of the day, answers every report from aggregates so raw rows can be deleted on schedule, and has one named test for each condition the privacy claim rests on. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - deploy target - ClickHouse or SQLite for low volume - first-party tracking script ## Explicit non-goals for v1 - Session replay and form field capture. - Cross-site or cross-application identifiers. - Joins against a customer table. - Advertising attribution and conversion modelling. - session replay privacy tooling - enterprise governance and integrations - high-volume ingestion and retention - bot and identity resolution ## Success criteria - No unload listener is confirmed present in the bundle - No full IP address is confirmed to reach storage or logs - A destroyed salt is confirmed absent from database, backups and logs - Reports are confirmed unchanged after raw deletion - One named test per exemption condition is confirmed present ===== BRIEF.md ===== # Build brief · Piwik PRO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party analytics collector for one site whose privacy claims are enforced by tests rather than asserted in a policy page to replace Piwik PRO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero An analytics tool that sets a tracker without asking is either exempt from consent or unlawful, and the conditions for exemption are published rather than a matter of taste. The French regulator's guidance on using analytics sets them out plainly: inform people the trackers exist, give them the ability to object, keep the purpose to audience measurement and A/B testing only, do not cross-check the data with any other processing such as a customer file or visit statistics from other sites, keep the tracker scoped to a single site or application publisher, truncate the last byte of the IP address, and limit the lifetime of the trackers to thirteen months. The same guidance notes that most large audience measurement offerings do not fall within the exemption whatever their configuration. So build to that list and treat every item as a test rather than a sentence in a policy. Add an identifier that follows somebody between sites, a join against a customer table, or a profile outliving thirteen months, and you have a product that needs a consent banner, which is not something you bolt on afterwards. Regulators differ and the text moves, so read your own regulator's current position before you ship. And no session replay: capturing what somebody typed before they sent it is a different product and a different risk. ### Stack (fixed, do not substitute) - Collector: A small first-party endpoint on your own host, POST only · a third-party endpoint is the thing the exemption conditions rule out - Client: A hand-written script that sends on visibilitychange using the beacon API · the unload event is not fired reliably and some browsers have stopped firing it at all - Storage: SQLite for a small site, with raw events and daily rollups as separate tables · the reports read rollups so the raw rows can be deleted on schedule without losing history - Identity: A daily salted hash of coarse request attributes, with the salt destroyed after its window · you need to count visits, and that is a different requirement from knowing who somebody is - Reporting: Pre-aggregated daily rollups only, with no query path into the raw table · if a report cannot be answered from an aggregate it is a report this build does not offer ### Data model (create this before Phase 1) - sites(id, host, created_at, salt_rotation_hour) · exactly one publisher, because the exemption conditions say so - events(id, site_id, occurred_at, received_at, name, path, referrer_host, country, device_class, visit_key, props_json) · no full IP address, no cookie identifier, no column that identifies a person - salts(day, value, created_at, destroyed_at) · one per day, destroyed after its window, which is what makes yesterday's visit keys unlinkable to today's - rollups_daily(site_id, day, path, name, count, visits) · everything the reports read comes from here - retention(site_id, raw_days, rollup_days) · enforced by a job that actually deletes, and tested at the boundary - bots(pattern, kind, source, added_on) · a list you maintain by hand, with where each entry came from - deletion_requests(id, received_at, scope, completed_at, rows_affected) · because somebody will ask and the answer needs a record - There is no users table and no identifier that survives the salt rotation. The schema is the privacy claim; the policy page is only a description of it. ### Phase 1 · A collector that fires when the page actually goes away Build: Send with the beacon API from a visibilitychange handler when the document becomes hidden, which MDN describes as the last event that is reliably fired and therefore the best place to send analytics data. Do not use unload: MDN states it is not reliably fired especially on mobile, that it blocks the back forward cache, and that Chrome has stopped firing it by default. Truncate the last byte of the address at the edge before anything is written. Trap: Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. Done when: Events are sent from a visibilitychange handler and no unload or pagehide listener exists anywhere in the script, verified by a test that greps the built bundle. The beacon call's return value is checked and a false return falls back to a smaller batch rather than dropping the data silently, tested by exceeding the documented 64 KiB limit. No full IP address is written to storage or to any log, tested by a fixture request and an assertion over both. The script is served first-party from the same host as the site, and the collector rejects requests whose origin is not the configured host. Do not build yet: No visit counting, no reports, no retention, no bot handling yet. ### Phase 2 · Counting visits without an identifier Build: Derive the visit key as a hash of the day's secret salt together with the truncated address, the user agent string and the site identifier. Rotate the salt daily at a configured hour and destroy the previous value rather than archiving it. Sessionise by grouping events sharing a visit key within a configured gap. Store nothing on the visitor's device. Trap: Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. Done when: Destroying a day's salt is irreversible and the previous value is absent from the database, the backups and the logs, tested by inspecting all three after a rotation. Nothing is written to the visitor's device, verified by a browser test asserting empty cookie, local and session storage after a full session. Daily unique counts are computed into the rollup before the salt is destroyed, tested at the rotation boundary. No report offers a cross-day unique visitor figure and the interface explains why in one sentence. Do not build yet: No reports, no retention job, no bot filtering yet. ### Phase 3 · Reports that survive the raw rows being deleted Build: Roll up daily per site, path and event name with counts and visits. Point every report at the rollup table and give the reporting layer no access path to the raw events at all. Run the retention job on a schedule so raw rows older than the configured window are deleted and rollups older than their own window follow. Show the retention windows on the reports page. Trap: A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. Done when: Deleting every raw event older than the window leaves all reports unchanged, tested by running retention and re-rendering. The reporting layer has no query path to the events table, enforced by a separate read-only connection whose grants exclude it and asserted in a test. Retention deletes at the configured boundary and the deletion is counted and logged without row contents, tested. Both retention windows are displayed on the reports page. Do not build yet: No bot filtering, no deletion requests, no export yet. ### Phase 4 · Traffic that is not people Build: Maintain the bots table by hand from a published list and record where each entry came from and when. Exclude matches at ingest but keep a count of what was excluded so the exclusion rate is visible. Exclude your own traffic with an explicit opt-out you set on your own devices. Show excluded counts beside every report figure rather than hiding them. Trap: Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. Done when: Every bots row records its source and the date it was added, enforced by a not-null constraint. Excluded events are counted per rule and the counts appear beside the report figures, checked in the interface test. A pattern matching an ordinary browser user agent fails a test containing a fixture of common real agents. Your own opt-out works without a cookie, tested, since setting a cookie to avoid tracking is its own small joke. Do not build yet: No deletion requests, no export, no operations work yet. ### Phase 5 · Proving the claims, and getting the data out Build: Write one test per condition from rule zero and name each test after the condition, so the suite reads as the compliance argument. Add a deletion path that takes a scope, deletes matching rows, records the count and cannot be made to print row contents. Export rollups and remaining raw events as CSV. Publish the privacy text from the same constants the code uses, so the two cannot drift apart. Trap: A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads. Done when: One named test exists per exemption condition and the suite fails if any is removed, checked by deleting one deliberately. The privacy text renders its numbers from the same configuration the collector and retention job read, tested by changing a value and asserting the page changes. A deletion request by scope deletes the matching rows, records the count, and logs nothing about their contents, tested. Export produces rollups and raw events as CSV and reimports into an empty database with identical report output. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay and form field capture, which record what somebody typed before they chose to send it and are a different product with a different legal footing. - Cross-site or cross-application identifiers of any kind, because the exemption conditions rule them out and adding one turns this into a product that needs a banner. - Joining analytics to a customer table, which the same conditions explicitly name as cross-checking with other processing. - Advertising attribution and conversion modelling, which exist to follow somebody between sites and cannot be built inside these constraints honestly. ### README must contain - The exemption conditions this build is designed against, quoted with the regulator and the date they were read, and the instruction to check them again before shipping - That identity is a daily salted hash whose salt is destroyed, so a cross-day unique visitor number cannot be produced by design - That reports read only daily rollups, which is what lets raw events be deleted on schedule - That the collector sends on visibilitychange and why unload is not used - How to answer a deletion request and what the record of one looks like ===== ARCHITECTURE.md ===== # Architecture · Piwik PRO ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small first-party endpoint on your own host, POST only | a third-party endpoint is the thing the exemption conditions rule out | | Client | A hand-written script that sends on visibilitychange using the beacon API | the unload event is not fired reliably and some browsers have stopped firing it at all | | Storage | SQLite for a small site, with raw events and daily rollups as separate tables | the reports read rollups so the raw rows can be deleted on schedule without losing history | | Identity | A daily salted hash of coarse request attributes, with the salt destroyed after its window | you need to count visits, and that is a different requirement from knowing who somebody is | | Reporting | Pre-aggregated daily rollups only, with no query path into the raw table | if a report cannot be answered from an aggregate it is a report this build does not offer | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Client | Page-exit detection and beacon batching. | Any script, provided it never listens for unload and checks the beacon return value. | | Ingest | Address truncation, bot exclusion, event writes. | Any server, provided truncation happens before storage and before logs. | | Identity | Daily salt, visit keys and destruction. | Not swappable for a stored identifier, which is the whole point. | | Rollup | Daily aggregates and the reporting boundary. | Any aggregation, provided reporting cannot reach raw events. | | Retention | Scheduled deletion and deletion requests. | Any scheduler, provided deletion is counted and logged without contents. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Events, rollups and salts. Back up the rollups; the raw events are transient by design. - `SITE_HOST` · required · The single publisher this deployment measures. The collector rejects other origins. - `SALT_ROTATION_HOUR` · required · Local hour at which the day's salt is destroyed after uniques are rolled up. - `RAW_RETENTION_DAYS` · required · Raw events are deleted after this. Reports do not depend on them. - `ROLLUP_RETENTION_DAYS` · required · Aggregates kept for roughly thirteen months, matching the tracker lifetime you claim. - `SESSION_GAP_MINUTES` · required · Events sharing a visit key within this gap are one visit. Stated on the reports page. - `ADMIN_TOKEN` · required, secret · The only credential for reports and deletion. There are no accounts. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Piwik PRO product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A small first-party endpoint on your own host, POST only, A hand-written script that sends on visibilitychange using the beacon API, SQLite for a small site, with raw events and daily rollups as separate tables, A daily salted hash of coarse request attributes, with the salt destroyed after its window, Pre-aggregated daily rollups only, with no query path into the raw table. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. - A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads. ===== MILESTONES.md ===== # Delivery milestones · Piwik PRO Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A collector that fires when the page actually goes away An event is recorded when somebody leaves, including on a phone, and the request never carries a full IP address into storage. ### Steps 1. Send on visibilitychange, never on unload It is the only page-exit signal that is reliably fired. Files: `src/client/collect.ts`, `tests/no-unload.test.ts` 2. Check the beacon return value A false return is a silently lost batch and the cap is 64 KiB. Files: `src/client/collect.ts`, `tests/beacon-limit.test.ts` 3. Truncate the address at the edge Before storage, before logs, before anything keeps a copy. Files: `src/server/ingest.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] No unload or pagehide listener exists in the bundle - [ ] A false beacon return falls back rather than dropping data - [ ] No full IP address reaches storage or logs - [ ] The collector refuses requests from other origins ### Watch out - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. ## M2 · Counting visits without an identifier Visits are countable for a day and become uncountable afterwards, by construction. ### Steps 1. Rotate and destroy the salt on a schedule An archived salt is a permanent identifier wearing a disguise. Files: `src/server/salt.ts`, `tests/salt-destroyed.test.ts` 2. Compute daily uniques before destroying it Afterwards the number is genuinely unrecoverable. Files: `src/server/rollup.ts` 3. Store nothing on the device Asserted in a browser test, not in a policy page. Files: `src/client/collect.ts`, `tests/no-storage.test.ts` ### Done when - [ ] A destroyed salt is absent from database, backups and logs - [ ] Nothing is written to cookie or browser storage - [ ] Daily uniques are computed before rotation - [ ] No cross-day unique figure is offered ### Watch out - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. ## M3 · Reports that survive the raw rows being deleted Every report reads a rollup, so retention can delete raw events without deleting history. ### Steps 1. Roll up daily and report only from rollups Reports that read raw data cannot survive deletion. Files: `src/server/rollup.ts`, `src/app/reports/page.tsx` 2. Give reporting a connection that cannot see raw events A rule enforced by grants is a rule that holds. Files: `src/server/db.ts`, `tests/reporting-isolation.test.ts` 3. Run retention on a schedule and show the windows Deletion should be boring and visible. Files: `src/jobs/retention.ts`, `tests/retention-boundary.test.ts` ### Done when - [ ] Reports are unchanged after raw deletion - [ ] Reporting cannot query the events table - [ ] Retention deletes at the configured boundary - [ ] Retention windows appear on the reports page ### Watch out - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. ## M4 · Traffic that is not people Automated traffic and your own visits are excluded with a rule you can point at, not a hunch. ### Steps 1. Record the source and date for every rule A pattern nobody can trace is a pattern nobody can fix. Files: `migrations/004_bots.sql`, `src/server/bots.ts` 2. Count exclusions per rule and show them An invisible filter is an unfalsifiable number. Files: `src/server/ingest.ts`, `src/app/reports/page.tsx` 3. Test rules against real user agents Overly broad patterns are the failure mode here. Files: `tests/bot-patterns.test.ts` ### Done when - [ ] Every bot rule records its source and date - [ ] Exclusion counts appear beside report figures - [ ] No rule matches a common real user agent - [ ] The self opt-out works without a cookie ### Watch out - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. ## M5 · Proving the claims, and getting the data out (production only) Each exemption condition is asserted by a test, deletion requests are answerable, and everything exports. ### Steps 1. Name one test per exemption condition The test suite is the argument you would actually make. Files: `tests/conditions/*.test.ts` 2. Render the privacy numbers from configuration Prose and code drift; shared constants do not. Files: `src/app/privacy/page.tsx`, `src/config.ts` 3. Make deletion a recorded, countable operation Somebody will ask, and the answer needs a row. Files: `src/server/deletion.ts`, `tests/deletion.test.ts` ### Done when - [ ] One named test per exemption condition, and removal fails the suite - [ ] The privacy page renders numbers from configuration - [ ] Deletion records counts and never logs contents - [ ] Export and reimport reproduce identical reports ### Watch out - A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads. ===== OPERATIONS.md ===== # Operations · Piwik PRO ## Backup Rollups nightly. Raw events deliberately not backed up beyond the retention window, because a backup that outlives the retention claim makes the claim untrue, and this should be written down where the backup script lives. ## Restore Restore rollups and confirm the report output matches the last exported CSV. Salts are never restored; a restored salt would resurrect an identifier that was supposed to be gone. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Exclusion rate per bot rule, beacon batches rejected for size, the age of the oldest raw event against the retention window, and whether yesterday's salt was actually destroyed. The last one is the check that matters most and the one most likely to fail silently. ## Incident checklist The incident in this build is a privacy claim that turned out to be false: a salt that was archived, an address that reached a log, a backup that outlived its window. Each has a named test, and the incident response is to find out why the test did not run. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] No unload listener is confirmed present in the bundle - [ ] No full IP address is confirmed to reach storage or logs - [ ] A destroyed salt is confirmed absent from database, backups and logs - [ ] Reports are confirmed unchanged after raw deletion - [ ] One named test per exemption condition is confirmed present ## Launch constraint Do not market omitted Piwik PRO capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Events, rollups and salts. Back up the rollups; the raw events are transient by design. DATABASE_PATH=./data/analytics.db # Required. The single publisher this deployment measures. The collector rejects other origins. SITE_HOST=example.com # Required. Local hour at which the day's salt is destroyed after uniques are rolled up. SALT_ROTATION_HOUR=3 # Required. Raw events are deleted after this. Reports do not depend on them. RAW_RETENTION_DAYS=30 # Required. Aggregates kept for roughly thirteen months, matching the tracker lifetime you claim. ROLLUP_RETENTION_DAYS=395 # Required. Events sharing a visit key within this gap are one visit. Stated on the reports page. SESSION_GAP_MINUTES=30 # Required · secret. The only credential for reports and deletion. There are no accounts. ADMIN_TOKEN=redacted
README.md
# Piwik PRO · indie build A first-party collector for one site whose hard problems are firing at the moment a page actually goes away rather than at an event browsers no longer honour, counting visits without an identifier that outlives a day, and keeping reports answerable from aggregates after the raw rows have been deleted. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small first-party endpoint on your own host, POST only | a third-party endpoint is the thing the exemption conditions rule out | | Client | A hand-written script that sends on visibilitychange using the beacon API | the unload event is not fired reliably and some browsers have stopped firing it at all | | Storage | SQLite for a small site, with raw events and daily rollups as separate tables | the reports read rollups so the raw rows can be deleted on schedule without losing history | | Identity | A daily salted hash of coarse request attributes, with the salt destroyed after its window | you need to count visits, and that is a different requirement from knowing who somebody is | | Reporting | Pre-aggregated daily rollups only, with no query path into the raw table | if a report cannot be answered from an aggregate it is a report this build does not offer | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the collector and the reporting application. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **A host serving the collector on the same domain as the site** · free if you already host the site - Why: First-party collection is one of the exemption conditions, and a collector on a different domain fails it before any code is written. - Get it: A subdomain of the measured site, or a path on the same origin behind your existing server. - Verify: The script and the collector are both served from the measured host - [ ] **Your own regulator's current guidance on analytics trackers** · free - Why: Rule zero is one regulator's published position and the text moves, so the conditions you build against need reading rather than inheriting. - Get it: Read the current guidance for the jurisdiction you operate in and write the conditions and the date into the README. - Verify: The README lists the conditions, the regulator and the date read - [ ] **A published list of automated user agents to seed the bots table** · free - Why: Hand-written patterns are how real browsers get excluded silently, and starting from a maintained list with a citation is much safer. - Get it: Any maintained public list; record the URL and the date in each row. - Verify: Every seeded row has a source and a date - [ ] **The retention windows and the tracker lifetime, decided before the schema** · free - Why: These numbers appear in the privacy text, the retention job and the tests, and deciding them late means a schema built for a longer window than you intend to claim. - Get it: Write the raw-event window, the rollup window and the tracker lifetime into configuration. - Verify: All three render on the privacy page from configuration ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Session replay and form field capture. - Cross-site or cross-application identifiers. - Joins against a customer table. - Advertising attribution and conversion modelling. - session replay privacy tooling - enterprise governance and integrations - high-volume ingestion and retention - bot and identity resolution If one of those is essential to you, that is the reason to keep paying for Piwik PRO, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Piwik PRO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party analytics collector for one site whose privacy claims are enforced by tests rather than asserted in a policy page to replace Piwik PRO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero An analytics tool that sets a tracker without asking is either exempt from consent or unlawful, and the conditions for exemption are published rather than a matter of taste. The French regulator's guidance on using analytics sets them out plainly: inform people the trackers exist, give them the ability to object, keep the purpose to audience measurement and A/B testing only, do not cross-check the data with any other processing such as a customer file or visit statistics from other sites, keep the tracker scoped to a single site or application publisher, truncate the last byte of the IP address, and limit the lifetime of the trackers to thirteen months. The same guidance notes that most large audience measurement offerings do not fall within the exemption whatever their configuration. So build to that list and treat every item as a test rather than a sentence in a policy. Add an identifier that follows somebody between sites, a join against a customer table, or a profile outliving thirteen months, and you have a product that needs a consent banner, which is not something you bolt on afterwards. Regulators differ and the text moves, so read your own regulator's current position before you ship. And no session replay: capturing what somebody typed before they sent it is a different product and a different risk. ### Stack (fixed, do not substitute) - Collector: A small first-party endpoint on your own host, POST only · a third-party endpoint is the thing the exemption conditions rule out - Client: A hand-written script that sends on visibilitychange using the beacon API · the unload event is not fired reliably and some browsers have stopped firing it at all - Storage: SQLite for a small site, with raw events and daily rollups as separate tables · the reports read rollups so the raw rows can be deleted on schedule without losing history - Identity: A daily salted hash of coarse request attributes, with the salt destroyed after its window · you need to count visits, and that is a different requirement from knowing who somebody is - Reporting: Pre-aggregated daily rollups only, with no query path into the raw table · if a report cannot be answered from an aggregate it is a report this build does not offer ### Data model (create this before Phase 1) - sites(id, host, created_at, salt_rotation_hour) · exactly one publisher, because the exemption conditions say so - events(id, site_id, occurred_at, received_at, name, path, referrer_host, country, device_class, visit_key, props_json) · no full IP address, no cookie identifier, no column that identifies a person - salts(day, value, created_at, destroyed_at) · one per day, destroyed after its window, which is what makes yesterday's visit keys unlinkable to today's - rollups_daily(site_id, day, path, name, count, visits) · everything the reports read comes from here - retention(site_id, raw_days, rollup_days) · enforced by a job that actually deletes, and tested at the boundary - bots(pattern, kind, source, added_on) · a list you maintain by hand, with where each entry came from - deletion_requests(id, received_at, scope, completed_at, rows_affected) · because somebody will ask and the answer needs a record - There is no users table and no identifier that survives the salt rotation. The schema is the privacy claim; the policy page is only a description of it. ### Phase 1 · A collector that fires when the page actually goes away Build: Send with the beacon API from a visibilitychange handler when the document becomes hidden, which MDN describes as the last event that is reliably fired and therefore the best place to send analytics data. Do not use unload: MDN states it is not reliably fired especially on mobile, that it blocks the back forward cache, and that Chrome has stopped firing it by default. Truncate the last byte of the address at the edge before anything is written. Trap: Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. Done when: Events are sent from a visibilitychange handler and no unload or pagehide listener exists anywhere in the script, verified by a test that greps the built bundle. The beacon call's return value is checked and a false return falls back to a smaller batch rather than dropping the data silently, tested by exceeding the documented 64 KiB limit. No full IP address is written to storage or to any log, tested by a fixture request and an assertion over both. The script is served first-party from the same host as the site, and the collector rejects requests whose origin is not the configured host. Do not build yet: No visit counting, no reports, no retention, no bot handling yet. ### Phase 2 · Counting visits without an identifier Build: Derive the visit key as a hash of the day's secret salt together with the truncated address, the user agent string and the site identifier. Rotate the salt daily at a configured hour and destroy the previous value rather than archiving it. Sessionise by grouping events sharing a visit key within a configured gap. Store nothing on the visitor's device. Trap: Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. Done when: Destroying a day's salt is irreversible and the previous value is absent from the database, the backups and the logs, tested by inspecting all three after a rotation. Nothing is written to the visitor's device, verified by a browser test asserting empty cookie, local and session storage after a full session. Daily unique counts are computed into the rollup before the salt is destroyed, tested at the rotation boundary. No report offers a cross-day unique visitor figure and the interface explains why in one sentence. Do not build yet: No reports, no retention job, no bot filtering yet. ### Phase 3 · Reports that survive the raw rows being deleted Build: Roll up daily per site, path and event name with counts and visits. Point every report at the rollup table and give the reporting layer no access path to the raw events at all. Run the retention job on a schedule so raw rows older than the configured window are deleted and rollups older than their own window follow. Show the retention windows on the reports page. Trap: A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. Done when: Deleting every raw event older than the window leaves all reports unchanged, tested by running retention and re-rendering. The reporting layer has no query path to the events table, enforced by a separate read-only connection whose grants exclude it and asserted in a test. Retention deletes at the configured boundary and the deletion is counted and logged without row contents, tested. Both retention windows are displayed on the reports page. Do not build yet: No bot filtering, no deletion requests, no export yet. ### Phase 4 · Traffic that is not people Build: Maintain the bots table by hand from a published list and record where each entry came from and when. Exclude matches at ingest but keep a count of what was excluded so the exclusion rate is visible. Exclude your own traffic with an explicit opt-out you set on your own devices. Show excluded counts beside every report figure rather than hiding them. Trap: Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. Done when: Every bots row records its source and the date it was added, enforced by a not-null constraint. Excluded events are counted per rule and the counts appear beside the report figures, checked in the interface test. A pattern matching an ordinary browser user agent fails a test containing a fixture of common real agents. Your own opt-out works without a cookie, tested, since setting a cookie to avoid tracking is its own small joke. Do not build yet: No deletion requests, no export, no operations work yet. ### Phase 5 · Proving the claims, and getting the data out Build: Write one test per condition from rule zero and name each test after the condition, so the suite reads as the compliance argument. Add a deletion path that takes a scope, deletes matching rows, records the count and cannot be made to print row contents. Export rollups and remaining raw events as CSV. Publish the privacy text from the same constants the code uses, so the two cannot drift apart. Trap: A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads. Done when: One named test exists per exemption condition and the suite fails if any is removed, checked by deleting one deliberately. The privacy text renders its numbers from the same configuration the collector and retention job read, tested by changing a value and asserting the page changes. A deletion request by scope deletes the matching rows, records the count, and logs nothing about their contents, tested. Export produces rollups and raw events as CSV and reimports into an empty database with identical report output. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay and form field capture, which record what somebody typed before they chose to send it and are a different product with a different legal footing. - Cross-site or cross-application identifiers of any kind, because the exemption conditions rule them out and adding one turns this into a product that needs a banner. - Joining analytics to a customer table, which the same conditions explicitly name as cross-checking with other processing. - Advertising attribution and conversion modelling, which exist to follow somebody between sites and cannot be built inside these constraints honestly. ### README must contain - The exemption conditions this build is designed against, quoted with the regulator and the date they were read, and the instruction to check them again before shipping - That identity is a daily salted hash whose salt is destroyed, so a cross-day unique visitor number cannot be produced by design - That reports read only daily rollups, which is what lets raw events be deleted on schedule - That the collector sends on visibilitychange and why unload is not used - How to answer a deletion request and what the record of one looks like
AGENTS.md
# Agent instructions · Piwik PRO indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A small first-party endpoint on your own host, POST only, A hand-written script that sends on visibilitychange using the beacon API, SQLite for a small site, with raw events and daily rollups as separate tables, A daily salted hash of coarse request attributes, with the salt destroyed after its window, Pre-aggregated daily rollups only, with no query path into the raw table. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic.
BUILD_PLAN.md
# Build plan · Piwik PRO A first-party collector for one site whose hard problems are firing at the moment a page actually goes away rather than at an event browsers no longer honour, counting visits without an identifier that outlives a day, and keeping reports answerable from aggregates after the raw rows have been deleted. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A collector that fires when the page actually goes away An event is recorded when somebody leaves, including on a phone, and the request never carries a full IP address into storage. ### Steps 1. Send on visibilitychange, never on unload It is the only page-exit signal that is reliably fired. Files: `src/client/collect.ts`, `tests/no-unload.test.ts` 2. Check the beacon return value A false return is a silently lost batch and the cap is 64 KiB. Files: `src/client/collect.ts`, `tests/beacon-limit.test.ts` 3. Truncate the address at the edge Before storage, before logs, before anything keeps a copy. Files: `src/server/ingest.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] No unload or pagehide listener exists in the bundle - [ ] A false beacon return falls back rather than dropping data - [ ] No full IP address reaches storage or logs - [ ] The collector refuses requests from other origins ### Watch out - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. ## Phase 2 · Counting visits without an identifier Visits are countable for a day and become uncountable afterwards, by construction. ### Steps 1. Rotate and destroy the salt on a schedule An archived salt is a permanent identifier wearing a disguise. Files: `src/server/salt.ts`, `tests/salt-destroyed.test.ts` 2. Compute daily uniques before destroying it Afterwards the number is genuinely unrecoverable. Files: `src/server/rollup.ts` 3. Store nothing on the device Asserted in a browser test, not in a policy page. Files: `src/client/collect.ts`, `tests/no-storage.test.ts` ### Done when - [ ] A destroyed salt is absent from database, backups and logs - [ ] Nothing is written to cookie or browser storage - [ ] Daily uniques are computed before rotation - [ ] No cross-day unique figure is offered ### Watch out - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. ## Phase 3 · Reports that survive the raw rows being deleted Every report reads a rollup, so retention can delete raw events without deleting history. ### Steps 1. Roll up daily and report only from rollups Reports that read raw data cannot survive deletion. Files: `src/server/rollup.ts`, `src/app/reports/page.tsx` 2. Give reporting a connection that cannot see raw events A rule enforced by grants is a rule that holds. Files: `src/server/db.ts`, `tests/reporting-isolation.test.ts` 3. Run retention on a schedule and show the windows Deletion should be boring and visible. Files: `src/jobs/retention.ts`, `tests/retention-boundary.test.ts` ### Done when - [ ] Reports are unchanged after raw deletion - [ ] Reporting cannot query the events table - [ ] Retention deletes at the configured boundary - [ ] Retention windows appear on the reports page ### Watch out - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. ## Phase 4 · Traffic that is not people Automated traffic and your own visits are excluded with a rule you can point at, not a hunch. ### Steps 1. Record the source and date for every rule A pattern nobody can trace is a pattern nobody can fix. Files: `migrations/004_bots.sql`, `src/server/bots.ts` 2. Count exclusions per rule and show them An invisible filter is an unfalsifiable number. Files: `src/server/ingest.ts`, `src/app/reports/page.tsx` 3. Test rules against real user agents Overly broad patterns are the failure mode here. Files: `tests/bot-patterns.test.ts` ### Done when - [ ] Every bot rule records its source and date - [ ] Exclusion counts appear beside report figures - [ ] No rule matches a common real user agent - [ ] The self opt-out works without a cookie ### Watch out - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. ## Not in this build - Session replay and form field capture. - Cross-site or cross-application identifiers. - Joins against a customer table. - Advertising attribution and conversion modelling. ## After v1, if you want it - A second site on the same deployment, which is only safe once the per-publisher isolation is enforced by grants rather than by a where clause - Server-side event ingestion from your own backend, which avoids the browser entirely for events that do not need it - A published methodology page describing exactly how a visit is counted, which is the number people argue about
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Events, rollups and salts. Back up the rollups; the raw events are transient by design. DATABASE_PATH=./data/analytics.db # Required. The single publisher this deployment measures. The collector rejects other origins. SITE_HOST=example.com # Required. Local hour at which the day's salt is destroyed after uniques are rolled up. SALT_ROTATION_HOUR=3 # Required. Raw events are deleted after this. Reports do not depend on them. RAW_RETENTION_DAYS=30 # Required. Aggregates kept for roughly thirteen months, matching the tracker lifetime you claim. ROLLUP_RETENTION_DAYS=395 # Required. Events sharing a visit key within this gap are one visit. Stated on the reports page. SESSION_GAP_MINUTES=30 # Required · secret. The only credential for reports and deletion. There are no accounts. ADMIN_TOKEN=redacted
PRODUCT.md
# Piwik PRO · product brief ## Problem The visible privacy analytics suite loop is buildable, but a credible replacement needs more than the first screen. Piwik PRO earns its keep through ingestion, storage, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A collector for one site that records the page exits browsers actually report, counts visits with a key that stops existing at the end of the day, answers every report from aggregates so raw rows can be deleted on schedule, and has one named test for each condition the privacy claim rests on. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - deploy target - ClickHouse or SQLite for low volume - first-party tracking script ## Explicit non-goals for v1 - Session replay and form field capture. - Cross-site or cross-application identifiers. - Joins against a customer table. - Advertising attribution and conversion modelling. - session replay privacy tooling - enterprise governance and integrations - high-volume ingestion and retention - bot and identity resolution ## Success criteria - No unload listener is confirmed present in the bundle - No full IP address is confirmed to reach storage or logs - A destroyed salt is confirmed absent from database, backups and logs - Reports are confirmed unchanged after raw deletion - One named test per exemption condition is confirmed present
BRIEF.md
# Build brief · Piwik PRO The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a first-party analytics collector for one site whose privacy claims are enforced by tests rather than asserted in a policy page to replace Piwik PRO. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero An analytics tool that sets a tracker without asking is either exempt from consent or unlawful, and the conditions for exemption are published rather than a matter of taste. The French regulator's guidance on using analytics sets them out plainly: inform people the trackers exist, give them the ability to object, keep the purpose to audience measurement and A/B testing only, do not cross-check the data with any other processing such as a customer file or visit statistics from other sites, keep the tracker scoped to a single site or application publisher, truncate the last byte of the IP address, and limit the lifetime of the trackers to thirteen months. The same guidance notes that most large audience measurement offerings do not fall within the exemption whatever their configuration. So build to that list and treat every item as a test rather than a sentence in a policy. Add an identifier that follows somebody between sites, a join against a customer table, or a profile outliving thirteen months, and you have a product that needs a consent banner, which is not something you bolt on afterwards. Regulators differ and the text moves, so read your own regulator's current position before you ship. And no session replay: capturing what somebody typed before they sent it is a different product and a different risk. ### Stack (fixed, do not substitute) - Collector: A small first-party endpoint on your own host, POST only · a third-party endpoint is the thing the exemption conditions rule out - Client: A hand-written script that sends on visibilitychange using the beacon API · the unload event is not fired reliably and some browsers have stopped firing it at all - Storage: SQLite for a small site, with raw events and daily rollups as separate tables · the reports read rollups so the raw rows can be deleted on schedule without losing history - Identity: A daily salted hash of coarse request attributes, with the salt destroyed after its window · you need to count visits, and that is a different requirement from knowing who somebody is - Reporting: Pre-aggregated daily rollups only, with no query path into the raw table · if a report cannot be answered from an aggregate it is a report this build does not offer ### Data model (create this before Phase 1) - sites(id, host, created_at, salt_rotation_hour) · exactly one publisher, because the exemption conditions say so - events(id, site_id, occurred_at, received_at, name, path, referrer_host, country, device_class, visit_key, props_json) · no full IP address, no cookie identifier, no column that identifies a person - salts(day, value, created_at, destroyed_at) · one per day, destroyed after its window, which is what makes yesterday's visit keys unlinkable to today's - rollups_daily(site_id, day, path, name, count, visits) · everything the reports read comes from here - retention(site_id, raw_days, rollup_days) · enforced by a job that actually deletes, and tested at the boundary - bots(pattern, kind, source, added_on) · a list you maintain by hand, with where each entry came from - deletion_requests(id, received_at, scope, completed_at, rows_affected) · because somebody will ask and the answer needs a record - There is no users table and no identifier that survives the salt rotation. The schema is the privacy claim; the policy page is only a description of it. ### Phase 1 · A collector that fires when the page actually goes away Build: Send with the beacon API from a visibilitychange handler when the document becomes hidden, which MDN describes as the last event that is reliably fired and therefore the best place to send analytics data. Do not use unload: MDN states it is not reliably fired especially on mobile, that it blocks the back forward cache, and that Chrome has stopped firing it by default. Truncate the last byte of the address at the edge before anything is written. Trap: Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. Done when: Events are sent from a visibilitychange handler and no unload or pagehide listener exists anywhere in the script, verified by a test that greps the built bundle. The beacon call's return value is checked and a false return falls back to a smaller batch rather than dropping the data silently, tested by exceeding the documented 64 KiB limit. No full IP address is written to storage or to any log, tested by a fixture request and an assertion over both. The script is served first-party from the same host as the site, and the collector rejects requests whose origin is not the configured host. Do not build yet: No visit counting, no reports, no retention, no bot handling yet. ### Phase 2 · Counting visits without an identifier Build: Derive the visit key as a hash of the day's secret salt together with the truncated address, the user agent string and the site identifier. Rotate the salt daily at a configured hour and destroy the previous value rather than archiving it. Sessionise by grouping events sharing a visit key within a configured gap. Store nothing on the visitor's device. Trap: Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. Done when: Destroying a day's salt is irreversible and the previous value is absent from the database, the backups and the logs, tested by inspecting all three after a rotation. Nothing is written to the visitor's device, verified by a browser test asserting empty cookie, local and session storage after a full session. Daily unique counts are computed into the rollup before the salt is destroyed, tested at the rotation boundary. No report offers a cross-day unique visitor figure and the interface explains why in one sentence. Do not build yet: No reports, no retention job, no bot filtering yet. ### Phase 3 · Reports that survive the raw rows being deleted Build: Roll up daily per site, path and event name with counts and visits. Point every report at the rollup table and give the reporting layer no access path to the raw events at all. Run the retention job on a schedule so raw rows older than the configured window are deleted and rollups older than their own window follow. Show the retention windows on the reports page. Trap: A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. Done when: Deleting every raw event older than the window leaves all reports unchanged, tested by running retention and re-rendering. The reporting layer has no query path to the events table, enforced by a separate read-only connection whose grants exclude it and asserted in a test. Retention deletes at the configured boundary and the deletion is counted and logged without row contents, tested. Both retention windows are displayed on the reports page. Do not build yet: No bot filtering, no deletion requests, no export yet. ### Phase 4 · Traffic that is not people Build: Maintain the bots table by hand from a published list and record where each entry came from and when. Exclude matches at ingest but keep a count of what was excluded so the exclusion rate is visible. Exclude your own traffic with an explicit opt-out you set on your own devices. Show excluded counts beside every report figure rather than hiding them. Trap: Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. Done when: Every bots row records its source and the date it was added, enforced by a not-null constraint. Excluded events are counted per rule and the counts appear beside the report figures, checked in the interface test. A pattern matching an ordinary browser user agent fails a test containing a fixture of common real agents. Your own opt-out works without a cookie, tested, since setting a cookie to avoid tracking is its own small joke. Do not build yet: No deletion requests, no export, no operations work yet. ### Phase 5 · Proving the claims, and getting the data out Build: Write one test per condition from rule zero and name each test after the condition, so the suite reads as the compliance argument. Add a deletion path that takes a scope, deletes matching rows, records the count and cannot be made to print row contents. Export rollups and remaining raw events as CSV. Publish the privacy text from the same constants the code uses, so the two cannot drift apart. Trap: A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads. Done when: One named test exists per exemption condition and the suite fails if any is removed, checked by deleting one deliberately. The privacy text renders its numbers from the same configuration the collector and retention job read, tested by changing a value and asserting the page changes. A deletion request by scope deletes the matching rows, records the count, and logs nothing about their contents, tested. Export produces rollups and raw events as CSV and reimports into an empty database with identical report output. Do not build yet: Nothing after this. ### Out of scope (and why) - Session replay and form field capture, which record what somebody typed before they chose to send it and are a different product with a different legal footing. - Cross-site or cross-application identifiers of any kind, because the exemption conditions rule them out and adding one turns this into a product that needs a banner. - Joining analytics to a customer table, which the same conditions explicitly name as cross-checking with other processing. - Advertising attribution and conversion modelling, which exist to follow somebody between sites and cannot be built inside these constraints honestly. ### README must contain - The exemption conditions this build is designed against, quoted with the regulator and the date they were read, and the instruction to check them again before shipping - That identity is a daily salted hash whose salt is destroyed, so a cross-day unique visitor number cannot be produced by design - That reports read only daily rollups, which is what lets raw events be deleted on schedule - That the collector sends on visibilitychange and why unload is not used - How to answer a deletion request and what the record of one looks like
ARCHITECTURE.md
# Architecture · Piwik PRO ## Stack | Part | Choice | Why | | --- | --- | --- | | Collector | A small first-party endpoint on your own host, POST only | a third-party endpoint is the thing the exemption conditions rule out | | Client | A hand-written script that sends on visibilitychange using the beacon API | the unload event is not fired reliably and some browsers have stopped firing it at all | | Storage | SQLite for a small site, with raw events and daily rollups as separate tables | the reports read rollups so the raw rows can be deleted on schedule without losing history | | Identity | A daily salted hash of coarse request attributes, with the salt destroyed after its window | you need to count visits, and that is a different requirement from knowing who somebody is | | Reporting | Pre-aggregated daily rollups only, with no query path into the raw table | if a report cannot be answered from an aggregate it is a report this build does not offer | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Client | Page-exit detection and beacon batching. | Any script, provided it never listens for unload and checks the beacon return value. | | Ingest | Address truncation, bot exclusion, event writes. | Any server, provided truncation happens before storage and before logs. | | Identity | Daily salt, visit keys and destruction. | Not swappable for a stored identifier, which is the whole point. | | Rollup | Daily aggregates and the reporting boundary. | Any aggregation, provided reporting cannot reach raw events. | | Retention | Scheduled deletion and deletion requests. | Any scheduler, provided deletion is counted and logged without contents. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Events, rollups and salts. Back up the rollups; the raw events are transient by design. - `SITE_HOST` · required · The single publisher this deployment measures. The collector rejects other origins. - `SALT_ROTATION_HOUR` · required · Local hour at which the day's salt is destroyed after uniques are rolled up. - `RAW_RETENTION_DAYS` · required · Raw events are deleted after this. Reports do not depend on them. - `ROLLUP_RETENTION_DAYS` · required · Aggregates kept for roughly thirteen months, matching the tracker lifetime you claim. - `SESSION_GAP_MINUTES` · required · Events sharing a visit key within this gap are one visit. Stated on the reports page. - `ADMIN_TOKEN` · required, secret · The only credential for reports and deletion. There are no accounts. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Piwik PRO product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A small first-party endpoint on your own host, POST only, A hand-written script that sends on visibilitychange using the beacon API, SQLite for a small site, with raw events and daily rollups as separate tables, A daily salted hash of coarse request attributes, with the salt destroyed after its window, Pre-aggregated daily rollups only, with no query path into the raw table. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. - A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads.
MILESTONES.md
# Delivery milestones · Piwik PRO Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A collector that fires when the page actually goes away An event is recorded when somebody leaves, including on a phone, and the request never carries a full IP address into storage. ### Steps 1. Send on visibilitychange, never on unload It is the only page-exit signal that is reliably fired. Files: `src/client/collect.ts`, `tests/no-unload.test.ts` 2. Check the beacon return value A false return is a silently lost batch and the cap is 64 KiB. Files: `src/client/collect.ts`, `tests/beacon-limit.test.ts` 3. Truncate the address at the edge Before storage, before logs, before anything keeps a copy. Files: `src/server/ingest.ts`, `tests/no-full-ip.test.ts` ### Done when - [ ] No unload or pagehide listener exists in the bundle - [ ] A false beacon return falls back rather than dropping data - [ ] No full IP address reaches storage or logs - [ ] The collector refuses requests from other origins ### Watch out - Sending on unload loses most mobile sessions outright, because a user who switches apps and later closes the browser from the app manager never fires it, and adding an unload listener also keeps the page out of the back forward cache and makes the site slower for everyone. Send on visibilitychange to hidden, and remember the beacon queue is capped at 64 KiB in total, so a batch that exceeds it returns false and is silently not sent unless you check the return value. ## M2 · Counting visits without an identifier Visits are countable for a day and become uncountable afterwards, by construction. ### Steps 1. Rotate and destroy the salt on a schedule An archived salt is a permanent identifier wearing a disguise. Files: `src/server/salt.ts`, `tests/salt-destroyed.test.ts` 2. Compute daily uniques before destroying it Afterwards the number is genuinely unrecoverable. Files: `src/server/rollup.ts` 3. Store nothing on the device Asserted in a browser test, not in a policy page. Files: `src/client/collect.ts`, `tests/no-storage.test.ts` ### Done when - [ ] A destroyed salt is absent from database, backups and logs - [ ] Nothing is written to cookie or browser storage - [ ] Daily uniques are computed before rotation - [ ] No cross-day unique figure is offered ### Watch out - Keeping the old salts so you can recompute historical uniques quietly turns a rotating hash back into a permanent identifier, because anyone holding the salt series can link a visitor across every day you kept. Destroy the salt when its window ends, compute the daily unique count into the rollup before you do, and accept that a cross-day unique visitor number is a figure this design cannot produce. ## M3 · Reports that survive the raw rows being deleted Every report reads a rollup, so retention can delete raw events without deleting history. ### Steps 1. Roll up daily and report only from rollups Reports that read raw data cannot survive deletion. Files: `src/server/rollup.ts`, `src/app/reports/page.tsx` 2. Give reporting a connection that cannot see raw events A rule enforced by grants is a rule that holds. Files: `src/server/db.ts`, `tests/reporting-isolation.test.ts` 3. Run retention on a schedule and show the windows Deletion should be boring and visible. Files: `src/jobs/retention.ts`, `tests/retention-boundary.test.ts` ### Done when - [ ] Reports are unchanged after raw deletion - [ ] Reporting cannot query the events table - [ ] Retention deletes at the configured boundary - [ ] Retention windows appear on the reports page ### Watch out - A report that queries the raw table works beautifully for two months and then goes blank the first time retention runs, and the usual response is to raise the retention window until the privacy claim is a fiction. Build the rollup first, forbid the reporting layer from reaching the raw table, and let deletion be the routine event it should be. ## M4 · Traffic that is not people Automated traffic and your own visits are excluded with a rule you can point at, not a hunch. ### Steps 1. Record the source and date for every rule A pattern nobody can trace is a pattern nobody can fix. Files: `migrations/004_bots.sql`, `src/server/bots.ts` 2. Count exclusions per rule and show them An invisible filter is an unfalsifiable number. Files: `src/server/ingest.ts`, `src/app/reports/page.tsx` 3. Test rules against real user agents Overly broad patterns are the failure mode here. Files: `tests/bot-patterns.test.ts` ### Done when - [ ] Every bot rule records its source and date - [ ] Exclusion counts appear beside report figures - [ ] No rule matches a common real user agent - [ ] The self opt-out works without a cookie ### Watch out - Silently dropping anything that looks automated means the day a real browser gets matched by an overly broad pattern you lose traffic you will never know about, and the numbers quietly drift. Count exclusions by rule, display the totals, and treat a rule whose share suddenly jumps as a bug in the rule rather than a change in the traffic. ## M5 · Proving the claims, and getting the data out (production only) Each exemption condition is asserted by a test, deletion requests are answerable, and everything exports. ### Steps 1. Name one test per exemption condition The test suite is the argument you would actually make. Files: `tests/conditions/*.test.ts` 2. Render the privacy numbers from configuration Prose and code drift; shared constants do not. Files: `src/app/privacy/page.tsx`, `src/config.ts` 3. Make deletion a recorded, countable operation Somebody will ask, and the answer needs a row. Files: `src/server/deletion.ts`, `tests/deletion.test.ts` ### Done when - [ ] One named test per exemption condition, and removal fails the suite - [ ] The privacy page renders numbers from configuration - [ ] Deletion records counts and never logs contents - [ ] Export and reimport reproduce identical reports ### Watch out - A privacy page written by hand drifts away from the code within a month, and the first person who checks will find the page claims a shorter retention than the job enforces, which is worse than having no page. Render the retention windows, the tracker lifetime and the truncation rule into the page from the same configuration the collector reads.
OPERATIONS.md
# Operations · Piwik PRO ## Backup Rollups nightly. Raw events deliberately not backed up beyond the retention window, because a backup that outlives the retention claim makes the claim untrue, and this should be written down where the backup script lives. ## Restore Restore rollups and confirm the report output matches the last exported CSV. Salts are never restored; a restored salt would resurrect an identifier that was supposed to be gone. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Exclusion rate per bot rule, beacon batches rejected for size, the age of the oldest raw event against the retention window, and whether yesterday's salt was actually destroyed. The last one is the check that matters most and the one most likely to fail silently. ## Incident checklist The incident in this build is a privacy claim that turned out to be false: a salt that was archived, an address that reached a log, a backup that outlived its window. Each has a named test, and the incident response is to find out why the test did not run. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] No unload listener is confirmed present in the bundle - [ ] No full IP address is confirmed to reach storage or logs - [ ] A destroyed salt is confirmed absent from database, backups and logs - [ ] Reports are confirmed unchanged after raw deletion - [ ] One named test per exemption condition is confirmed present ## Launch constraint Do not market omitted Piwik PRO capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Events, rollups and salts. Back up the rollups; the raw events are transient by design. DATABASE_PATH=./data/analytics.db # Required. The single publisher this deployment measures. The collector rejects other origins. SITE_HOST=example.com # Required. Local hour at which the day's salt is destroyed after uniques are rolled up. SALT_ROTATION_HOUR=3 # Required. Raw events are deleted after this. Reports do not depend on them. RAW_RETENTION_DAYS=30 # Required. Aggregates kept for roughly thirteen months, matching the tracker lifetime you claim. ROLLUP_RETENTION_DAYS=395 # Required. Events sharing a visit key within this gap are one visit. Stated on the reports page. SESSION_GAP_MINUTES=30 # Required · secret. The only credential for reports and deletion. There are no accounts. ADMIN_TOKEN=redacted
$ choose a build depth, inspect the files, then open the complete pack in your agent
Piwik PRO: Customers pay for trusted numbers, retention, privacy controls, and a pipeline that remains accurate while traffic and schemas change.
xsession replay privacy tooling
xenterprise governance and integrations
xhigh-volume ingestion and retention
xbot and identity resolution
share on X ↗"I just replaced Piwik PRO with one prompt"
Piwik PRO pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| business | $41.54/workspace | — | Starts below a maximum of 2,000,000 actions/month across 1-20 domains; 300 Data Activation profiles/month, 25 months retention, 50 custom reports per website, and up to 5,000 CMP-scanned subpages |
| data fundamentals | — | $422.36/workspace | 1-100 million actions/month, 1,000 Data Activation profiles/month, 5 CMP domains, 75 custom reports per website, and 2-hour real-time lookback |
| trusted insights | custom | — | 1-500 million actions/month, 4,000 Data Activation profiles/month, 10 CMP domains, 150 custom reports per website, and 12-hour real-time lookback |
| secure intelligence | custom | — | Starts at 2 million actions/month with no upper limit, unlimited Data Activation profiles, 15 CMP domains, unlimited custom reports, and 24-hour real-time lookback |
free tierno free tier
billingBusiness: monthly or annual in advance; Enterprise: annual contract; 30-day no-card Business trial; auto-renews
hidden costsBusiness is automatically moved to a higher paid configuration after exceeding actions/domains; premium services, exports, unlimited activation, extra retention, training, implementation and onboarding are add-ons. Subscription fees automatically rise 4% on every anniversary, and bank-transfer customers pay transaction fees.
verified 2026-08-12 · source ↗
Vibecode Piwik PRO
Kinda. The core of Piwik PRO is buildable in a weekend with the prompt on this page, but there are real gaps: session replay privacy tooling, enterprise governance and integrations. Read the honest list above before committing.
How much does Piwik PRO cost?
Piwik PRO's pricing is usage-based or varies by plan · The pricing model is not safely comparable as one fixed monthly USD amount. Review the official source before publication..
What do I lose by replacing Piwik PRO?
Honestly: session replay privacy tooling; enterprise governance and integrations; high-volume ingestion and retention; bot and identity resolution. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Piwik PRO?
Yes: Plausible Analytics (Open-source privacy-focused web analytics.), Umami (Lightweight self-hosted web analytics.). Using prior art is also vibecoding; the prompt is for when you want it exactly your way.