Build Bevel

KINDAreplaces $14.99/mosaves $179.88/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A local analytics dashboard over your own health export whose hard problems are ingesting a file too large to hold in memory, resolving two devices that both recorded the same day, and presenting each metric at the resolution the published validation supports rather than the resolution the file happens to have.

estimated effort weekendthe files for this build are in the project pack

RuntimePython 3.12 with a local web dashboard, nothing leaving the machineIngestAn incremental XML parser writing straight to SQLite in batchesStorageSQLite, one row per exported record, source and device retainedAnalysisA declared set of comparisons with intervals and sample sizesPresentationEvery figure carries its device, its sample size and its known agreement

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the importer, the analysis and the local dashboard.
Get it python.org, or your package manager. open ↗
Verify python3 --version prints 3.12 or higher
have readyfree, plus the wait while it generates
Why Everything here is built against the real file, and the real file is far larger and far messier than any sample you will find.
Get it From the health application on your phone, request a full export and transfer the archive to the machine running this.
Verify The archive extracts and the importer reports a record count and export date
have readyfree
Why The extracted export and the database together are several times the archive, and running out part way through an import is the most common first failure.
Get it Check free space against the archive size multiplied by a factor you measured on the first import.
Verify A full import completes and the measured total size is written in the README
decidefree
Why The published validation differs sharply by metric, and deciding presentation after the charts exist means deciding it from whatever looked best.
Get it Write down, per metric, whether it is shown as a value, a band or a baseline, and which published figure justifies that. open ↗
Verify The table is in the README and each rule is asserted in a test
decidefree
Why Declaring them afterwards turns a dashboard into a search for whatever matches the story you already had.
Get it List the metric pairs and lags you care about in the configuration file with today's date.
Verify The dashboard shows only declared pairs and prints the total comparison count

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- imports(id, at, zip_path, export_date, record_count, sources_json, state) · state is partial or complete, never assumed
- records(id, import_id, type, source_name, source_version, device, unit, value, start_at, end_at) · the raw grain, insert only
- sleep_segments(id, record_id, source_name, stage, start_at, end_at) · one row per interval exactly as exported
- nights(date, sleep_start, sleep_end, total_sleep_min, stage_minutes_json, primary_source, overlap_dropped_min, conflict)
- daily(date, metric, value, n_records, primary_source, conflict) · one chosen source per metric per day, and the choice is recorded
- baselines(metric, date, window_days, median, p25, p75, n) · what any single value is compared against
- comparisons(id, metric_a, metric_b, lag_days, window_days, n, coefficient, ci_low, ci_high, declared_at)
- There is no column for a composite score, and stage minutes are stored so they can be shown in bands rather than to the minute.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_PATH`	required	`./data/health.db`	Raw records and derived tables. Everything about your body; keep it local.
`EXTRACT_DIR`	required	`./data/extract`	Where the archive is unpacked. Removed by the delete command along with the archive.
`IMPORT_BATCH_SIZE`	required	`5000`	Rows per insert batch. Larger is faster and uses more memory.
`SOURCE_PREFERENCE`	required	`watch,phone`	Order used to pick one source per metric per day. Changing it recomputes derived tables.
`BASELINE_WINDOW_DAYS`	required	`28`	Rolling window any single value is compared against.
`MIN_COMPARISON_N`	required	`30`	Below this many paired days, a comparison is shown as not yet computable.
`STALE_AFTER_DAYS`	required	`14`	Data age at which the dashboard prompts for a fresh export.

The build, in order

1
An export that does not fit in memory
A multi-hundred-megabyte export imports within a stated memory ceiling and either completes or leaves nothing.
1. The file is bigger than the memory you want to spend.
  Files src/ingest/stream.pytests/test_memory.py
2. Duplicated health records cannot be spotted by eye later.
  Files src/ingest/dedupe.py
3. A half import is a dashboard quietly showing half a life.
  Files src/ingest/transaction.py
done when · tick each as it passes
watch out
- Loading the export with a document parser builds the whole file as objects in memory, takes every gigabyte the machine has and dies several minutes in, and the obvious fix of raising the limit only moves the moment of death. Parse incrementally, clear each element as you finish with it, and batch the inserts so the transaction log does not become the new problem.
- Re-importing after adding a metric duplicates every record you already had, and because health data is full of legitimately repeated values you cannot spot it afterwards by eye. Key records on their natural identity and make a re-import a no-op for anything already present, then prove it by importing the same file twice and comparing counts.
next phase
2
Two devices, one day, and the double count
Each metric on each day comes from one chosen source, and what was discarded is recorded.
1. Adding two devices together invents a day.
  Files src/resolve/sources.pytests/test_double_count.py
2. Two devices recording one night is not two nights.
  Files src/resolve/sleep.py
3. The raw import is the only thing you cannot re-derive.
  Files src/resolve/recompute.py
done when · tick each as it passes
watch out
- Summing every step or sleep record for a day adds the phone's count to the watch's and gives somebody who carried both a day that never happened, which is the single most common error in analysis of these exports. Choose one source per metric per day, write down the rule, and show the count of days where the sources disagreed rather than hiding the decision.
- Summing sleep intervals from two sources that both recorded the same night produces more sleep than there were hours, and a chart that occasionally exceeds twenty-four hours is a chart nobody trusts again. Merge intervals by union, store the overlap you removed, and assert that no night exceeds the elapsed wall clock between its start and end.
next phase
3
Sleep at the resolution the device actually has
Total sleep and timing lead, stages appear in bands with their published agreement attached.
1. They are the parts the devices are measured as detecting well.
  Files src/web/sleep.pysrc/web/templates/sleep.html
2. The number and its uncertainty travel together or not at all.
  Files src/analysis/stages.pytests/test_no_bare_minutes.py
3. Two noisy classifications do not make a trend.
  Files src/analysis/baseline.py
done when · tick each as it passes
watch out
- Rendering stage durations to the minute implies a measurement the device does not make: the published validation reports four-stage agreement between kappa 0.21 and 0.53 across six wearables and correctly classified deep sleep epochs between about 47 and 70 percent, so the deep sleep figure is close to a coin flip per epoch. Show bands, put the agreement figure beside the number, and remove every code path that can display a stage minute count on its own.
- Comparing tonight's stage minutes against last night's and drawing an arrow presents the difference between two noisy classifications as a change in the person. Compare against a rolling baseline with its spread, and require a stated minimum number of nights before any comparison is drawn at all.
next phase
4
Comparisons declared before they are computed
The correlations on the dashboard are the ones you chose in advance, reported with intervals and counts.
1. A hypothesis chosen after the data is not a hypothesis.
  Files config/comparisons.tomlsrc/analysis/declare.py
2. The coefficient alone hides how little is behind it.
  Files src/analysis/correlate.pytests/test_intervals.py
3. It is the context that makes one result readable.
  Files src/web/templates/insights.html
done when · tick each as it passes
watch out
- A panel that tries every metric against every other one over a month will find several convincing relationships, because that is what running forty tests does, and the one you remember is the one that matched a story you already had. Fix the pairs in configuration with a declaration date, show the total number of comparisons run beside the results, and treat anything you add afterwards as a new declaration rather than a finding.
- Reporting a coefficient without its interval and sample size makes a relationship computed from eleven nights look like one computed from three hundred. Show the interval and the count everywhere the coefficient appears, and refuse to render below the minimum.
5
Data that goes stale, and saying soproduct builder
The dashboard is honest about how old the export is, everything exports, and deleting means deleting.
1. An axis that runs to today is a lie about the data.
  Files src/web/charts.py
2. The staleness is the honest headline of this whole build.
  Files src/web/templates/base.html
3. A deleted database next to an extracted export is not deleted.
  Files src/admin/delete.pytests/test_delete.py
done when · tick each as it passes
watch out
- Drawing today's date on the axis when the data ends eleven days ago makes a stale number look current, and the whole failure mode of this build is a dashboard that quietly stops being about now. Stop the axis at the export date, print the data age on every page, and show a prompt to re-export once the age crosses a configured threshold.
- Leaving the extracted export files beside the database means a delete that removes the database leaves a complete copy of your health record on disk, usually in a directory nobody remembers. Make deletion one command covering the database, the extracted files and the original archive, and print what it removed.

what this build does not replace

Live background synchronisation.
A composite readiness or recovery score.
Clinical interpretation of any kind.
Non-Apple device sources.
Live background sync; you are re-exporting a zip by hand
A phone app, widgets and notifications
Their opinionated composite scores and plain-English daily readouts
Non-Apple device integrations and whatever normalisation they do across sources
Charts that a designer looked at

after v1, if you want it

A per-metric page showing what the published validation says about that metric, which is the honest version of an explanation screen
Incremental import of successive exports so the history accumulates rather than being replaced each time
A single annual summary at the timescale the agreement figures actually support

Need the files? The project pack on the verdict page hands your agent the whole brief · more wellness.