Build MacWhisper

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A dictation tool where the whole engineering problem is that the best local speech model is not a streaming model. It processes fixed windows, so live dictation means gating on speech activity, chunking with overlap, and reconciling text that the model revises as it hears more.

estimated effort multi-daythe files for this build are in the project pack

Recognitionwhisper.cpp with a model sized to the machineGatingVoice activity detection ahead of the modelChunkingOverlapping windows reconciled on their overlapInsertionTyping into the focused application, with a review bufferRuntimeA native application with a global hotkey

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

have readyfree if you own one
Why Model size determines both accuracy and whether live dictation keeps up, and a machine that swaps will produce unusable latency.
Get it Check available memory and choose a model that fits comfortably alongside your normal work.
Verify A five minute file transcribes without the machine swapping
installfree, and models are a large download
Why The recognition engine, and the source of every timing and confidence the build depends on.
Get it Build whisper.cpp and download a model. Recent versions include voice activity detection support, which phase 2 depends on. open ↗
Verify A short file transcribes with word timings and the build includes voice activity detection
installfree
Why A global hotkey, accessibility insertion and audio capture are all native work.
Get it Install from the App Store, then the command line tools.
Verify A trivial application builds and registers a global hotkey
have readyfree
Why Phase 2 is tuned against real background noise, and a quiet room is much easier than a real one with a fan, typing and a conversation next door.
Get it Record thirty seconds of your workspace in silence, and thirty seconds while typing.
Verify Both recordings exist and the detector gates on both
have readyfree
Why Vocabulary correction is only testable against terms that actually fail, and the ones that fail are specific to your work.
Get it Dictate a few paragraphs of your own writing and note every proper noun and technical term that comes out wrong.
Verify At least fifteen terms are listed, including one that appears as a substring of a common word

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- sessions(id, started_at, ended_at, mode, target_app) · mode is dictation or file transcription
- segments(id, session_id, ordinal, start_ms, end_ms, text, confidence, revised_from) · revised_from records that the model changed its mind
- vocabulary(id, phrase, replacement, scope) · names and terms the model reliably gets wrong
- files(id, path, duration_ms, model, transcribed_at, output_path) · batch transcription
- settings(key, value) · hotkey, model, insertion behaviour
- A segment can be superseded rather than only appended, because a streaming approximation over a non-streaming model produces revisions.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`MODEL_PATH`	required	`~/Library/Application Support/dictation/ggml-medium.en.bin`	The recognition model. Larger is more accurate and slower, which matters for live dictation.
`DICTATION_HOTKEY`	required	`Fn+Fn`	The global hotkey. Choose something nothing else claims, which is harder than it sounds.
`VAD_THRESHOLD`	required	`0.5`	Speech activity sensitivity. Too high loses quiet speech, too low lets the model invent.
`COMMIT_STABLE_MS`	required	`600`	How long text must stay unchanged before committing. Shorter feels faster and rewrites more.
`VOCABULARY_PATH`	required	`./config/vocabulary.txt`	Names and terms with replacements. Applied on word boundaries after recognition.
`ALLOW_NETWORK`	required	`model_download_only`	Present as a constant so the answer is visible in configuration. Nothing else may use the network.

The build, in order

1
Transcribing a file correctly
A long recording transcribes with nothing lost at the window seams.
1. The window is fixed. The seams are where the words go missing.
  Files src/chunk.swifttests/ChunkTests.swift
2. A shorter file transcribed whole is the reference. Compare word counts.
  Files tests/SeamTests.swift
3. An hour of audio is minutes of work. Silence during it feels broken.
  Files src/Transcriber.swift
done when · tick each as it passes
watch out
- The model processes fixed-length windows, so chunking without overlap drops words at every boundary and chunking with naive overlap duplicates them. Merge by matching text across the overlap, and verify the word count and final timestamp against a single-pass transcription of a shorter file.
next phase
2
Not letting the model talk to itself
Silence produces silence rather than invented sentences.
1. Nothing reaches the model without speech in it.
  Files src/VAD.swift
2. A fan, typing, a room. Synthetic silence is much easier than real quiet.
  Files tests/NoiseTests.swift
3. A detector that never gates is not working, and you will not notice otherwise.
  Files src/VAD.swift
done when · tick each as it passes
watch out
- Fed a window of silence or background noise, the model will confidently produce a plausible sentence, and in dictation that means text appearing in your document when nobody spoke. This is not an efficiency optimisation, it is the thing that makes the feature usable at all.
next phase
3
Live dictation, with the model changing its mind
Words appear as you speak and corrections are handled gracefully.
1. Stability over a window is the commit condition, not the model returning.
  Files src/Live.swift
2. Once it has left for the target application it is the user's problem to edit.
  Files src/Live.swift
3. A number, logged. It is what the feature is judged on.
  Files src/Metrics.swift
done when · tick each as it passes
watch out
- Committing every recognition result immediately produces text that visibly rewrites itself as the model revises, which is unusable for dictation because you cannot type around it. Hold recent text as provisional, commit only once a window has passed without it changing, and never revise text already committed to the target application.
next phase
4
Typing into whatever is in front of you
Dictated text lands in the application you were already using.
1. Not at commit. Following focus types into the wrong window.
  Files src/Insert.swift
2. Some applications reject synthetic input. Losing the text is not acceptable.
  Files src/Buffer.swift
3. Accessibility access is broad. Say what it is used for.
  Files src/Permissions.swift
done when · tick each as it passes
watch out
- Inserting into whatever happens to be focused at the moment text commits will type into the wrong window when focus changes mid-sentence, which in the wrong application is anything from embarrassing to destructive. Capture the target at the start of dictation and stop rather than following focus.
5
Vocabulary, batch work and keeping it localproduct builder
Names come out right, files process unattended, and nothing leaves the machine.
1. Simple substitution corrupts words containing the term.
  Files src/Vocabulary.swifttests/VocabularyTests.swift
2. Timings must be correct in the subtitle format, which is where they usually are not.
  Files src/Batch.swift
3. The privacy claim is the reason to use this. Prove it rather than stating it.
  Files tests/OfflineTests.swift
done when · tick each as it passes
watch out
- Applying vocabulary replacements by simple string substitution will corrupt words that happen to contain the term, so replacing a short name mangles unrelated text throughout the transcript. Match on word boundaries with the surrounding context, and test against a transcript containing the term as a substring.

what this build does not replace

Speaker diarisation.
Real-time translation.
Cloud recognition as a fallback.
Custom model training on your own voice.
cross-call team analytics
meeting-bot auto-join
live multi-speaker accuracy
calendar and CRM integrations

after v1, if you want it

Commands mixed into dictation, such as spoken punctuation and new paragraph
Per-application settings, since dictating code and dictating prose want different handling
A correction pass that learns from what you edit afterwards

Need the files? The project pack on the verdict page hands your agent the whole brief · more dictation.