Build Cleanvoice AI

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A podcast editor where you delete words in a transcript and the audio follows. The engineering is in cutting audio at points that do not click, keeping several recorded tracks in sync when one is cut, and never destroying the original recording.

estimated effort multi-daythe files for this build are in the project pack

Transcriptionwhisper.cpp with word-level timings, run locallyAudioffmpeg for decoding and rendering, with sample-accurate cut pointsEditingA non-destructive edit list over untouched source filesMultitrackCuts applied across all tracks togetherRuntimePython 3.12 orchestrating them

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Orchestrates transcription, detection and rendering.
Get it python.org or your package manager, then a virtual environment.
Verify python3 --version prints 3.12 or higher
installfree
Why Decoding, sample-accurate cutting and rendering. Reimplementing any of it is not the project.
Get it Your package manager, and confirm it decodes the formats your recorder produces. open ↗
Verify It decodes your recordings and reports exact durations
installfree, but transcription takes real machine time
Why Word-level timings are the foundation of every edit position, and segment-level timings are not precise enough.
Get it Build whisper.cpp and choose a model sized for your machine. Confirm it emits word timings rather than only segments. open ↗
Verify A five minute recording produces word-level timings
have readyfree if you record one
Why Every hard problem here appears only in real material: coarticulated fillers, tracks starting at different times, clock drift and crosstalk between microphones.
Get it A recording with at least two separate microphone tracks and an hour of conversation. Your own is best, since you may edit it freely.
Verify At least two tracks of an hour, with the tracks not starting at exactly the same instant
have readyfree if you own a pair
Why Every claim about a cut being clean is verified by listening, and laptop speakers hide clicks entirely.
Get it Any closed headphones you can listen critically on, in a quiet room.
Verify You can hear a deliberately badly placed cut as a click

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- episodes(id, title, sample_rate, duration_ms, created_at)
- tracks(id, episode_id, path, speaker, offset_ms) · one per microphone, with its alignment offset
- words(id, track_id, ordinal, text, start_ms, end_ms, confidence)
- detections(id, track_id, kind, start_ms, end_ms, confidence, accepted) · kind is filler, silence, breath or mouth_click
- edits(id, episode_id, kind, start_ms, end_ms, applied_at, undone_at) · the edit list, never applied to source
- renders(id, episode_id, edit_revision, path, at)
- Source files are opened read only. Everything else is a list of intentions over them.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`SOURCE_DIR`	required	`./sources`	Recordings, opened read only. Nothing in the build writes here.
`RENDER_DIR`	required	`./renders`	Output episodes. Always derivable from sources and the edit list.
`WHISPER_MODEL`	required	`./models/ggml-medium.en.bin`	Larger models give better timings and take longer. Timing quality matters more than word accuracy here.
`CROSSFADE_MS`	required	`8`	The join length at a cut. Too short clicks, too long smears the speech.
`CUTPOINT_SEARCH_MS`	required	`40`	How far around a word boundary to look for a low-energy point.
`LOUDNESS_TARGET_LUFS`	required	`-16`	The normalisation target. State it rather than just making the file louder.

The build, in order

1
Words with times you can cut on
A transcript whose word timings are accurate enough to edit by.
1. Word level, not segment level. Segment timings are not precise enough to cut on.
  Files src/transcribe.py
2. Verify the word count and final timestamp against the whole file.
  Files src/chunk.pytests/test_chunk.py
3. Clicking a word must play from that point, or you cannot trust the timings.
  Files src/align.py
done when · tick each as it passes
watch out
- Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file.
next phase
2
Finding what to remove
Fillers, long silences and mouth sounds are found and proposed.
1. With their word timings, so the cut point comes from the same source.
  Files src/detect_filler.py
2. Silence by level and duration, mouth sounds by short high-frequency energy.
  Files src/detect_audio.py
3. Bulk review by kind and confidence is what makes this usable without being reckless.
  Files src/detections.py
done when · tick each as it passes
watch out
- Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review.
next phase
3
Cuts that do not click
Removing a word leaves a join you cannot hear.
1. Near the boundary, not on it. This is the whole difference between clean and clicking.
  Files src/cutpoint.pytests/test_cutpoint.py
2. Short enough to be inaudible, long enough to remove the discontinuity.
  Files src/cut.py
3. Better to leave a filler in than to leave a click.
  Files src/cut.py
done when · tick each as it passes
watch out
- Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds.
next phase
4
Several microphones, one timeline
Cutting one speaker does not desynchronise the others.
1. Recording software rarely starts every track at the same instant.
  Files src/align_tracks.py
2. Detection is per track. Cutting is not.
  Files src/timeline.pytests/test_sync.py
3. Two devices with different clocks drift. Silently misaligning is the worst outcome.
  Files src/drift.py
done when · tick each as it passes
watch out
- Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track.
5
Rendering, and never touching the originalproduct builder
A finished episode renders, and the source recordings are provably untouched.
1. Sources plus intentions. The render is always derivable and never authoritative.
  Files src/render.py
2. In the suite. A write to a source is an irreversible loss.
  Files tests/test_sources_untouched.py
3. State the target rather than making the file merely louder.
  Files src/loudness.py
done when · tick each as it passes
watch out
- Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ.

what this build does not replace

Real-time processing during recording.
Commercial-quality voice isolation and noise removal.
Music beds, chapters and show notes.
Speaker diarisation on a single mixed track.
specialized cleanup models, cloud processing, and batch speed
remote studio reliability
licensed music libraries
hosting distribution
advanced mastering and support

after v1, if you want it

Crosstalk gating, so one speaker's microphone stops carrying the other
A tightening pass that shortens rather than removes pauses, which sounds more natural
Per-speaker levelling before the mix, once the timeline is trustworthy

Need the files? The project pack on the verdict page hands your agent the whole brief · more podcasting.