Vibecode Cleanvoice AI
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Cleanvoice AI, remove filler words, mouth sounds, and long silences from a transcript-aligned edit list. The hard boundary is specialized cleanup models, cloud processing, and batch speed, plus audio infrastructure, distribution, and production polish.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Cleanvoice AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Cleanvoice AI · indie build A podcast editor where you delete words in a transcript and the audio follows. The engineering is in cutting audio at points that do not click, keeping several recorded tracks in sync when one is cut, and never destroying the original recording. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timings, run locally | every edit position comes from a word timing, so the timings are the foundation | | Audio | ffmpeg for decoding and rendering, with sample-accurate cut points | cutting on a word boundary is not the same as cutting where it sounds right | | Editing | A non-destructive edit list over untouched source files | the original recording is irreplaceable and every edit must be reversible | | Multitrack | Cuts applied across all tracks together | cutting one speaker's track alone destroys the alignment of the whole episode | | Runtime | Python 3.12 orchestrating them | the work is timing arithmetic, decisions and bookkeeping | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Orchestrates transcription, detection and rendering. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **ffmpeg with common codecs** · free - Why: Decoding, sample-accurate cutting and rendering. Reimplementing any of it is not the project. - Get it: Your package manager, and confirm it decodes the formats your recorder produces. - Verify: It decodes your recordings and reports exact durations - [ ] **whisper.cpp with a model** · free, but transcription takes real machine time - Why: Word-level timings are the foundation of every edit position, and segment-level timings are not precise enough. - Get it: Build whisper.cpp and choose a model sized for your machine. Confirm it emits word timings rather than only segments. - Verify: A five minute recording produces word-level timings - [ ] **A real multitrack episode** · free if you record one - Why: Every hard problem here appears only in real material: coarticulated fillers, tracks starting at different times, clock drift and crosstalk between microphones. - Get it: A recording with at least two separate microphone tracks and an hour of conversation. Your own is best, since you may edit it freely. - Verify: At least two tracks of an hour, with the tracks not starting at exactly the same instant - [ ] **Headphones you can hear artefacts on** · free if you own a pair - Why: Every claim about a cut being clean is verified by listening, and laptop speakers hide clicks entirely. - Get it: Any closed headphones you can listen critically on, in a quiet room. - Verify: You can hear a deliberately badly placed cut as a click ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Real-time processing during recording. - Commercial-quality voice isolation and noise removal. - Music beds, chapters and show notes. - Speaker diarisation on a single mixed track. - specialized cleanup models, cloud processing, and batch speed - remote studio reliability - licensed music libraries - hosting distribution - advanced mastering and support If one of those is essential to you, that is the reason to keep paying for Cleanvoice AI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Cleanvoice AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me filler word and silence removal for podcasts, edited through the transcript to replace CleanVoice AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timings, run locally · every edit position comes from a word timing, so the timings are the foundation - Audio: ffmpeg for decoding and rendering, with sample-accurate cut points · cutting on a word boundary is not the same as cutting where it sounds right - Editing: A non-destructive edit list over untouched source files · the original recording is irreplaceable and every edit must be reversible - Multitrack: Cuts applied across all tracks together · cutting one speaker's track alone destroys the alignment of the whole episode - Runtime: Python 3.12 orchestrating them · the work is timing arithmetic, decisions and bookkeeping ### Data model (create this before Phase 1) - episodes(id, title, sample_rate, duration_ms, created_at) - tracks(id, episode_id, path, speaker, offset_ms) · one per microphone, with its alignment offset - words(id, track_id, ordinal, text, start_ms, end_ms, confidence) - detections(id, track_id, kind, start_ms, end_ms, confidence, accepted) · kind is filler, silence, breath or mouth_click - edits(id, episode_id, kind, start_ms, end_ms, applied_at, undone_at) · the edit list, never applied to source - renders(id, episode_id, edit_revision, path, at) - Source files are opened read only. Everything else is a list of intentions over them. ### Phase 1 · Words with times you can cut on Build: Transcribe locally with word-level timings. Handle long recordings by chunking with overlap and merging without losing or duplicating words at the seams. Show the transcript aligned to a waveform. Trap: Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. Done when: An hour-long recording transcribes with word timings and no words lost or duplicated at chunk seams, verified by comparing against a single-chunk transcription of a shorter file. The final word's timestamp matches the audio duration within tolerance. Clicking a word plays from that point. Transcription runs with networking disabled. Do not build yet: No detection, cutting or multitrack. ### Phase 2 · Finding what to remove Build: Detect filler words from the transcript, long silences from the audio, and mouth clicks and breaths from short high-frequency transients. Propose every detection with a confidence. Nothing is removed automatically. Trap: Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. Done when: Fillers, silences and mouth sounds are detected with confidences and none are removed automatically. A bulk review accepts or rejects by kind and confidence. A pause that carries meaning can be kept individually. Detection over an hour completes in a time you would accept. Do not build yet: No cutting or rendering yet. ### Phase 3 · Cuts that do not click Build: Apply cuts with a short crossfade, choosing cut points at low-energy positions near the word boundary rather than exactly on it. Handle a filler between two coarticulated words where a clean cut is not available. Trap: Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. Done when: A cut produces no audible click, verified by inspecting the waveform at the join and listening. Two hundred cuts in an episode produce no accumulation of artefacts. A filler between coarticulated words is either cut acceptably or flagged as unsafe rather than cut badly. The cut is sample accurate and reproducible. Do not build yet: No multitrack or rendering yet. ### Phase 4 · Several microphones, one timeline Build: Align tracks to a common timeline, then apply every cut across all tracks together. Keep per-track detections separate so a filler is found on the speaker who said it. Handle tracks that start at different times or drift. Trap: Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. Done when: A cut on one speaker's track keeps all tracks in sync, verified by checking alignment at the end of an episode with a hundred cuts. Tracks starting at different times align correctly. A track with clock drift is detected and reported rather than silently misaligning. Detection remains per track. Do not build yet: No rendering or export yet. ### Phase 5 · Rendering, and never touching the original Build: Render the edit list to a mixed output with loudness normalisation to a stated target. Assert the source files are unmodified by checksum. Export the edit list so a render is reproducible from sources. Trap: Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ. Done when: A render produces a mixed episode normalised to a stated target. Source checksums are identical before and after, asserted in the test suite. The exported edit list re-renders byte-identically from the sources. An episode with a hundred cuts renders without a discontinuity anywhere. Do not build yet: Nothing after this. ### Out of scope (and why) - Real-time processing during recording, which is a completely different latency problem. - Voice isolation and background noise removal at commercial quality, which is where the hosted products have invested most heavily. - Music beds, chapters and show notes, which is podcast production rather than audio repair. - Speaker diarisation on a single mixed track, which is unreliable enough that separate tracks are the honest answer. ### README must contain - That cuts are placed at low-energy points near a word boundary rather than on it, and why - That every cut applies to all tracks so a multitrack episode stays in sync - That detections are proposed with confidence and nothing is removed automatically - That source files are opened read only and checksummed in the test suite - That an exported edit list re-renders the episode identically from sources ===== AGENTS.md ===== # Agent instructions · Cleanvoice AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: whisper.cpp with word-level timings, run locally, ffmpeg for decoding and rendering, with sample-accurate cut points, A non-destructive edit list over untouched source files, Cuts applied across all tracks together, Python 3.12 orchestrating them. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. ===== BUILD_PLAN.md ===== # Build plan · Cleanvoice AI A podcast editor where you delete words in a transcript and the audio follows. The engineering is in cutting audio at points that do not click, keeping several recorded tracks in sync when one is cut, and never destroying the original recording. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Words with times you can cut on A transcript whose word timings are accurate enough to edit by. ### Steps 1. Transcribe with word timings locally Word level, not segment level. Segment timings are not precise enough to cut on. Files: `src/transcribe.py` 2. Chunk with overlap and merge carefully Verify the word count and final timestamp against the whole file. Files: `src/chunk.py`, `tests/test_chunk.py` 3. Align transcript to waveform Clicking a word must play from that point, or you cannot trust the timings. Files: `src/align.py` ### Done when - [ ] An hour transcribes with no words lost at seams - [ ] The final timestamp matches the audio duration - [ ] Clicking a word plays from that point - [ ] Transcription works offline ### Watch out - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. ## Phase 2 · Finding what to remove Fillers, long silences and mouth sounds are found and proposed. ### Steps 1. Detect fillers from the transcript With their word timings, so the cut point comes from the same source. Files: `src/detect_filler.py` 2. Detect silences and transients from audio Silence by level and duration, mouth sounds by short high-frequency energy. Files: `src/detect_audio.py` 3. Propose with confidence, remove nothing Bulk review by kind and confidence is what makes this usable without being reckless. Files: `src/detections.py` ### Done when - [ ] All three kinds are detected with confidences - [ ] Nothing is removed without acceptance - [ ] Bulk review works by kind and confidence - [ ] An hour detects in an acceptable time ### Watch out - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. ## Phase 3 · Cuts that do not click Removing a word leaves a join you cannot hear. ### Steps 1. Search for a low-energy cut point Near the boundary, not on it. This is the whole difference between clean and clicking. Files: `src/cutpoint.py`, `tests/test_cutpoint.py` 2. Crossfade across a few milliseconds Short enough to be inaudible, long enough to remove the discontinuity. Files: `src/cut.py` 3. Flag cuts that cannot be made cleanly Better to leave a filler in than to leave a click. Files: `src/cut.py` ### Done when - [ ] A cut produces no audible click - [ ] Two hundred cuts accumulate no artefacts - [ ] An unsafe cut is flagged rather than made badly - [ ] Cuts are sample accurate and reproducible ### Watch out - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. ## Phase 4 · Several microphones, one timeline Cutting one speaker does not desynchronise the others. ### Steps 1. Align tracks with measured offsets Recording software rarely starts every track at the same instant. Files: `src/align_tracks.py` 2. Apply cuts to the timeline, not a track Detection is per track. Cutting is not. Files: `src/timeline.py`, `tests/test_sync.py` 3. Detect drift and report it Two devices with different clocks drift. Silently misaligning is the worst outcome. Files: `src/drift.py` ### Done when - [ ] A hundred cuts leave all tracks in sync - [ ] Tracks starting at different times align - [ ] Clock drift is detected and reported - [ ] Detection remains per track ### Watch out - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. ## Not in this build - Real-time processing during recording. - Commercial-quality voice isolation and noise removal. - Music beds, chapters and show notes. - Speaker diarisation on a single mixed track. ## After v1, if you want it - Crosstalk gating, so one speaker's microphone stops carrying the other - A tightening pass that shortens rather than removes pauses, which sounds more natural - Per-speaker levelling before the mix, once the timeline is trustworthy ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, opened read only. Nothing in the build writes here. SOURCE_DIR=./sources # Required. Output episodes. Always derivable from sources and the edit list. RENDER_DIR=./renders # Required. Larger models give better timings and take longer. Timing quality matters more than word accuracy here. WHISPER_MODEL=./models/ggml-medium.en.bin # Required. The join length at a cut. Too short clicks, too long smears the speech. CROSSFADE_MS=8 # Required. How far around a word boundary to look for a low-energy point. CUTPOINT_SEARCH_MS=40 # Required. The normalisation target. State it rather than just making the file louder. LOUDNESS_TARGET_LUFS=-16
You are building a lean indie version of Cleanvoice AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Cleanvoice AI · indie build A podcast editor where you delete words in a transcript and the audio follows. The engineering is in cutting audio at points that do not click, keeping several recorded tracks in sync when one is cut, and never destroying the original recording. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timings, run locally | every edit position comes from a word timing, so the timings are the foundation | | Audio | ffmpeg for decoding and rendering, with sample-accurate cut points | cutting on a word boundary is not the same as cutting where it sounds right | | Editing | A non-destructive edit list over untouched source files | the original recording is irreplaceable and every edit must be reversible | | Multitrack | Cuts applied across all tracks together | cutting one speaker's track alone destroys the alignment of the whole episode | | Runtime | Python 3.12 orchestrating them | the work is timing arithmetic, decisions and bookkeeping | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Orchestrates transcription, detection and rendering. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **ffmpeg with common codecs** · free - Why: Decoding, sample-accurate cutting and rendering. Reimplementing any of it is not the project. - Get it: Your package manager, and confirm it decodes the formats your recorder produces. - Verify: It decodes your recordings and reports exact durations - [ ] **whisper.cpp with a model** · free, but transcription takes real machine time - Why: Word-level timings are the foundation of every edit position, and segment-level timings are not precise enough. - Get it: Build whisper.cpp and choose a model sized for your machine. Confirm it emits word timings rather than only segments. - Verify: A five minute recording produces word-level timings - [ ] **A real multitrack episode** · free if you record one - Why: Every hard problem here appears only in real material: coarticulated fillers, tracks starting at different times, clock drift and crosstalk between microphones. - Get it: A recording with at least two separate microphone tracks and an hour of conversation. Your own is best, since you may edit it freely. - Verify: At least two tracks of an hour, with the tracks not starting at exactly the same instant - [ ] **Headphones you can hear artefacts on** · free if you own a pair - Why: Every claim about a cut being clean is verified by listening, and laptop speakers hide clicks entirely. - Get it: Any closed headphones you can listen critically on, in a quiet room. - Verify: You can hear a deliberately badly placed cut as a click ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Real-time processing during recording. - Commercial-quality voice isolation and noise removal. - Music beds, chapters and show notes. - Speaker diarisation on a single mixed track. - specialized cleanup models, cloud processing, and batch speed - remote studio reliability - licensed music libraries - hosting distribution - advanced mastering and support If one of those is essential to you, that is the reason to keep paying for Cleanvoice AI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Cleanvoice AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me filler word and silence removal for podcasts, edited through the transcript to replace CleanVoice AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timings, run locally · every edit position comes from a word timing, so the timings are the foundation - Audio: ffmpeg for decoding and rendering, with sample-accurate cut points · cutting on a word boundary is not the same as cutting where it sounds right - Editing: A non-destructive edit list over untouched source files · the original recording is irreplaceable and every edit must be reversible - Multitrack: Cuts applied across all tracks together · cutting one speaker's track alone destroys the alignment of the whole episode - Runtime: Python 3.12 orchestrating them · the work is timing arithmetic, decisions and bookkeeping ### Data model (create this before Phase 1) - episodes(id, title, sample_rate, duration_ms, created_at) - tracks(id, episode_id, path, speaker, offset_ms) · one per microphone, with its alignment offset - words(id, track_id, ordinal, text, start_ms, end_ms, confidence) - detections(id, track_id, kind, start_ms, end_ms, confidence, accepted) · kind is filler, silence, breath or mouth_click - edits(id, episode_id, kind, start_ms, end_ms, applied_at, undone_at) · the edit list, never applied to source - renders(id, episode_id, edit_revision, path, at) - Source files are opened read only. Everything else is a list of intentions over them. ### Phase 1 · Words with times you can cut on Build: Transcribe locally with word-level timings. Handle long recordings by chunking with overlap and merging without losing or duplicating words at the seams. Show the transcript aligned to a waveform. Trap: Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. Done when: An hour-long recording transcribes with word timings and no words lost or duplicated at chunk seams, verified by comparing against a single-chunk transcription of a shorter file. The final word's timestamp matches the audio duration within tolerance. Clicking a word plays from that point. Transcription runs with networking disabled. Do not build yet: No detection, cutting or multitrack. ### Phase 2 · Finding what to remove Build: Detect filler words from the transcript, long silences from the audio, and mouth clicks and breaths from short high-frequency transients. Propose every detection with a confidence. Nothing is removed automatically. Trap: Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. Done when: Fillers, silences and mouth sounds are detected with confidences and none are removed automatically. A bulk review accepts or rejects by kind and confidence. A pause that carries meaning can be kept individually. Detection over an hour completes in a time you would accept. Do not build yet: No cutting or rendering yet. ### Phase 3 · Cuts that do not click Build: Apply cuts with a short crossfade, choosing cut points at low-energy positions near the word boundary rather than exactly on it. Handle a filler between two coarticulated words where a clean cut is not available. Trap: Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. Done when: A cut produces no audible click, verified by inspecting the waveform at the join and listening. Two hundred cuts in an episode produce no accumulation of artefacts. A filler between coarticulated words is either cut acceptably or flagged as unsafe rather than cut badly. The cut is sample accurate and reproducible. Do not build yet: No multitrack or rendering yet. ### Phase 4 · Several microphones, one timeline Build: Align tracks to a common timeline, then apply every cut across all tracks together. Keep per-track detections separate so a filler is found on the speaker who said it. Handle tracks that start at different times or drift. Trap: Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. Done when: A cut on one speaker's track keeps all tracks in sync, verified by checking alignment at the end of an episode with a hundred cuts. Tracks starting at different times align correctly. A track with clock drift is detected and reported rather than silently misaligning. Detection remains per track. Do not build yet: No rendering or export yet. ### Phase 5 · Rendering, and never touching the original Build: Render the edit list to a mixed output with loudness normalisation to a stated target. Assert the source files are unmodified by checksum. Export the edit list so a render is reproducible from sources. Trap: Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ. Done when: A render produces a mixed episode normalised to a stated target. Source checksums are identical before and after, asserted in the test suite. The exported edit list re-renders byte-identically from the sources. An episode with a hundred cuts renders without a discontinuity anywhere. Do not build yet: Nothing after this. ### Out of scope (and why) - Real-time processing during recording, which is a completely different latency problem. - Voice isolation and background noise removal at commercial quality, which is where the hosted products have invested most heavily. - Music beds, chapters and show notes, which is podcast production rather than audio repair. - Speaker diarisation on a single mixed track, which is unreliable enough that separate tracks are the honest answer. ### README must contain - That cuts are placed at low-energy points near a word boundary rather than on it, and why - That every cut applies to all tracks so a multitrack episode stays in sync - That detections are proposed with confidence and nothing is removed automatically - That source files are opened read only and checksummed in the test suite - That an exported edit list re-renders the episode identically from sources ===== AGENTS.md ===== # Agent instructions · Cleanvoice AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: whisper.cpp with word-level timings, run locally, ffmpeg for decoding and rendering, with sample-accurate cut points, A non-destructive edit list over untouched source files, Cuts applied across all tracks together, Python 3.12 orchestrating them. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. ===== BUILD_PLAN.md ===== # Build plan · Cleanvoice AI A podcast editor where you delete words in a transcript and the audio follows. The engineering is in cutting audio at points that do not click, keeping several recorded tracks in sync when one is cut, and never destroying the original recording. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Words with times you can cut on A transcript whose word timings are accurate enough to edit by. ### Steps 1. Transcribe with word timings locally Word level, not segment level. Segment timings are not precise enough to cut on. Files: `src/transcribe.py` 2. Chunk with overlap and merge carefully Verify the word count and final timestamp against the whole file. Files: `src/chunk.py`, `tests/test_chunk.py` 3. Align transcript to waveform Clicking a word must play from that point, or you cannot trust the timings. Files: `src/align.py` ### Done when - [ ] An hour transcribes with no words lost at seams - [ ] The final timestamp matches the audio duration - [ ] Clicking a word plays from that point - [ ] Transcription works offline ### Watch out - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. ## Phase 2 · Finding what to remove Fillers, long silences and mouth sounds are found and proposed. ### Steps 1. Detect fillers from the transcript With their word timings, so the cut point comes from the same source. Files: `src/detect_filler.py` 2. Detect silences and transients from audio Silence by level and duration, mouth sounds by short high-frequency energy. Files: `src/detect_audio.py` 3. Propose with confidence, remove nothing Bulk review by kind and confidence is what makes this usable without being reckless. Files: `src/detections.py` ### Done when - [ ] All three kinds are detected with confidences - [ ] Nothing is removed without acceptance - [ ] Bulk review works by kind and confidence - [ ] An hour detects in an acceptable time ### Watch out - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. ## Phase 3 · Cuts that do not click Removing a word leaves a join you cannot hear. ### Steps 1. Search for a low-energy cut point Near the boundary, not on it. This is the whole difference between clean and clicking. Files: `src/cutpoint.py`, `tests/test_cutpoint.py` 2. Crossfade across a few milliseconds Short enough to be inaudible, long enough to remove the discontinuity. Files: `src/cut.py` 3. Flag cuts that cannot be made cleanly Better to leave a filler in than to leave a click. Files: `src/cut.py` ### Done when - [ ] A cut produces no audible click - [ ] Two hundred cuts accumulate no artefacts - [ ] An unsafe cut is flagged rather than made badly - [ ] Cuts are sample accurate and reproducible ### Watch out - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. ## Phase 4 · Several microphones, one timeline Cutting one speaker does not desynchronise the others. ### Steps 1. Align tracks with measured offsets Recording software rarely starts every track at the same instant. Files: `src/align_tracks.py` 2. Apply cuts to the timeline, not a track Detection is per track. Cutting is not. Files: `src/timeline.py`, `tests/test_sync.py` 3. Detect drift and report it Two devices with different clocks drift. Silently misaligning is the worst outcome. Files: `src/drift.py` ### Done when - [ ] A hundred cuts leave all tracks in sync - [ ] Tracks starting at different times align - [ ] Clock drift is detected and reported - [ ] Detection remains per track ### Watch out - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. ## Not in this build - Real-time processing during recording. - Commercial-quality voice isolation and noise removal. - Music beds, chapters and show notes. - Speaker diarisation on a single mixed track. ## After v1, if you want it - Crosstalk gating, so one speaker's microphone stops carrying the other - A tightening pass that shortens rather than removes pauses, which sounds more natural - Per-speaker levelling before the mix, once the timeline is trustworthy ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, opened read only. Nothing in the build writes here. SOURCE_DIR=./sources # Required. Output episodes. Always derivable from sources and the edit list. RENDER_DIR=./renders # Required. Larger models give better timings and take longer. Timing quality matters more than word accuracy here. WHISPER_MODEL=./models/ggml-medium.en.bin # Required. The join length at a cut. Too short clicks, too long smears the speech. CROSSFADE_MS=8 # Required. How far around a word boundary to look for a low-energy point. CUTPOINT_SEARCH_MS=40 # Required. The normalisation target. State it rather than just making the file louder. LOUDNESS_TARGET_LUFS=-16
You are building a production product version of Cleanvoice AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Cleanvoice AI · product brief ## Problem The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Cleanvoice AI, remove filler words, mouth sounds, and long silences from a transcript-aligned edit list. The hard boundary is specialized cleanup models, cloud processing, and batch speed, plus audio infrastructure, distribution, and production polish. ## Product outcome A podcast editor where you cut words out of a transcript, the audio follows without a single audible click, every microphone stays in sync, and the original recordings are provably untouched. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - ffmpeg - local audio files - optional transcription API key - sufficient disk space ## Explicit non-goals for v1 - Real-time processing during recording. - Commercial-quality voice isolation and noise removal. - Music beds, chapters and show notes. - Speaker diarisation on a single mixed track. - specialized cleanup models, cloud processing, and batch speed - remote studio reliability - licensed music libraries - hosting distribution - advanced mastering and support ## Success criteria - An hour transcribes with no words lost at chunk seams - A cut is confirmed to produce no audible click - A hundred cuts are confirmed to leave all tracks in sync - Source checksums are confirmed unchanged in the suite - An exported edit list is confirmed to re-render identically ===== BRIEF.md ===== # Build brief · Cleanvoice AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me filler word and silence removal for podcasts, edited through the transcript to replace CleanVoice AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timings, run locally · every edit position comes from a word timing, so the timings are the foundation - Audio: ffmpeg for decoding and rendering, with sample-accurate cut points · cutting on a word boundary is not the same as cutting where it sounds right - Editing: A non-destructive edit list over untouched source files · the original recording is irreplaceable and every edit must be reversible - Multitrack: Cuts applied across all tracks together · cutting one speaker's track alone destroys the alignment of the whole episode - Runtime: Python 3.12 orchestrating them · the work is timing arithmetic, decisions and bookkeeping ### Data model (create this before Phase 1) - episodes(id, title, sample_rate, duration_ms, created_at) - tracks(id, episode_id, path, speaker, offset_ms) · one per microphone, with its alignment offset - words(id, track_id, ordinal, text, start_ms, end_ms, confidence) - detections(id, track_id, kind, start_ms, end_ms, confidence, accepted) · kind is filler, silence, breath or mouth_click - edits(id, episode_id, kind, start_ms, end_ms, applied_at, undone_at) · the edit list, never applied to source - renders(id, episode_id, edit_revision, path, at) - Source files are opened read only. Everything else is a list of intentions over them. ### Phase 1 · Words with times you can cut on Build: Transcribe locally with word-level timings. Handle long recordings by chunking with overlap and merging without losing or duplicating words at the seams. Show the transcript aligned to a waveform. Trap: Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. Done when: An hour-long recording transcribes with word timings and no words lost or duplicated at chunk seams, verified by comparing against a single-chunk transcription of a shorter file. The final word's timestamp matches the audio duration within tolerance. Clicking a word plays from that point. Transcription runs with networking disabled. Do not build yet: No detection, cutting or multitrack. ### Phase 2 · Finding what to remove Build: Detect filler words from the transcript, long silences from the audio, and mouth clicks and breaths from short high-frequency transients. Propose every detection with a confidence. Nothing is removed automatically. Trap: Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. Done when: Fillers, silences and mouth sounds are detected with confidences and none are removed automatically. A bulk review accepts or rejects by kind and confidence. A pause that carries meaning can be kept individually. Detection over an hour completes in a time you would accept. Do not build yet: No cutting or rendering yet. ### Phase 3 · Cuts that do not click Build: Apply cuts with a short crossfade, choosing cut points at low-energy positions near the word boundary rather than exactly on it. Handle a filler between two coarticulated words where a clean cut is not available. Trap: Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. Done when: A cut produces no audible click, verified by inspecting the waveform at the join and listening. Two hundred cuts in an episode produce no accumulation of artefacts. A filler between coarticulated words is either cut acceptably or flagged as unsafe rather than cut badly. The cut is sample accurate and reproducible. Do not build yet: No multitrack or rendering yet. ### Phase 4 · Several microphones, one timeline Build: Align tracks to a common timeline, then apply every cut across all tracks together. Keep per-track detections separate so a filler is found on the speaker who said it. Handle tracks that start at different times or drift. Trap: Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. Done when: A cut on one speaker's track keeps all tracks in sync, verified by checking alignment at the end of an episode with a hundred cuts. Tracks starting at different times align correctly. A track with clock drift is detected and reported rather than silently misaligning. Detection remains per track. Do not build yet: No rendering or export yet. ### Phase 5 · Rendering, and never touching the original Build: Render the edit list to a mixed output with loudness normalisation to a stated target. Assert the source files are unmodified by checksum. Export the edit list so a render is reproducible from sources. Trap: Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ. Done when: A render produces a mixed episode normalised to a stated target. Source checksums are identical before and after, asserted in the test suite. The exported edit list re-renders byte-identically from the sources. An episode with a hundred cuts renders without a discontinuity anywhere. Do not build yet: Nothing after this. ### Out of scope (and why) - Real-time processing during recording, which is a completely different latency problem. - Voice isolation and background noise removal at commercial quality, which is where the hosted products have invested most heavily. - Music beds, chapters and show notes, which is podcast production rather than audio repair. - Speaker diarisation on a single mixed track, which is unreliable enough that separate tracks are the honest answer. ### README must contain - That cuts are placed at low-energy points near a word boundary rather than on it, and why - That every cut applies to all tracks so a multitrack episode stays in sync - That detections are proposed with confidence and nothing is removed automatically - That source files are opened read only and checksummed in the test suite - That an exported edit list re-renders the episode identically from sources ===== ARCHITECTURE.md ===== # Architecture · Cleanvoice AI ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timings, run locally | every edit position comes from a word timing, so the timings are the foundation | | Audio | ffmpeg for decoding and rendering, with sample-accurate cut points | cutting on a word boundary is not the same as cutting where it sounds right | | Editing | A non-destructive edit list over untouched source files | the original recording is irreplaceable and every edit must be reversible | | Multitrack | Cuts applied across all tracks together | cutting one speaker's track alone destroys the alignment of the whole episode | | Runtime | Python 3.12 orchestrating them | the work is timing arithmetic, decisions and bookkeeping | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Transcription | Word timings, chunked and merged. | Any engine producing word-level timings. Segment timings are not sufficient. | | Detection | Proposing fillers, silences and mouth sounds. | Any detector, provided it proposes with confidence rather than removing. | | Cutting | Choosing cut points and crossfading. | Not swappable for cutting on the boundary, which clicks on every edit. | | Timeline | Applying cuts across all tracks. | Any implementation, provided a cut is never applied to one track alone. | | Render | Producing output from sources and edits. | Any renderer, provided sources stay read only and checksummed. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `SOURCE_DIR` · required · Recordings, opened read only. Nothing in the build writes here. - `RENDER_DIR` · required · Output episodes. Always derivable from sources and the edit list. - `WHISPER_MODEL` · required · Larger models give better timings and take longer. Timing quality matters more than word accuracy here. - `CROSSFADE_MS` · required · The join length at a cut. Too short clicks, too long smears the speech. - `CUTPOINT_SEARCH_MS` · required · How far around a word boundary to look for a low-energy point. - `LOUDNESS_TARGET_LUFS` · required · The normalisation target. State it rather than just making the file louder. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Cleanvoice AI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: whisper.cpp with word-level timings, run locally, ffmpeg for decoding and rendering, with sample-accurate cut points, A non-destructive edit list over untouched source files, Cuts applied across all tracks together, Python 3.12 orchestrating them. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. - Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ. ===== MILESTONES.md ===== # Delivery milestones · Cleanvoice AI Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Words with times you can cut on A transcript whose word timings are accurate enough to edit by. ### Steps 1. Transcribe with word timings locally Word level, not segment level. Segment timings are not precise enough to cut on. Files: `src/transcribe.py` 2. Chunk with overlap and merge carefully Verify the word count and final timestamp against the whole file. Files: `src/chunk.py`, `tests/test_chunk.py` 3. Align transcript to waveform Clicking a word must play from that point, or you cannot trust the timings. Files: `src/align.py` ### Done when - [ ] An hour transcribes with no words lost at seams - [ ] The final timestamp matches the audio duration - [ ] Clicking a word plays from that point - [ ] Transcription works offline ### Watch out - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. ## M2 · Finding what to remove Fillers, long silences and mouth sounds are found and proposed. ### Steps 1. Detect fillers from the transcript With their word timings, so the cut point comes from the same source. Files: `src/detect_filler.py` 2. Detect silences and transients from audio Silence by level and duration, mouth sounds by short high-frequency energy. Files: `src/detect_audio.py` 3. Propose with confidence, remove nothing Bulk review by kind and confidence is what makes this usable without being reckless. Files: `src/detections.py` ### Done when - [ ] All three kinds are detected with confidences - [ ] Nothing is removed without acceptance - [ ] Bulk review works by kind and confidence - [ ] An hour detects in an acceptable time ### Watch out - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. ## M3 · Cuts that do not click Removing a word leaves a join you cannot hear. ### Steps 1. Search for a low-energy cut point Near the boundary, not on it. This is the whole difference between clean and clicking. Files: `src/cutpoint.py`, `tests/test_cutpoint.py` 2. Crossfade across a few milliseconds Short enough to be inaudible, long enough to remove the discontinuity. Files: `src/cut.py` 3. Flag cuts that cannot be made cleanly Better to leave a filler in than to leave a click. Files: `src/cut.py` ### Done when - [ ] A cut produces no audible click - [ ] Two hundred cuts accumulate no artefacts - [ ] An unsafe cut is flagged rather than made badly - [ ] Cuts are sample accurate and reproducible ### Watch out - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. ## M4 · Several microphones, one timeline Cutting one speaker does not desynchronise the others. ### Steps 1. Align tracks with measured offsets Recording software rarely starts every track at the same instant. Files: `src/align_tracks.py` 2. Apply cuts to the timeline, not a track Detection is per track. Cutting is not. Files: `src/timeline.py`, `tests/test_sync.py` 3. Detect drift and report it Two devices with different clocks drift. Silently misaligning is the worst outcome. Files: `src/drift.py` ### Done when - [ ] A hundred cuts leave all tracks in sync - [ ] Tracks starting at different times align - [ ] Clock drift is detected and reported - [ ] Detection remains per track ### Watch out - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. ## M5 · Rendering, and never touching the original (production only) A finished episode renders, and the source recordings are provably untouched. ### Steps 1. Render from the edit list Sources plus intentions. The render is always derivable and never authoritative. Files: `src/render.py` 2. Checksum sources before and after In the suite. A write to a source is an irreversible loss. Files: `tests/test_sources_untouched.py` 3. Normalise to a stated target State the target rather than making the file merely louder. Files: `src/loudness.py` ### Done when - [ ] A render produces a normalised mixed episode - [ ] Source checksums are unchanged, asserted in tests - [ ] The exported edit list re-renders identically - [ ] A hundred cuts render with no discontinuity ### Watch out - Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ. ===== OPERATIONS.md ===== # Operations · Cleanvoice AI ## Backup Source recordings are irreplaceable and large. Back them up before editing anything, since the whole non-destructive design is undermined if the sources themselves are lost. Edit lists are tiny and worth versioning. ## Restore Restore sources and edit lists, re-render, and compare against a kept render. Byte equality proves the pipeline is deterministic and the restore is complete. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring For a local tool the checks that matter run in the suite: source checksums unchanged, word counts preserved across chunk seams, and track alignment after a full edit list. ## Incident checklist An episode that drifts out of sync partway through means a cut was applied to one track only, and the point of divergence identifies the edit. A clicking episode means cut points are landing on boundaries rather than in low-energy windows. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] An hour transcribes with no words lost at chunk seams - [ ] A cut is confirmed to produce no audible click - [ ] A hundred cuts are confirmed to leave all tracks in sync - [ ] Source checksums are confirmed unchanged in the suite - [ ] An exported edit list is confirmed to re-render identically ## Launch constraint Do not market omitted Cleanvoice AI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, opened read only. Nothing in the build writes here. SOURCE_DIR=./sources # Required. Output episodes. Always derivable from sources and the edit list. RENDER_DIR=./renders # Required. Larger models give better timings and take longer. Timing quality matters more than word accuracy here. WHISPER_MODEL=./models/ggml-medium.en.bin # Required. The join length at a cut. Too short clicks, too long smears the speech. CROSSFADE_MS=8 # Required. How far around a word boundary to look for a low-energy point. CUTPOINT_SEARCH_MS=40 # Required. The normalisation target. State it rather than just making the file louder. LOUDNESS_TARGET_LUFS=-16
README.md
# Cleanvoice AI · indie build A podcast editor where you delete words in a transcript and the audio follows. The engineering is in cutting audio at points that do not click, keeping several recorded tracks in sync when one is cut, and never destroying the original recording. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timings, run locally | every edit position comes from a word timing, so the timings are the foundation | | Audio | ffmpeg for decoding and rendering, with sample-accurate cut points | cutting on a word boundary is not the same as cutting where it sounds right | | Editing | A non-destructive edit list over untouched source files | the original recording is irreplaceable and every edit must be reversible | | Multitrack | Cuts applied across all tracks together | cutting one speaker's track alone destroys the alignment of the whole episode | | Runtime | Python 3.12 orchestrating them | the work is timing arithmetic, decisions and bookkeeping | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Orchestrates transcription, detection and rendering. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **ffmpeg with common codecs** · free - Why: Decoding, sample-accurate cutting and rendering. Reimplementing any of it is not the project. - Get it: Your package manager, and confirm it decodes the formats your recorder produces. - Verify: It decodes your recordings and reports exact durations - [ ] **whisper.cpp with a model** · free, but transcription takes real machine time - Why: Word-level timings are the foundation of every edit position, and segment-level timings are not precise enough. - Get it: Build whisper.cpp and choose a model sized for your machine. Confirm it emits word timings rather than only segments. - Verify: A five minute recording produces word-level timings - [ ] **A real multitrack episode** · free if you record one - Why: Every hard problem here appears only in real material: coarticulated fillers, tracks starting at different times, clock drift and crosstalk between microphones. - Get it: A recording with at least two separate microphone tracks and an hour of conversation. Your own is best, since you may edit it freely. - Verify: At least two tracks of an hour, with the tracks not starting at exactly the same instant - [ ] **Headphones you can hear artefacts on** · free if you own a pair - Why: Every claim about a cut being clean is verified by listening, and laptop speakers hide clicks entirely. - Get it: Any closed headphones you can listen critically on, in a quiet room. - Verify: You can hear a deliberately badly placed cut as a click ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Real-time processing during recording. - Commercial-quality voice isolation and noise removal. - Music beds, chapters and show notes. - Speaker diarisation on a single mixed track. - specialized cleanup models, cloud processing, and batch speed - remote studio reliability - licensed music libraries - hosting distribution - advanced mastering and support If one of those is essential to you, that is the reason to keep paying for Cleanvoice AI, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Cleanvoice AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me filler word and silence removal for podcasts, edited through the transcript to replace CleanVoice AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timings, run locally · every edit position comes from a word timing, so the timings are the foundation - Audio: ffmpeg for decoding and rendering, with sample-accurate cut points · cutting on a word boundary is not the same as cutting where it sounds right - Editing: A non-destructive edit list over untouched source files · the original recording is irreplaceable and every edit must be reversible - Multitrack: Cuts applied across all tracks together · cutting one speaker's track alone destroys the alignment of the whole episode - Runtime: Python 3.12 orchestrating them · the work is timing arithmetic, decisions and bookkeeping ### Data model (create this before Phase 1) - episodes(id, title, sample_rate, duration_ms, created_at) - tracks(id, episode_id, path, speaker, offset_ms) · one per microphone, with its alignment offset - words(id, track_id, ordinal, text, start_ms, end_ms, confidence) - detections(id, track_id, kind, start_ms, end_ms, confidence, accepted) · kind is filler, silence, breath or mouth_click - edits(id, episode_id, kind, start_ms, end_ms, applied_at, undone_at) · the edit list, never applied to source - renders(id, episode_id, edit_revision, path, at) - Source files are opened read only. Everything else is a list of intentions over them. ### Phase 1 · Words with times you can cut on Build: Transcribe locally with word-level timings. Handle long recordings by chunking with overlap and merging without losing or duplicating words at the seams. Show the transcript aligned to a waveform. Trap: Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. Done when: An hour-long recording transcribes with word timings and no words lost or duplicated at chunk seams, verified by comparing against a single-chunk transcription of a shorter file. The final word's timestamp matches the audio duration within tolerance. Clicking a word plays from that point. Transcription runs with networking disabled. Do not build yet: No detection, cutting or multitrack. ### Phase 2 · Finding what to remove Build: Detect filler words from the transcript, long silences from the audio, and mouth clicks and breaths from short high-frequency transients. Propose every detection with a confidence. Nothing is removed automatically. Trap: Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. Done when: Fillers, silences and mouth sounds are detected with confidences and none are removed automatically. A bulk review accepts or rejects by kind and confidence. A pause that carries meaning can be kept individually. Detection over an hour completes in a time you would accept. Do not build yet: No cutting or rendering yet. ### Phase 3 · Cuts that do not click Build: Apply cuts with a short crossfade, choosing cut points at low-energy positions near the word boundary rather than exactly on it. Handle a filler between two coarticulated words where a clean cut is not available. Trap: Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. Done when: A cut produces no audible click, verified by inspecting the waveform at the join and listening. Two hundred cuts in an episode produce no accumulation of artefacts. A filler between coarticulated words is either cut acceptably or flagged as unsafe rather than cut badly. The cut is sample accurate and reproducible. Do not build yet: No multitrack or rendering yet. ### Phase 4 · Several microphones, one timeline Build: Align tracks to a common timeline, then apply every cut across all tracks together. Keep per-track detections separate so a filler is found on the speaker who said it. Handle tracks that start at different times or drift. Trap: Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. Done when: A cut on one speaker's track keeps all tracks in sync, verified by checking alignment at the end of an episode with a hundred cuts. Tracks starting at different times align correctly. A track with clock drift is detected and reported rather than silently misaligning. Detection remains per track. Do not build yet: No rendering or export yet. ### Phase 5 · Rendering, and never touching the original Build: Render the edit list to a mixed output with loudness normalisation to a stated target. Assert the source files are unmodified by checksum. Export the edit list so a render is reproducible from sources. Trap: Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ. Done when: A render produces a mixed episode normalised to a stated target. Source checksums are identical before and after, asserted in the test suite. The exported edit list re-renders byte-identically from the sources. An episode with a hundred cuts renders without a discontinuity anywhere. Do not build yet: Nothing after this. ### Out of scope (and why) - Real-time processing during recording, which is a completely different latency problem. - Voice isolation and background noise removal at commercial quality, which is where the hosted products have invested most heavily. - Music beds, chapters and show notes, which is podcast production rather than audio repair. - Speaker diarisation on a single mixed track, which is unreliable enough that separate tracks are the honest answer. ### README must contain - That cuts are placed at low-energy points near a word boundary rather than on it, and why - That every cut applies to all tracks so a multitrack episode stays in sync - That detections are proposed with confidence and nothing is removed automatically - That source files are opened read only and checksummed in the test suite - That an exported edit list re-renders the episode identically from sources
AGENTS.md
# Agent instructions · Cleanvoice AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: whisper.cpp with word-level timings, run locally, ffmpeg for decoding and rendering, with sample-accurate cut points, A non-destructive edit list over untouched source files, Cuts applied across all tracks together, Python 3.12 orchestrating them. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track.
BUILD_PLAN.md
# Build plan · Cleanvoice AI A podcast editor where you delete words in a transcript and the audio follows. The engineering is in cutting audio at points that do not click, keeping several recorded tracks in sync when one is cut, and never destroying the original recording. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Words with times you can cut on A transcript whose word timings are accurate enough to edit by. ### Steps 1. Transcribe with word timings locally Word level, not segment level. Segment timings are not precise enough to cut on. Files: `src/transcribe.py` 2. Chunk with overlap and merge carefully Verify the word count and final timestamp against the whole file. Files: `src/chunk.py`, `tests/test_chunk.py` 3. Align transcript to waveform Clicking a word must play from that point, or you cannot trust the timings. Files: `src/align.py` ### Done when - [ ] An hour transcribes with no words lost at seams - [ ] The final timestamp matches the audio duration - [ ] Clicking a word plays from that point - [ ] Transcription works offline ### Watch out - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. ## Phase 2 · Finding what to remove Fillers, long silences and mouth sounds are found and proposed. ### Steps 1. Detect fillers from the transcript With their word timings, so the cut point comes from the same source. Files: `src/detect_filler.py` 2. Detect silences and transients from audio Silence by level and duration, mouth sounds by short high-frequency energy. Files: `src/detect_audio.py` 3. Propose with confidence, remove nothing Bulk review by kind and confidence is what makes this usable without being reckless. Files: `src/detections.py` ### Done when - [ ] All three kinds are detected with confidences - [ ] Nothing is removed without acceptance - [ ] Bulk review works by kind and confidence - [ ] An hour detects in an acceptable time ### Watch out - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. ## Phase 3 · Cuts that do not click Removing a word leaves a join you cannot hear. ### Steps 1. Search for a low-energy cut point Near the boundary, not on it. This is the whole difference between clean and clicking. Files: `src/cutpoint.py`, `tests/test_cutpoint.py` 2. Crossfade across a few milliseconds Short enough to be inaudible, long enough to remove the discontinuity. Files: `src/cut.py` 3. Flag cuts that cannot be made cleanly Better to leave a filler in than to leave a click. Files: `src/cut.py` ### Done when - [ ] A cut produces no audible click - [ ] Two hundred cuts accumulate no artefacts - [ ] An unsafe cut is flagged rather than made badly - [ ] Cuts are sample accurate and reproducible ### Watch out - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. ## Phase 4 · Several microphones, one timeline Cutting one speaker does not desynchronise the others. ### Steps 1. Align tracks with measured offsets Recording software rarely starts every track at the same instant. Files: `src/align_tracks.py` 2. Apply cuts to the timeline, not a track Detection is per track. Cutting is not. Files: `src/timeline.py`, `tests/test_sync.py` 3. Detect drift and report it Two devices with different clocks drift. Silently misaligning is the worst outcome. Files: `src/drift.py` ### Done when - [ ] A hundred cuts leave all tracks in sync - [ ] Tracks starting at different times align - [ ] Clock drift is detected and reported - [ ] Detection remains per track ### Watch out - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. ## Not in this build - Real-time processing during recording. - Commercial-quality voice isolation and noise removal. - Music beds, chapters and show notes. - Speaker diarisation on a single mixed track. ## After v1, if you want it - Crosstalk gating, so one speaker's microphone stops carrying the other - A tightening pass that shortens rather than removes pauses, which sounds more natural - Per-speaker levelling before the mix, once the timeline is trustworthy
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, opened read only. Nothing in the build writes here. SOURCE_DIR=./sources # Required. Output episodes. Always derivable from sources and the edit list. RENDER_DIR=./renders # Required. Larger models give better timings and take longer. Timing quality matters more than word accuracy here. WHISPER_MODEL=./models/ggml-medium.en.bin # Required. The join length at a cut. Too short clicks, too long smears the speech. CROSSFADE_MS=8 # Required. How far around a word boundary to look for a low-energy point. CUTPOINT_SEARCH_MS=40 # Required. The normalisation target. State it rather than just making the file louder. LOUDNESS_TARGET_LUFS=-16
PRODUCT.md
# Cleanvoice AI · product brief ## Problem The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Cleanvoice AI, remove filler words, mouth sounds, and long silences from a transcript-aligned edit list. The hard boundary is specialized cleanup models, cloud processing, and batch speed, plus audio infrastructure, distribution, and production polish. ## Product outcome A podcast editor where you cut words out of a transcript, the audio follows without a single audible click, every microphone stays in sync, and the original recordings are provably untouched. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - ffmpeg - local audio files - optional transcription API key - sufficient disk space ## Explicit non-goals for v1 - Real-time processing during recording. - Commercial-quality voice isolation and noise removal. - Music beds, chapters and show notes. - Speaker diarisation on a single mixed track. - specialized cleanup models, cloud processing, and batch speed - remote studio reliability - licensed music libraries - hosting distribution - advanced mastering and support ## Success criteria - An hour transcribes with no words lost at chunk seams - A cut is confirmed to produce no audible click - A hundred cuts are confirmed to leave all tracks in sync - Source checksums are confirmed unchanged in the suite - An exported edit list is confirmed to re-render identically
BRIEF.md
# Build brief · Cleanvoice AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me filler word and silence removal for podcasts, edited through the transcript to replace CleanVoice AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timings, run locally · every edit position comes from a word timing, so the timings are the foundation - Audio: ffmpeg for decoding and rendering, with sample-accurate cut points · cutting on a word boundary is not the same as cutting where it sounds right - Editing: A non-destructive edit list over untouched source files · the original recording is irreplaceable and every edit must be reversible - Multitrack: Cuts applied across all tracks together · cutting one speaker's track alone destroys the alignment of the whole episode - Runtime: Python 3.12 orchestrating them · the work is timing arithmetic, decisions and bookkeeping ### Data model (create this before Phase 1) - episodes(id, title, sample_rate, duration_ms, created_at) - tracks(id, episode_id, path, speaker, offset_ms) · one per microphone, with its alignment offset - words(id, track_id, ordinal, text, start_ms, end_ms, confidence) - detections(id, track_id, kind, start_ms, end_ms, confidence, accepted) · kind is filler, silence, breath or mouth_click - edits(id, episode_id, kind, start_ms, end_ms, applied_at, undone_at) · the edit list, never applied to source - renders(id, episode_id, edit_revision, path, at) - Source files are opened read only. Everything else is a list of intentions over them. ### Phase 1 · Words with times you can cut on Build: Transcribe locally with word-level timings. Handle long recordings by chunking with overlap and merging without losing or duplicating words at the seams. Show the transcript aligned to a waveform. Trap: Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. Done when: An hour-long recording transcribes with word timings and no words lost or duplicated at chunk seams, verified by comparing against a single-chunk transcription of a shorter file. The final word's timestamp matches the audio duration within tolerance. Clicking a word plays from that point. Transcription runs with networking disabled. Do not build yet: No detection, cutting or multitrack. ### Phase 2 · Finding what to remove Build: Detect filler words from the transcript, long silences from the audio, and mouth clicks and breaths from short high-frequency transients. Propose every detection with a confidence. Nothing is removed automatically. Trap: Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. Done when: Fillers, silences and mouth sounds are detected with confidences and none are removed automatically. A bulk review accepts or rejects by kind and confidence. A pause that carries meaning can be kept individually. Detection over an hour completes in a time you would accept. Do not build yet: No cutting or rendering yet. ### Phase 3 · Cuts that do not click Build: Apply cuts with a short crossfade, choosing cut points at low-energy positions near the word boundary rather than exactly on it. Handle a filler between two coarticulated words where a clean cut is not available. Trap: Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. Done when: A cut produces no audible click, verified by inspecting the waveform at the join and listening. Two hundred cuts in an episode produce no accumulation of artefacts. A filler between coarticulated words is either cut acceptably or flagged as unsafe rather than cut badly. The cut is sample accurate and reproducible. Do not build yet: No multitrack or rendering yet. ### Phase 4 · Several microphones, one timeline Build: Align tracks to a common timeline, then apply every cut across all tracks together. Keep per-track detections separate so a filler is found on the speaker who said it. Handle tracks that start at different times or drift. Trap: Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. Done when: A cut on one speaker's track keeps all tracks in sync, verified by checking alignment at the end of an episode with a hundred cuts. Tracks starting at different times align correctly. A track with clock drift is detected and reported rather than silently misaligning. Detection remains per track. Do not build yet: No rendering or export yet. ### Phase 5 · Rendering, and never touching the original Build: Render the edit list to a mixed output with loudness normalisation to a stated target. Assert the source files are unmodified by checksum. Export the edit list so a render is reproducible from sources. Trap: Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ. Done when: A render produces a mixed episode normalised to a stated target. Source checksums are identical before and after, asserted in the test suite. The exported edit list re-renders byte-identically from the sources. An episode with a hundred cuts renders without a discontinuity anywhere. Do not build yet: Nothing after this. ### Out of scope (and why) - Real-time processing during recording, which is a completely different latency problem. - Voice isolation and background noise removal at commercial quality, which is where the hosted products have invested most heavily. - Music beds, chapters and show notes, which is podcast production rather than audio repair. - Speaker diarisation on a single mixed track, which is unreliable enough that separate tracks are the honest answer. ### README must contain - That cuts are placed at low-energy points near a word boundary rather than on it, and why - That every cut applies to all tracks so a multitrack episode stays in sync - That detections are proposed with confidence and nothing is removed automatically - That source files are opened read only and checksummed in the test suite - That an exported edit list re-renders the episode identically from sources
ARCHITECTURE.md
# Architecture · Cleanvoice AI ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timings, run locally | every edit position comes from a word timing, so the timings are the foundation | | Audio | ffmpeg for decoding and rendering, with sample-accurate cut points | cutting on a word boundary is not the same as cutting where it sounds right | | Editing | A non-destructive edit list over untouched source files | the original recording is irreplaceable and every edit must be reversible | | Multitrack | Cuts applied across all tracks together | cutting one speaker's track alone destroys the alignment of the whole episode | | Runtime | Python 3.12 orchestrating them | the work is timing arithmetic, decisions and bookkeeping | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Transcription | Word timings, chunked and merged. | Any engine producing word-level timings. Segment timings are not sufficient. | | Detection | Proposing fillers, silences and mouth sounds. | Any detector, provided it proposes with confidence rather than removing. | | Cutting | Choosing cut points and crossfading. | Not swappable for cutting on the boundary, which clicks on every edit. | | Timeline | Applying cuts across all tracks. | Any implementation, provided a cut is never applied to one track alone. | | Render | Producing output from sources and edits. | Any renderer, provided sources stay read only and checksummed. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `SOURCE_DIR` · required · Recordings, opened read only. Nothing in the build writes here. - `RENDER_DIR` · required · Output episodes. Always derivable from sources and the edit list. - `WHISPER_MODEL` · required · Larger models give better timings and take longer. Timing quality matters more than word accuracy here. - `CROSSFADE_MS` · required · The join length at a cut. Too short clicks, too long smears the speech. - `CUTPOINT_SEARCH_MS` · required · How far around a word boundary to look for a low-energy point. - `LOUDNESS_TARGET_LUFS` · required · The normalisation target. State it rather than just making the file louder. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Cleanvoice AI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: whisper.cpp with word-level timings, run locally, ffmpeg for decoding and rendering, with sample-accurate cut points, A non-destructive edit list over untouched source files, Cuts applied across all tracks together, Python 3.12 orchestrating them. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. - Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ.
MILESTONES.md
# Delivery milestones · Cleanvoice AI Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Words with times you can cut on A transcript whose word timings are accurate enough to edit by. ### Steps 1. Transcribe with word timings locally Word level, not segment level. Segment timings are not precise enough to cut on. Files: `src/transcribe.py` 2. Chunk with overlap and merge carefully Verify the word count and final timestamp against the whole file. Files: `src/chunk.py`, `tests/test_chunk.py` 3. Align transcript to waveform Clicking a word must play from that point, or you cannot trust the timings. Files: `src/align.py` ### Done when - [ ] An hour transcribes with no words lost at seams - [ ] The final timestamp matches the audio duration - [ ] Clicking a word plays from that point - [ ] Transcription works offline ### Watch out - Chunking a long recording without overlap loses words at every boundary, and with naive overlap it duplicates them, so an hour-long episode ends up with a transcript that drifts from the audio. Overlap the chunks, merge on the overlap by matching text, and verify the word count and final timestamp against the whole file. ## M2 · Finding what to remove Fillers, long silences and mouth sounds are found and proposed. ### Steps 1. Detect fillers from the transcript With their word timings, so the cut point comes from the same source. Files: `src/detect_filler.py` 2. Detect silences and transients from audio Silence by level and duration, mouth sounds by short high-frequency energy. Files: `src/detect_audio.py` 3. Propose with confidence, remove nothing Bulk review by kind and confidence is what makes this usable without being reckless. Files: `src/detections.py` ### Done when - [ ] All three kinds are detected with confidences - [ ] Nothing is removed without acceptance - [ ] Bulk review works by kind and confidence - [ ] An hour detects in an acceptable time ### Watch out - Removing every detected filler automatically strips the speech of its natural rhythm and produces an episode that sounds unsettlingly clipped, and it also removes the ones that were doing real work, since a pause before an answer carries meaning. Propose with confidence, default to accepting only the clearest, and always allow a bulk review. ## M3 · Cuts that do not click Removing a word leaves a join you cannot hear. ### Steps 1. Search for a low-energy cut point Near the boundary, not on it. This is the whole difference between clean and clicking. Files: `src/cutpoint.py`, `tests/test_cutpoint.py` 2. Crossfade across a few milliseconds Short enough to be inaudible, long enough to remove the discontinuity. Files: `src/cut.py` 3. Flag cuts that cannot be made cleanly Better to leave a filler in than to leave a click. Files: `src/cut.py` ### Done when - [ ] A cut produces no audible click - [ ] Two hundred cuts accumulate no artefacts - [ ] An unsafe cut is flagged rather than made badly - [ ] Cuts are sample accurate and reproducible ### Watch out - Cutting exactly on a word boundary joins two waveforms at arbitrary amplitudes, which is a discontinuity and therefore an audible click on every single edit, and an episode with two hundred edits is unlistenable. Search a small window around the boundary for a low-energy point and crossfade across a few milliseconds. ## M4 · Several microphones, one timeline Cutting one speaker does not desynchronise the others. ### Steps 1. Align tracks with measured offsets Recording software rarely starts every track at the same instant. Files: `src/align_tracks.py` 2. Apply cuts to the timeline, not a track Detection is per track. Cutting is not. Files: `src/timeline.py`, `tests/test_sync.py` 3. Detect drift and report it Two devices with different clocks drift. Silently misaligning is the worst outcome. Files: `src/drift.py` ### Done when - [ ] A hundred cuts leave all tracks in sync - [ ] Tracks starting at different times align - [ ] Clock drift is detected and reported - [ ] Detection remains per track ### Watch out - Cutting only the track where the filler occurred shortens that track and leaves every other track a word longer, so from that point the episode is out of sync and gets progressively worse with each edit. Cuts are timeline operations applied to all tracks, even though detection is per track. ## M5 · Rendering, and never touching the original (production only) A finished episode renders, and the source recordings are provably untouched. ### Steps 1. Render from the edit list Sources plus intentions. The render is always derivable and never authoritative. Files: `src/render.py` 2. Checksum sources before and after In the suite. A write to a source is an irreversible loss. Files: `tests/test_sources_untouched.py` 3. Normalise to a stated target State the target rather than making the file merely louder. Files: `src/loudness.py` ### Done when - [ ] A render produces a normalised mixed episode - [ ] Source checksums are unchanged, asserted in tests - [ ] The exported edit list re-renders identically - [ ] A hundred cuts render with no discontinuity ### Watch out - Any path that writes to a source file, including a helpfully added normalisation step, destroys an irreplaceable recording, and the loss is discovered when someone wants to re-edit. Open sources read only, checksum them before and after every render, and fail the build if they differ.
OPERATIONS.md
# Operations · Cleanvoice AI ## Backup Source recordings are irreplaceable and large. Back them up before editing anything, since the whole non-destructive design is undermined if the sources themselves are lost. Edit lists are tiny and worth versioning. ## Restore Restore sources and edit lists, re-render, and compare against a kept render. Byte equality proves the pipeline is deterministic and the restore is complete. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring For a local tool the checks that matter run in the suite: source checksums unchanged, word counts preserved across chunk seams, and track alignment after a full edit list. ## Incident checklist An episode that drifts out of sync partway through means a cut was applied to one track only, and the point of divergence identifies the edit. A clicking episode means cut points are landing on boundaries rather than in low-energy windows. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] An hour transcribes with no words lost at chunk seams - [ ] A cut is confirmed to produce no audible click - [ ] A hundred cuts are confirmed to leave all tracks in sync - [ ] Source checksums are confirmed unchanged in the suite - [ ] An exported edit list is confirmed to re-render identically ## Launch constraint Do not market omitted Cleanvoice AI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, opened read only. Nothing in the build writes here. SOURCE_DIR=./sources # Required. Output episodes. Always derivable from sources and the edit list. RENDER_DIR=./renders # Required. Larger models give better timings and take longer. Timing quality matters more than word accuracy here. WHISPER_MODEL=./models/ggml-medium.en.bin # Required. The join length at a cut. Too short clicks, too long smears the speech. CROSSFADE_MS=8 # Required. How far around a word boundary to look for a low-energy point. CUTPOINT_SEARCH_MS=40 # Required. The normalisation target. State it rather than just making the file louder. LOUDNESS_TARGET_LUFS=-16
$ choose a build depth, inspect the files, then open the complete pack in your agent
People still pay for Cleanvoice AI because creators pay to remove fragile audio plumbing and publishing chores from a release schedule. The recurring cost buys codec support, loudness standards, transcription, storage, feeds, analytics, and deliverability to directories, not just the visible interface.
xspecialized cleanup models, cloud processing, and batch speed
xremote studio reliability
xlicensed music libraries
xhosting distribution
xadvanced mastering and support
share on X ↗"I just replaced Cleanvoice AI with one prompt"
Cleanvoice AI pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| pay as you go, 5 hours | custom | — | 5 processing hours total; credits valid for 2 years |
| pay as you go, 10 hours | custom | — | 10 processing hours total; credits valid for 2 years |
| pay as you go, 30 hours | custom | — | 30 processing hours total; credits valid for 2 years |
| subscription, 10 hours | $11 | — | 10 processing hours/month; unused hours roll over up to 30 hours |
| subscription, 30 hours | $30 | — | 30 processing hours/month; unused hours roll over up to 90 hours |
| subscription, 100 hours | $90 | — | 100 processing hours/month; unused hours roll over up to 300 hours |
| custom | custom | — | 200+ processing hours/month with negotiated price and support |
free tierno free tier; trial includes 30 minutes and requires no card
billingmonthly subscriptions plus one-time credit packs; yearly toggle exists but public yearly dollar rates were not exposed
hidden costsMinimum billed unit is 1 minute; one-time credits expire after 2 years, subscription rollover is capped at 3× the plan, and subscription credits expire when the paid term ends.
verified 2026-08-14 · source ↗
Vibecode Cleanvoice AI
Kinda. The core of Cleanvoice AI is buildable in a weekend with the prompt on this page, but there are real gaps: specialized cleanup models, cloud processing, and batch speed, remote studio reliability. Read the honest list above before committing.
How much does Cleanvoice AI cost?
Cleanvoice AI's pricing is usage-based or varies by plan · Vendor pricing is usage or hour based; no stable monthly amount is recorded..
What do I lose by replacing Cleanvoice AI?
Honestly: specialized cleanup models, cloud processing, and batch speed; remote studio reliability; licensed music libraries; hosting distribution; advanced mastering and support. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Cleanvoice AI?
Yes: Audacity (Long-running open-source multitrack audio editor and useful implementation prior art.). Using prior art is also vibecoding; the prompt is for when you want it exactly your way.