Vibecode Descript
track this build5 phases, 15 steps, beginner friendly0%KINDA · weekend project
You can build transcript-based cutting for simple clips, but Descript's editor, media pipeline, AI voice/video tooling, publishing, and collaboration are a serious product.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Descript. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Descript · indie build A tool that transcribes media with word-level timestamps, presents the transcript as the editing surface, turns text deletions into a cut list, plays back the edit without rendering, and only renders through ffmpeg when you export. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timestamps | the timestamps are the entire mechanism; a transcript without them is just text | | Runtime | TypeScript with a desktop shell | local media files, a real editor and no upload | | Preview | Playback that skips cut ranges rather than rendering | rendering on every edit makes the tool unusable; skipping is instant | | Render | ffmpeg with a filter graph built from the cut list | one render at export, from the untouched source | | Storage | The source media untouched, plus a project file | every edit is a list of ranges, so nothing is destroyed and everything is reversible | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the build and the desktop shell. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **ffmpeg with libx264** · free - Why: Conversion for transcription and the final render both go through it. - Get it: brew install ffmpeg, your package manager, or a static build. - Verify: ffmpeg -encoders lists libx264 - [ ] **whisper.cpp with a model that supports word timestamps** · free, but transcription takes real time on your machine - Why: Word-level timestamps are the mechanism. Without them there is no product here. - Get it: Build whisper.cpp and download a model. Confirm the word timestamp output before building anything on it. - Verify: Output contains per-word start and end times - [ ] **Two recordings, one long and one with two speakers** · free - Why: Sync drift, caption timing and speaker handling only reveal themselves at length and with more than one voice. - Get it: Use your own recordings. One over ten minutes and one conversation. - Verify: Both are available and their durations recorded - [ ] **Free disk for renders and converted audio** · free - Why: Every export writes a new file and every source needs a converted audio copy for transcription. - Get it: Estimate several times your source size and check free space. - Verify: Free space comfortably exceeds the estimate ## Quick start ```sh ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A polished nonlinear editor with layers and transitions. - Voice cloning and generated corrections. - Real-time collaborative editing. - The workflow polish of the commercial product. - polished nonlinear editor - overdub/voice tools - filler-word workflows - templates - cloud collaboration - rendering reliability If one of those is essential to you, that is the reason to keep paying for Descript, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Descript The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a transcript-driven editor where deleting a word deletes that piece of the audio and video to replace Descript. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timestamps · the timestamps are the entire mechanism; a transcript without them is just text - Runtime: TypeScript with a desktop shell · local media files, a real editor and no upload - Preview: Playback that skips cut ranges rather than rendering · rendering on every edit makes the tool unusable; skipping is instant - Render: ffmpeg with a filter graph built from the cut list · one render at export, from the untouched source - Storage: The source media untouched, plus a project file · every edit is a list of ranges, so nothing is destroyed and everything is reversible ### Data model (create this before Phase 1) - media(id, path, kind, duration_ms, sample_rate, has_video, content_hash) - words(id, media_id, ordinal, text, start_ms, end_ms, confidence, speaker) · the editing surface - cuts(id, media_id, start_ms, end_ms, reason) · reason is manual, filler or silence - speakers(media_id, label, display_name) - projects(id, name, media_ids_json, created_at) - A cut is a range, never a modification. The source file is read-only for the life of the project. ### Phase 1 · Word-level timestamps Build: Transcribe with word-level timestamps rather than segment-level. Convert audio to the format the transcriber needs. Verify alignment by playing a single word and hearing that word, not the one before or after it. Trap: Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. Done when: Playing the range of a single word from the middle of a sentence plays that word cleanly. Word timestamps are monotonic with no overlaps. A ten minute file transcribes in a time you can state. The source file is unmodified, verified by checksum. Do not build yet: No editing, no playback, no rendering. ### Phase 2 · Deleting text, deleting time Build: Render the transcript as the editing surface. Selecting a span of words and deleting creates a cut from the first word's start to the last word's end. Merge adjacent cuts. Undo restores both the text and the cut. Trap: Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. Done when: Deleting three words removes exactly that audio when previewed. Adjacent cuts merge into one range rather than accumulating. Undo restores the words and removes the cut. A cut at a word boundary with no surrounding silence is flagged as potentially audible. Do not build yet: No preview playback yet, so verification is by inspecting ranges. ### Phase 3 · Preview without rendering Build: Play the source media and skip cut ranges during playback by seeking past them. Keep the transcript scrolling in sync with playback. Handle a cut that lands during a seek without stuttering. Trap: Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. Done when: An edit is audible in preview immediately after it is made, with no render step. Playback skips cut ranges and the transcript scrolls in sync. Several consecutive short cuts play smoothly rather than stuttering through each seek. Scrubbing the transcript seeks the media to that word. Do not build yet: No filler removal or rendering yet. ### Phase 4 · Filler words and silences Build: Detect filler words from the transcript and long silences from the audio. Present both as proposed cuts you review and accept or reject, individually or in bulk. Never apply them automatically. Trap: Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. Done when: Filler detection proposes cuts and applies none until accepted. Accepting all fillers is one action and is undoable in one step. Silence detection uses a threshold and minimum duration you set, and a silence shorter than the minimum is not proposed. Rejecting a proposal keeps it rejected across sessions. Do not build yet: No rendering yet. ### Phase 5 · Rendering the export Build: Build an ffmpeg filter graph from the cut list, trimming and concatenating the kept ranges for both audio and video. Keep them in sync across every cut. Show real progress and never overwrite the source. Trap: Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. Done when: An exported file matches the preview, verified by comparing a passage after several cuts. Audio and video remain in sync at the end of a file with twenty cuts. Export progress reflects real work rather than jumping. The source file is unmodified after export, verified by checksum. Do not build yet: Multi-speaker and export formats come last. ### Phase 6 · Speakers, captions and formats Build: Label speakers and let a transcript be filtered by speaker. Export captions derived from the edited transcript, with timings recomputed after cuts. Add audio-only and social aspect exports. Trap: Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file. Done when: Speaker labels are assigned and the transcript can be filtered by speaker. Exported captions are correctly timed at the end of a file with many cuts, verified by playing the export with the captions loaded. An audio-only export contains the same edit. A vertical export reframes rather than letterboxing. Do not build yet: Nothing after this. ### Out of scope (and why) - The polished nonlinear editor. Once you need layers, transitions and effects, this is a text editor pretending, and a real editor is the right tool. - Voice cloning and generated speech corrections, which are a model and a set of consent questions rather than a feature. - Real-time collaboration on an edit. - The accumulated workflow polish of the commercial product, which is most of what makes it pleasant. ### README must contain - That transcription must produce word-level timestamps, and that segment level cannot support this - That cuts are ranges and the source media is never modified - That preview skips ranges rather than rendering, and export is the only render - That filler and silence removal propose rather than apply - That caption timings are recomputed against the edited timeline ===== AGENTS.md ===== # Agent instructions · Descript indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: whisper.cpp with word-level timestamps, TypeScript with a desktop shell, Playback that skips cut ranges rather than rendering, ffmpeg with a filter graph built from the cut list, The source media untouched, plus a project file. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. ===== BUILD_PLAN.md ===== # Build plan · Descript A tool that transcribes media with word-level timestamps, presents the transcript as the editing surface, turns text deletions into a cut list, plays back the edit without rendering, and only renders through ffmpeg when you export. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Word-level timestamps Every word in the transcript knows exactly when it was said. ### Steps 1. Convert audio for the transcriber Mono 16 kHz 16 bit is what whisper.cpp expects. Convert from the source rather than feeding it the original. ```sh ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` 2. Request word-level timestamps Explicitly. Segment level is the default and it cannot support this product. Files: `src/transcribe.ts` ```sh whisper-cli -m models/ggml-base.en.bin -f audio.wav --output-json-full ``` 3. Verify a single word plays correctly Pick a word from the middle of a sentence, play its range, and listen. This is the test that matters. ### Done when - [ ] A single word's range plays that word cleanly - [ ] Word timestamps are monotonic with no overlaps - [ ] A ten minute file transcribes in a stated time - [ ] The source file checksum is unchanged ### Watch out - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. ## Phase 2 · Deleting text, deleting time Selecting words and pressing delete produces a cut range, and the transcript reflows. ### Steps 1. Make the transcript the editor Words as spans with their timestamps attached. Selection maps to a time range. Files: `src/editor.ts` 2. Create and merge cuts A deletion becomes a range. Adjacent or overlapping ranges merge, or the cut list grows without bound. Files: `src/cuts.ts` 3. Pad into silence where possible Extend slightly into the gap between words. Flag cuts with no gap available. Files: `src/cuts.ts` ### Done when - [ ] Deleting three words produces the corresponding cut range - [ ] Adjacent cuts merge into one - [ ] Undo restores words and removes the cut - [ ] A cut with no surrounding silence is flagged ### Watch out - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. ## Phase 3 · Preview without rendering You hear the edit immediately, without waiting for anything to encode. ### Steps 1. Skip ranges during playback On approaching a cut, seek to its end. Keep a small lookahead so the seek is issued before the boundary. Files: `src/preview.ts` 2. Sync the transcript to playback Highlight the current word. This is also how you notice alignment problems. Files: `src/preview.ts` 3. Handle consecutive cuts Several short cuts in a row must not produce several audible seeks. Merge the playback plan rather than seeking repeatedly. Files: `src/preview.ts` ### Done when - [ ] An edit is audible immediately with no render - [ ] The transcript scrolls in sync with playback - [ ] Consecutive short cuts play smoothly - [ ] Clicking a word seeks the media there ### Watch out - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. ## Phase 4 · Filler words and silences The obvious cleanup is one action, and you can see exactly what it will remove. ### Steps 1. Detect fillers from the transcript A configurable list. Language and speaker specific, so make the list editable rather than fixed. Files: `src/fillers.ts` 2. Detect silences from the audio Threshold and minimum duration, both settings. A pause is not a silence to remove. Files: `src/silence.ts` ```sh ffmpeg -i audio.wav -af silencedetect=n=-35dB:d=0.6 -f null - ``` 3. Propose, never apply Review with counts, bulk accept as one undoable action, and remembered rejections. Files: `src/proposals.ts` ### Done when - [ ] Detection proposes and applies nothing automatically - [ ] Bulk accept is one undoable action - [ ] A silence shorter than the minimum is not proposed - [ ] Rejected proposals stay rejected across sessions ### Watch out - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. ## Phase 5 · Rendering the export The exported file matches the preview, including the video. ### Steps 1. Build the filter graph from cuts Trim and concatenate kept ranges in one ffmpeg invocation for both streams. Files: `src/render.ts` 2. Verify sync after many cuts Twenty cuts in a ten minute file, checked at the end. Drift accumulates and only shows at length. 3. Never write to the source Output alongside, checksum the source before and after as a test. Files: `src/render.test.ts` ### Done when - [ ] The export matches the preview after several cuts - [ ] Audio and video stay in sync after twenty cuts - [ ] Progress reflects real work - [ ] The source is unmodified by checksum ### Watch out - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. ## Not in this build - A polished nonlinear editor with layers and transitions. - Voice cloning and generated corrections. - Real-time collaborative editing. - The workflow polish of the commercial product. ## After v1, if you want it - Automatic speaker diarisation to seed the labels you then correct - A word-level search across a library of transcripts - Chapter markers derived from the edited transcript ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. The transcription model. Larger is more accurate and slower. WHISPER_MODEL_PATH=./models/ggml-base.en.bin # Required. Where renders are written. Never the source directory. OUTPUT_DIR=./exports # Required. How far a cut extends into surrounding silence, to avoid clicks at the edit. CUT_PAD_MS=40 # Required. Level below which audio counts as silence for detection. SILENCE_THRESHOLD_DB=-35 # Required. Minimum silence length before it is proposed as a cut. A pause is not a silence to remove. SILENCE_MIN_MS=600 # Required. The filler list, editable because it is language and speaker specific. FILLER_WORDS=um,uh,like,you know
You are building a lean indie version of Descript. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Descript · indie build A tool that transcribes media with word-level timestamps, presents the transcript as the editing surface, turns text deletions into a cut list, plays back the edit without rendering, and only renders through ffmpeg when you export. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timestamps | the timestamps are the entire mechanism; a transcript without them is just text | | Runtime | TypeScript with a desktop shell | local media files, a real editor and no upload | | Preview | Playback that skips cut ranges rather than rendering | rendering on every edit makes the tool unusable; skipping is instant | | Render | ffmpeg with a filter graph built from the cut list | one render at export, from the untouched source | | Storage | The source media untouched, plus a project file | every edit is a list of ranges, so nothing is destroyed and everything is reversible | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the build and the desktop shell. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **ffmpeg with libx264** · free - Why: Conversion for transcription and the final render both go through it. - Get it: brew install ffmpeg, your package manager, or a static build. - Verify: ffmpeg -encoders lists libx264 - [ ] **whisper.cpp with a model that supports word timestamps** · free, but transcription takes real time on your machine - Why: Word-level timestamps are the mechanism. Without them there is no product here. - Get it: Build whisper.cpp and download a model. Confirm the word timestamp output before building anything on it. - Verify: Output contains per-word start and end times - [ ] **Two recordings, one long and one with two speakers** · free - Why: Sync drift, caption timing and speaker handling only reveal themselves at length and with more than one voice. - Get it: Use your own recordings. One over ten minutes and one conversation. - Verify: Both are available and their durations recorded - [ ] **Free disk for renders and converted audio** · free - Why: Every export writes a new file and every source needs a converted audio copy for transcription. - Get it: Estimate several times your source size and check free space. - Verify: Free space comfortably exceeds the estimate ## Quick start ```sh ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A polished nonlinear editor with layers and transitions. - Voice cloning and generated corrections. - Real-time collaborative editing. - The workflow polish of the commercial product. - polished nonlinear editor - overdub/voice tools - filler-word workflows - templates - cloud collaboration - rendering reliability If one of those is essential to you, that is the reason to keep paying for Descript, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Descript The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a transcript-driven editor where deleting a word deletes that piece of the audio and video to replace Descript. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timestamps · the timestamps are the entire mechanism; a transcript without them is just text - Runtime: TypeScript with a desktop shell · local media files, a real editor and no upload - Preview: Playback that skips cut ranges rather than rendering · rendering on every edit makes the tool unusable; skipping is instant - Render: ffmpeg with a filter graph built from the cut list · one render at export, from the untouched source - Storage: The source media untouched, plus a project file · every edit is a list of ranges, so nothing is destroyed and everything is reversible ### Data model (create this before Phase 1) - media(id, path, kind, duration_ms, sample_rate, has_video, content_hash) - words(id, media_id, ordinal, text, start_ms, end_ms, confidence, speaker) · the editing surface - cuts(id, media_id, start_ms, end_ms, reason) · reason is manual, filler or silence - speakers(media_id, label, display_name) - projects(id, name, media_ids_json, created_at) - A cut is a range, never a modification. The source file is read-only for the life of the project. ### Phase 1 · Word-level timestamps Build: Transcribe with word-level timestamps rather than segment-level. Convert audio to the format the transcriber needs. Verify alignment by playing a single word and hearing that word, not the one before or after it. Trap: Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. Done when: Playing the range of a single word from the middle of a sentence plays that word cleanly. Word timestamps are monotonic with no overlaps. A ten minute file transcribes in a time you can state. The source file is unmodified, verified by checksum. Do not build yet: No editing, no playback, no rendering. ### Phase 2 · Deleting text, deleting time Build: Render the transcript as the editing surface. Selecting a span of words and deleting creates a cut from the first word's start to the last word's end. Merge adjacent cuts. Undo restores both the text and the cut. Trap: Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. Done when: Deleting three words removes exactly that audio when previewed. Adjacent cuts merge into one range rather than accumulating. Undo restores the words and removes the cut. A cut at a word boundary with no surrounding silence is flagged as potentially audible. Do not build yet: No preview playback yet, so verification is by inspecting ranges. ### Phase 3 · Preview without rendering Build: Play the source media and skip cut ranges during playback by seeking past them. Keep the transcript scrolling in sync with playback. Handle a cut that lands during a seek without stuttering. Trap: Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. Done when: An edit is audible in preview immediately after it is made, with no render step. Playback skips cut ranges and the transcript scrolls in sync. Several consecutive short cuts play smoothly rather than stuttering through each seek. Scrubbing the transcript seeks the media to that word. Do not build yet: No filler removal or rendering yet. ### Phase 4 · Filler words and silences Build: Detect filler words from the transcript and long silences from the audio. Present both as proposed cuts you review and accept or reject, individually or in bulk. Never apply them automatically. Trap: Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. Done when: Filler detection proposes cuts and applies none until accepted. Accepting all fillers is one action and is undoable in one step. Silence detection uses a threshold and minimum duration you set, and a silence shorter than the minimum is not proposed. Rejecting a proposal keeps it rejected across sessions. Do not build yet: No rendering yet. ### Phase 5 · Rendering the export Build: Build an ffmpeg filter graph from the cut list, trimming and concatenating the kept ranges for both audio and video. Keep them in sync across every cut. Show real progress and never overwrite the source. Trap: Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. Done when: An exported file matches the preview, verified by comparing a passage after several cuts. Audio and video remain in sync at the end of a file with twenty cuts. Export progress reflects real work rather than jumping. The source file is unmodified after export, verified by checksum. Do not build yet: Multi-speaker and export formats come last. ### Phase 6 · Speakers, captions and formats Build: Label speakers and let a transcript be filtered by speaker. Export captions derived from the edited transcript, with timings recomputed after cuts. Add audio-only and social aspect exports. Trap: Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file. Done when: Speaker labels are assigned and the transcript can be filtered by speaker. Exported captions are correctly timed at the end of a file with many cuts, verified by playing the export with the captions loaded. An audio-only export contains the same edit. A vertical export reframes rather than letterboxing. Do not build yet: Nothing after this. ### Out of scope (and why) - The polished nonlinear editor. Once you need layers, transitions and effects, this is a text editor pretending, and a real editor is the right tool. - Voice cloning and generated speech corrections, which are a model and a set of consent questions rather than a feature. - Real-time collaboration on an edit. - The accumulated workflow polish of the commercial product, which is most of what makes it pleasant. ### README must contain - That transcription must produce word-level timestamps, and that segment level cannot support this - That cuts are ranges and the source media is never modified - That preview skips ranges rather than rendering, and export is the only render - That filler and silence removal propose rather than apply - That caption timings are recomputed against the edited timeline ===== AGENTS.md ===== # Agent instructions · Descript indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: whisper.cpp with word-level timestamps, TypeScript with a desktop shell, Playback that skips cut ranges rather than rendering, ffmpeg with a filter graph built from the cut list, The source media untouched, plus a project file. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. ===== BUILD_PLAN.md ===== # Build plan · Descript A tool that transcribes media with word-level timestamps, presents the transcript as the editing surface, turns text deletions into a cut list, plays back the edit without rendering, and only renders through ffmpeg when you export. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Word-level timestamps Every word in the transcript knows exactly when it was said. ### Steps 1. Convert audio for the transcriber Mono 16 kHz 16 bit is what whisper.cpp expects. Convert from the source rather than feeding it the original. ```sh ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` 2. Request word-level timestamps Explicitly. Segment level is the default and it cannot support this product. Files: `src/transcribe.ts` ```sh whisper-cli -m models/ggml-base.en.bin -f audio.wav --output-json-full ``` 3. Verify a single word plays correctly Pick a word from the middle of a sentence, play its range, and listen. This is the test that matters. ### Done when - [ ] A single word's range plays that word cleanly - [ ] Word timestamps are monotonic with no overlaps - [ ] A ten minute file transcribes in a stated time - [ ] The source file checksum is unchanged ### Watch out - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. ## Phase 2 · Deleting text, deleting time Selecting words and pressing delete produces a cut range, and the transcript reflows. ### Steps 1. Make the transcript the editor Words as spans with their timestamps attached. Selection maps to a time range. Files: `src/editor.ts` 2. Create and merge cuts A deletion becomes a range. Adjacent or overlapping ranges merge, or the cut list grows without bound. Files: `src/cuts.ts` 3. Pad into silence where possible Extend slightly into the gap between words. Flag cuts with no gap available. Files: `src/cuts.ts` ### Done when - [ ] Deleting three words produces the corresponding cut range - [ ] Adjacent cuts merge into one - [ ] Undo restores words and removes the cut - [ ] A cut with no surrounding silence is flagged ### Watch out - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. ## Phase 3 · Preview without rendering You hear the edit immediately, without waiting for anything to encode. ### Steps 1. Skip ranges during playback On approaching a cut, seek to its end. Keep a small lookahead so the seek is issued before the boundary. Files: `src/preview.ts` 2. Sync the transcript to playback Highlight the current word. This is also how you notice alignment problems. Files: `src/preview.ts` 3. Handle consecutive cuts Several short cuts in a row must not produce several audible seeks. Merge the playback plan rather than seeking repeatedly. Files: `src/preview.ts` ### Done when - [ ] An edit is audible immediately with no render - [ ] The transcript scrolls in sync with playback - [ ] Consecutive short cuts play smoothly - [ ] Clicking a word seeks the media there ### Watch out - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. ## Phase 4 · Filler words and silences The obvious cleanup is one action, and you can see exactly what it will remove. ### Steps 1. Detect fillers from the transcript A configurable list. Language and speaker specific, so make the list editable rather than fixed. Files: `src/fillers.ts` 2. Detect silences from the audio Threshold and minimum duration, both settings. A pause is not a silence to remove. Files: `src/silence.ts` ```sh ffmpeg -i audio.wav -af silencedetect=n=-35dB:d=0.6 -f null - ``` 3. Propose, never apply Review with counts, bulk accept as one undoable action, and remembered rejections. Files: `src/proposals.ts` ### Done when - [ ] Detection proposes and applies nothing automatically - [ ] Bulk accept is one undoable action - [ ] A silence shorter than the minimum is not proposed - [ ] Rejected proposals stay rejected across sessions ### Watch out - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. ## Phase 5 · Rendering the export The exported file matches the preview, including the video. ### Steps 1. Build the filter graph from cuts Trim and concatenate kept ranges in one ffmpeg invocation for both streams. Files: `src/render.ts` 2. Verify sync after many cuts Twenty cuts in a ten minute file, checked at the end. Drift accumulates and only shows at length. 3. Never write to the source Output alongside, checksum the source before and after as a test. Files: `src/render.test.ts` ### Done when - [ ] The export matches the preview after several cuts - [ ] Audio and video stay in sync after twenty cuts - [ ] Progress reflects real work - [ ] The source is unmodified by checksum ### Watch out - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. ## Not in this build - A polished nonlinear editor with layers and transitions. - Voice cloning and generated corrections. - Real-time collaborative editing. - The workflow polish of the commercial product. ## After v1, if you want it - Automatic speaker diarisation to seed the labels you then correct - A word-level search across a library of transcripts - Chapter markers derived from the edited transcript ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. The transcription model. Larger is more accurate and slower. WHISPER_MODEL_PATH=./models/ggml-base.en.bin # Required. Where renders are written. Never the source directory. OUTPUT_DIR=./exports # Required. How far a cut extends into surrounding silence, to avoid clicks at the edit. CUT_PAD_MS=40 # Required. Level below which audio counts as silence for detection. SILENCE_THRESHOLD_DB=-35 # Required. Minimum silence length before it is proposed as a cut. A pause is not a silence to remove. SILENCE_MIN_MS=600 # Required. The filler list, editable because it is language and speaker specific. FILLER_WORDS=um,uh,like,you know
You are building a production product version of Descript. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Descript · product brief ## Problem You can build transcript-based cutting for simple clips, but Descript's editor, media pipeline, AI voice/video tooling, publishing, and collaboration are a serious product. ## Product outcome An editor where cutting text cuts media, previews are instant because nothing is rendered until export, cleanup is proposed rather than applied, and the source file is never touched. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - ffmpeg - speech-to-text/alignment - desktop or web editor - video rendering pipeline - optional hosted storage ## Explicit non-goals for v1 - A polished nonlinear editor with layers and transitions. - Voice cloning and generated corrections. - Real-time collaborative editing. - The workflow polish of the commercial product. - polished nonlinear editor - overdub/voice tools - filler-word workflows - templates - cloud collaboration - rendering reliability ## Success criteria - Single word playback confirms alignment on both fixtures - Audio and video sync is verified after twenty cuts - Caption timings are verified at the end of a heavily edited file - Source checksums are unchanged after export - Preview is confirmed to require no render step ===== BRIEF.md ===== # Build brief · Descript The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a transcript-driven editor where deleting a word deletes that piece of the audio and video to replace Descript. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timestamps · the timestamps are the entire mechanism; a transcript without them is just text - Runtime: TypeScript with a desktop shell · local media files, a real editor and no upload - Preview: Playback that skips cut ranges rather than rendering · rendering on every edit makes the tool unusable; skipping is instant - Render: ffmpeg with a filter graph built from the cut list · one render at export, from the untouched source - Storage: The source media untouched, plus a project file · every edit is a list of ranges, so nothing is destroyed and everything is reversible ### Data model (create this before Phase 1) - media(id, path, kind, duration_ms, sample_rate, has_video, content_hash) - words(id, media_id, ordinal, text, start_ms, end_ms, confidence, speaker) · the editing surface - cuts(id, media_id, start_ms, end_ms, reason) · reason is manual, filler or silence - speakers(media_id, label, display_name) - projects(id, name, media_ids_json, created_at) - A cut is a range, never a modification. The source file is read-only for the life of the project. ### Phase 1 · Word-level timestamps Build: Transcribe with word-level timestamps rather than segment-level. Convert audio to the format the transcriber needs. Verify alignment by playing a single word and hearing that word, not the one before or after it. Trap: Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. Done when: Playing the range of a single word from the middle of a sentence plays that word cleanly. Word timestamps are monotonic with no overlaps. A ten minute file transcribes in a time you can state. The source file is unmodified, verified by checksum. Do not build yet: No editing, no playback, no rendering. ### Phase 2 · Deleting text, deleting time Build: Render the transcript as the editing surface. Selecting a span of words and deleting creates a cut from the first word's start to the last word's end. Merge adjacent cuts. Undo restores both the text and the cut. Trap: Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. Done when: Deleting three words removes exactly that audio when previewed. Adjacent cuts merge into one range rather than accumulating. Undo restores the words and removes the cut. A cut at a word boundary with no surrounding silence is flagged as potentially audible. Do not build yet: No preview playback yet, so verification is by inspecting ranges. ### Phase 3 · Preview without rendering Build: Play the source media and skip cut ranges during playback by seeking past them. Keep the transcript scrolling in sync with playback. Handle a cut that lands during a seek without stuttering. Trap: Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. Done when: An edit is audible in preview immediately after it is made, with no render step. Playback skips cut ranges and the transcript scrolls in sync. Several consecutive short cuts play smoothly rather than stuttering through each seek. Scrubbing the transcript seeks the media to that word. Do not build yet: No filler removal or rendering yet. ### Phase 4 · Filler words and silences Build: Detect filler words from the transcript and long silences from the audio. Present both as proposed cuts you review and accept or reject, individually or in bulk. Never apply them automatically. Trap: Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. Done when: Filler detection proposes cuts and applies none until accepted. Accepting all fillers is one action and is undoable in one step. Silence detection uses a threshold and minimum duration you set, and a silence shorter than the minimum is not proposed. Rejecting a proposal keeps it rejected across sessions. Do not build yet: No rendering yet. ### Phase 5 · Rendering the export Build: Build an ffmpeg filter graph from the cut list, trimming and concatenating the kept ranges for both audio and video. Keep them in sync across every cut. Show real progress and never overwrite the source. Trap: Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. Done when: An exported file matches the preview, verified by comparing a passage after several cuts. Audio and video remain in sync at the end of a file with twenty cuts. Export progress reflects real work rather than jumping. The source file is unmodified after export, verified by checksum. Do not build yet: Multi-speaker and export formats come last. ### Phase 6 · Speakers, captions and formats Build: Label speakers and let a transcript be filtered by speaker. Export captions derived from the edited transcript, with timings recomputed after cuts. Add audio-only and social aspect exports. Trap: Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file. Done when: Speaker labels are assigned and the transcript can be filtered by speaker. Exported captions are correctly timed at the end of a file with many cuts, verified by playing the export with the captions loaded. An audio-only export contains the same edit. A vertical export reframes rather than letterboxing. Do not build yet: Nothing after this. ### Out of scope (and why) - The polished nonlinear editor. Once you need layers, transitions and effects, this is a text editor pretending, and a real editor is the right tool. - Voice cloning and generated speech corrections, which are a model and a set of consent questions rather than a feature. - Real-time collaboration on an edit. - The accumulated workflow polish of the commercial product, which is most of what makes it pleasant. ### README must contain - That transcription must produce word-level timestamps, and that segment level cannot support this - That cuts are ranges and the source media is never modified - That preview skips ranges rather than rendering, and export is the only render - That filler and silence removal propose rather than apply - That caption timings are recomputed against the edited timeline ===== ARCHITECTURE.md ===== # Architecture · Descript ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timestamps | the timestamps are the entire mechanism; a transcript without them is just text | | Runtime | TypeScript with a desktop shell | local media files, a real editor and no upload | | Preview | Playback that skips cut ranges rather than rendering | rendering on every edit makes the tool unusable; skipping is instant | | Render | ffmpeg with a filter graph built from the cut list | one render at export, from the untouched source | | Storage | The source media untouched, plus a project file | every edit is a list of ranges, so nothing is destroyed and everything is reversible | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Transcribe | Words and their timestamps. | A hosted transcription service fits the same interface and gives up the offline property. Word-level output is non-negotiable either way. | | Cuts | Ranges, merging and padding. | Pure data over time ranges, testable with no media at all. | | Preview | Playback that skips ranges. | The no-render rule is what makes the tool usable and must survive any change. | | Render | Building the filter graph and exporting. | ffmpeg is the practical choice. Single-pass concatenation rather than per-cut files is the performance decision. | | Proposals | Filler and silence suggestions. | Detection can improve; proposing rather than applying is a product commitment. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `WHISPER_MODEL_PATH` · required · The transcription model. Larger is more accurate and slower. - `OUTPUT_DIR` · required · Where renders are written. Never the source directory. - `CUT_PAD_MS` · required · How far a cut extends into surrounding silence, to avoid clicks at the edit. - `SILENCE_THRESHOLD_DB` · required · Level below which audio counts as silence for detection. - `SILENCE_MIN_MS` · required · Minimum silence length before it is proposed as a cut. A pause is not a silence to remove. - `FILLER_WORDS` · required · The filler list, editable because it is language and speaker specific. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Descript product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: whisper.cpp with word-level timestamps, TypeScript with a desktop shell, Playback that skips cut ranges rather than rendering, ffmpeg with a filter graph built from the cut list, The source media untouched, plus a project file. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. - Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file. ===== MILESTONES.md ===== # Delivery milestones · Descript Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Word-level timestamps Every word in the transcript knows exactly when it was said. ### Steps 1. Convert audio for the transcriber Mono 16 kHz 16 bit is what whisper.cpp expects. Convert from the source rather than feeding it the original. ```sh ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` 2. Request word-level timestamps Explicitly. Segment level is the default and it cannot support this product. Files: `src/transcribe.ts` ```sh whisper-cli -m models/ggml-base.en.bin -f audio.wav --output-json-full ``` 3. Verify a single word plays correctly Pick a word from the middle of a sentence, play its range, and listen. This is the test that matters. ### Done when - [ ] A single word's range plays that word cleanly - [ ] Word timestamps are monotonic with no overlaps - [ ] A ten minute file transcribes in a stated time - [ ] The source file checksum is unchanged ### Watch out - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. ## M2 · Deleting text, deleting time Selecting words and pressing delete produces a cut range, and the transcript reflows. ### Steps 1. Make the transcript the editor Words as spans with their timestamps attached. Selection maps to a time range. Files: `src/editor.ts` 2. Create and merge cuts A deletion becomes a range. Adjacent or overlapping ranges merge, or the cut list grows without bound. Files: `src/cuts.ts` 3. Pad into silence where possible Extend slightly into the gap between words. Flag cuts with no gap available. Files: `src/cuts.ts` ### Done when - [ ] Deleting three words produces the corresponding cut range - [ ] Adjacent cuts merge into one - [ ] Undo restores words and removes the cut - [ ] A cut with no surrounding silence is flagged ### Watch out - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. ## M3 · Preview without rendering You hear the edit immediately, without waiting for anything to encode. ### Steps 1. Skip ranges during playback On approaching a cut, seek to its end. Keep a small lookahead so the seek is issued before the boundary. Files: `src/preview.ts` 2. Sync the transcript to playback Highlight the current word. This is also how you notice alignment problems. Files: `src/preview.ts` 3. Handle consecutive cuts Several short cuts in a row must not produce several audible seeks. Merge the playback plan rather than seeking repeatedly. Files: `src/preview.ts` ### Done when - [ ] An edit is audible immediately with no render - [ ] The transcript scrolls in sync with playback - [ ] Consecutive short cuts play smoothly - [ ] Clicking a word seeks the media there ### Watch out - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. ## M4 · Filler words and silences The obvious cleanup is one action, and you can see exactly what it will remove. ### Steps 1. Detect fillers from the transcript A configurable list. Language and speaker specific, so make the list editable rather than fixed. Files: `src/fillers.ts` 2. Detect silences from the audio Threshold and minimum duration, both settings. A pause is not a silence to remove. Files: `src/silence.ts` ```sh ffmpeg -i audio.wav -af silencedetect=n=-35dB:d=0.6 -f null - ``` 3. Propose, never apply Review with counts, bulk accept as one undoable action, and remembered rejections. Files: `src/proposals.ts` ### Done when - [ ] Detection proposes and applies nothing automatically - [ ] Bulk accept is one undoable action - [ ] A silence shorter than the minimum is not proposed - [ ] Rejected proposals stay rejected across sessions ### Watch out - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. ## M5 · Rendering the export The exported file matches the preview, including the video. ### Steps 1. Build the filter graph from cuts Trim and concatenate kept ranges in one ffmpeg invocation for both streams. Files: `src/render.ts` 2. Verify sync after many cuts Twenty cuts in a ten minute file, checked at the end. Drift accumulates and only shows at length. 3. Never write to the source Output alongside, checksum the source before and after as a test. Files: `src/render.test.ts` ### Done when - [ ] The export matches the preview after several cuts - [ ] Audio and video stay in sync after twenty cuts - [ ] Progress reflects real work - [ ] The source is unmodified by checksum ### Watch out - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. ## M6 · Speakers, captions and formats (production only) Multi-speaker recordings are usable and the output carries its transcript. ### Steps 1. Assign and filter by speaker Labels applied to word ranges, correctable by hand since automatic assignment will be imperfect. Files: `src/speakers.ts` 2. Recompute caption timings Against the edited timeline, not the original. Check a caption near the end of a heavily cut file. Files: `src/captions.ts` 3. Add export variants Audio only and a vertical crop. Both reuse the same cut list. Files: `src/exports.ts` ### Done when - [ ] Speaker labels are assignable and filterable - [ ] Captions are correctly timed at the end of a heavily edited file - [ ] An audio-only export carries the same edit - [ ] A vertical export reframes rather than letterboxes ### Watch out - Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file. ===== OPERATIONS.md ===== # Operations · Descript ## Backup Source media is irreplaceable and belongs in a backup. Project files are small and hold the cut list, which is the actual work. Renders are regenerable from both. ## Restore Restore sources and project files, then re-export. Because cuts are ranges over unmodified sources, a restore reproduces the edit exactly rather than approximately. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring For a personal tool, watch transcription and render times against media length. A render slower than real time by a large factor usually means per-cut file operations rather than a single filter graph. ## Incident checklist The realistic failure is a transcript misaligned with the media, which makes every cut wrong. Verify alignment on import rather than discovering it mid-edit, which is what the single word playback check in Phase 1 is for. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Single word playback confirms alignment on both fixtures - [ ] Audio and video sync is verified after twenty cuts - [ ] Caption timings are verified at the end of a heavily edited file - [ ] Source checksums are unchanged after export - [ ] Preview is confirmed to require no render step ## Launch constraint Do not market omitted Descript capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. The transcription model. Larger is more accurate and slower. WHISPER_MODEL_PATH=./models/ggml-base.en.bin # Required. Where renders are written. Never the source directory. OUTPUT_DIR=./exports # Required. How far a cut extends into surrounding silence, to avoid clicks at the edit. CUT_PAD_MS=40 # Required. Level below which audio counts as silence for detection. SILENCE_THRESHOLD_DB=-35 # Required. Minimum silence length before it is proposed as a cut. A pause is not a silence to remove. SILENCE_MIN_MS=600 # Required. The filler list, editable because it is language and speaker specific. FILLER_WORDS=um,uh,like,you know
README.md
# Descript · indie build A tool that transcribes media with word-level timestamps, presents the transcript as the editing surface, turns text deletions into a cut list, plays back the edit without rendering, and only renders through ffmpeg when you export. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timestamps | the timestamps are the entire mechanism; a transcript without them is just text | | Runtime | TypeScript with a desktop shell | local media files, a real editor and no upload | | Preview | Playback that skips cut ranges rather than rendering | rendering on every edit makes the tool unusable; skipping is instant | | Render | ffmpeg with a filter graph built from the cut list | one render at export, from the untouched source | | Storage | The source media untouched, plus a project file | every edit is a list of ranges, so nothing is destroyed and everything is reversible | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the build and the desktop shell. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **ffmpeg with libx264** · free - Why: Conversion for transcription and the final render both go through it. - Get it: brew install ffmpeg, your package manager, or a static build. - Verify: ffmpeg -encoders lists libx264 - [ ] **whisper.cpp with a model that supports word timestamps** · free, but transcription takes real time on your machine - Why: Word-level timestamps are the mechanism. Without them there is no product here. - Get it: Build whisper.cpp and download a model. Confirm the word timestamp output before building anything on it. - Verify: Output contains per-word start and end times - [ ] **Two recordings, one long and one with two speakers** · free - Why: Sync drift, caption timing and speaker handling only reveal themselves at length and with more than one voice. - Get it: Use your own recordings. One over ten minutes and one conversation. - Verify: Both are available and their durations recorded - [ ] **Free disk for renders and converted audio** · free - Why: Every export writes a new file and every source needs a converted audio copy for transcription. - Get it: Estimate several times your source size and check free space. - Verify: Free space comfortably exceeds the estimate ## Quick start ```sh ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A polished nonlinear editor with layers and transitions. - Voice cloning and generated corrections. - Real-time collaborative editing. - The workflow polish of the commercial product. - polished nonlinear editor - overdub/voice tools - filler-word workflows - templates - cloud collaboration - rendering reliability If one of those is essential to you, that is the reason to keep paying for Descript, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Descript The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a transcript-driven editor where deleting a word deletes that piece of the audio and video to replace Descript. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timestamps · the timestamps are the entire mechanism; a transcript without them is just text - Runtime: TypeScript with a desktop shell · local media files, a real editor and no upload - Preview: Playback that skips cut ranges rather than rendering · rendering on every edit makes the tool unusable; skipping is instant - Render: ffmpeg with a filter graph built from the cut list · one render at export, from the untouched source - Storage: The source media untouched, plus a project file · every edit is a list of ranges, so nothing is destroyed and everything is reversible ### Data model (create this before Phase 1) - media(id, path, kind, duration_ms, sample_rate, has_video, content_hash) - words(id, media_id, ordinal, text, start_ms, end_ms, confidence, speaker) · the editing surface - cuts(id, media_id, start_ms, end_ms, reason) · reason is manual, filler or silence - speakers(media_id, label, display_name) - projects(id, name, media_ids_json, created_at) - A cut is a range, never a modification. The source file is read-only for the life of the project. ### Phase 1 · Word-level timestamps Build: Transcribe with word-level timestamps rather than segment-level. Convert audio to the format the transcriber needs. Verify alignment by playing a single word and hearing that word, not the one before or after it. Trap: Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. Done when: Playing the range of a single word from the middle of a sentence plays that word cleanly. Word timestamps are monotonic with no overlaps. A ten minute file transcribes in a time you can state. The source file is unmodified, verified by checksum. Do not build yet: No editing, no playback, no rendering. ### Phase 2 · Deleting text, deleting time Build: Render the transcript as the editing surface. Selecting a span of words and deleting creates a cut from the first word's start to the last word's end. Merge adjacent cuts. Undo restores both the text and the cut. Trap: Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. Done when: Deleting three words removes exactly that audio when previewed. Adjacent cuts merge into one range rather than accumulating. Undo restores the words and removes the cut. A cut at a word boundary with no surrounding silence is flagged as potentially audible. Do not build yet: No preview playback yet, so verification is by inspecting ranges. ### Phase 3 · Preview without rendering Build: Play the source media and skip cut ranges during playback by seeking past them. Keep the transcript scrolling in sync with playback. Handle a cut that lands during a seek without stuttering. Trap: Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. Done when: An edit is audible in preview immediately after it is made, with no render step. Playback skips cut ranges and the transcript scrolls in sync. Several consecutive short cuts play smoothly rather than stuttering through each seek. Scrubbing the transcript seeks the media to that word. Do not build yet: No filler removal or rendering yet. ### Phase 4 · Filler words and silences Build: Detect filler words from the transcript and long silences from the audio. Present both as proposed cuts you review and accept or reject, individually or in bulk. Never apply them automatically. Trap: Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. Done when: Filler detection proposes cuts and applies none until accepted. Accepting all fillers is one action and is undoable in one step. Silence detection uses a threshold and minimum duration you set, and a silence shorter than the minimum is not proposed. Rejecting a proposal keeps it rejected across sessions. Do not build yet: No rendering yet. ### Phase 5 · Rendering the export Build: Build an ffmpeg filter graph from the cut list, trimming and concatenating the kept ranges for both audio and video. Keep them in sync across every cut. Show real progress and never overwrite the source. Trap: Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. Done when: An exported file matches the preview, verified by comparing a passage after several cuts. Audio and video remain in sync at the end of a file with twenty cuts. Export progress reflects real work rather than jumping. The source file is unmodified after export, verified by checksum. Do not build yet: Multi-speaker and export formats come last. ### Phase 6 · Speakers, captions and formats Build: Label speakers and let a transcript be filtered by speaker. Export captions derived from the edited transcript, with timings recomputed after cuts. Add audio-only and social aspect exports. Trap: Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file. Done when: Speaker labels are assigned and the transcript can be filtered by speaker. Exported captions are correctly timed at the end of a file with many cuts, verified by playing the export with the captions loaded. An audio-only export contains the same edit. A vertical export reframes rather than letterboxing. Do not build yet: Nothing after this. ### Out of scope (and why) - The polished nonlinear editor. Once you need layers, transitions and effects, this is a text editor pretending, and a real editor is the right tool. - Voice cloning and generated speech corrections, which are a model and a set of consent questions rather than a feature. - Real-time collaboration on an edit. - The accumulated workflow polish of the commercial product, which is most of what makes it pleasant. ### README must contain - That transcription must produce word-level timestamps, and that segment level cannot support this - That cuts are ranges and the source media is never modified - That preview skips ranges rather than rendering, and export is the only render - That filler and silence removal propose rather than apply - That caption timings are recomputed against the edited timeline
AGENTS.md
# Agent instructions · Descript indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: whisper.cpp with word-level timestamps, TypeScript with a desktop shell, Playback that skips cut ranges rather than rendering, ffmpeg with a filter graph built from the cut list, The source media untouched, plus a project file. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts.
BUILD_PLAN.md
# Build plan · Descript A tool that transcribes media with word-level timestamps, presents the transcript as the editing surface, turns text deletions into a cut list, plays back the edit without rendering, and only renders through ffmpeg when you export. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Word-level timestamps Every word in the transcript knows exactly when it was said. ### Steps 1. Convert audio for the transcriber Mono 16 kHz 16 bit is what whisper.cpp expects. Convert from the source rather than feeding it the original. ```sh ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` 2. Request word-level timestamps Explicitly. Segment level is the default and it cannot support this product. Files: `src/transcribe.ts` ```sh whisper-cli -m models/ggml-base.en.bin -f audio.wav --output-json-full ``` 3. Verify a single word plays correctly Pick a word from the middle of a sentence, play its range, and listen. This is the test that matters. ### Done when - [ ] A single word's range plays that word cleanly - [ ] Word timestamps are monotonic with no overlaps - [ ] A ten minute file transcribes in a stated time - [ ] The source file checksum is unchanged ### Watch out - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. ## Phase 2 · Deleting text, deleting time Selecting words and pressing delete produces a cut range, and the transcript reflows. ### Steps 1. Make the transcript the editor Words as spans with their timestamps attached. Selection maps to a time range. Files: `src/editor.ts` 2. Create and merge cuts A deletion becomes a range. Adjacent or overlapping ranges merge, or the cut list grows without bound. Files: `src/cuts.ts` 3. Pad into silence where possible Extend slightly into the gap between words. Flag cuts with no gap available. Files: `src/cuts.ts` ### Done when - [ ] Deleting three words produces the corresponding cut range - [ ] Adjacent cuts merge into one - [ ] Undo restores words and removes the cut - [ ] A cut with no surrounding silence is flagged ### Watch out - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. ## Phase 3 · Preview without rendering You hear the edit immediately, without waiting for anything to encode. ### Steps 1. Skip ranges during playback On approaching a cut, seek to its end. Keep a small lookahead so the seek is issued before the boundary. Files: `src/preview.ts` 2. Sync the transcript to playback Highlight the current word. This is also how you notice alignment problems. Files: `src/preview.ts` 3. Handle consecutive cuts Several short cuts in a row must not produce several audible seeks. Merge the playback plan rather than seeking repeatedly. Files: `src/preview.ts` ### Done when - [ ] An edit is audible immediately with no render - [ ] The transcript scrolls in sync with playback - [ ] Consecutive short cuts play smoothly - [ ] Clicking a word seeks the media there ### Watch out - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. ## Phase 4 · Filler words and silences The obvious cleanup is one action, and you can see exactly what it will remove. ### Steps 1. Detect fillers from the transcript A configurable list. Language and speaker specific, so make the list editable rather than fixed. Files: `src/fillers.ts` 2. Detect silences from the audio Threshold and minimum duration, both settings. A pause is not a silence to remove. Files: `src/silence.ts` ```sh ffmpeg -i audio.wav -af silencedetect=n=-35dB:d=0.6 -f null - ``` 3. Propose, never apply Review with counts, bulk accept as one undoable action, and remembered rejections. Files: `src/proposals.ts` ### Done when - [ ] Detection proposes and applies nothing automatically - [ ] Bulk accept is one undoable action - [ ] A silence shorter than the minimum is not proposed - [ ] Rejected proposals stay rejected across sessions ### Watch out - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. ## Phase 5 · Rendering the export The exported file matches the preview, including the video. ### Steps 1. Build the filter graph from cuts Trim and concatenate kept ranges in one ffmpeg invocation for both streams. Files: `src/render.ts` 2. Verify sync after many cuts Twenty cuts in a ten minute file, checked at the end. Drift accumulates and only shows at length. 3. Never write to the source Output alongside, checksum the source before and after as a test. Files: `src/render.test.ts` ### Done when - [ ] The export matches the preview after several cuts - [ ] Audio and video stay in sync after twenty cuts - [ ] Progress reflects real work - [ ] The source is unmodified by checksum ### Watch out - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. ## Not in this build - A polished nonlinear editor with layers and transitions. - Voice cloning and generated corrections. - Real-time collaborative editing. - The workflow polish of the commercial product. ## After v1, if you want it - Automatic speaker diarisation to seed the labels you then correct - A word-level search across a library of transcripts - Chapter markers derived from the edited transcript
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. The transcription model. Larger is more accurate and slower. WHISPER_MODEL_PATH=./models/ggml-base.en.bin # Required. Where renders are written. Never the source directory. OUTPUT_DIR=./exports # Required. How far a cut extends into surrounding silence, to avoid clicks at the edit. CUT_PAD_MS=40 # Required. Level below which audio counts as silence for detection. SILENCE_THRESHOLD_DB=-35 # Required. Minimum silence length before it is proposed as a cut. A pause is not a silence to remove. SILENCE_MIN_MS=600 # Required. The filler list, editable because it is language and speaker specific. FILLER_WORDS=um,uh,like,you know
PRODUCT.md
# Descript · product brief ## Problem You can build transcript-based cutting for simple clips, but Descript's editor, media pipeline, AI voice/video tooling, publishing, and collaboration are a serious product. ## Product outcome An editor where cutting text cuts media, previews are instant because nothing is rendered until export, cleanup is proposed rather than applied, and the source file is never touched. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - ffmpeg - speech-to-text/alignment - desktop or web editor - video rendering pipeline - optional hosted storage ## Explicit non-goals for v1 - A polished nonlinear editor with layers and transitions. - Voice cloning and generated corrections. - Real-time collaborative editing. - The workflow polish of the commercial product. - polished nonlinear editor - overdub/voice tools - filler-word workflows - templates - cloud collaboration - rendering reliability ## Success criteria - Single word playback confirms alignment on both fixtures - Audio and video sync is verified after twenty cuts - Caption timings are verified at the end of a heavily edited file - Source checksums are unchanged after export - Preview is confirmed to require no render step
BRIEF.md
# Build brief · Descript The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a transcript-driven editor where deleting a word deletes that piece of the audio and video to replace Descript. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Transcription: whisper.cpp with word-level timestamps · the timestamps are the entire mechanism; a transcript without them is just text - Runtime: TypeScript with a desktop shell · local media files, a real editor and no upload - Preview: Playback that skips cut ranges rather than rendering · rendering on every edit makes the tool unusable; skipping is instant - Render: ffmpeg with a filter graph built from the cut list · one render at export, from the untouched source - Storage: The source media untouched, plus a project file · every edit is a list of ranges, so nothing is destroyed and everything is reversible ### Data model (create this before Phase 1) - media(id, path, kind, duration_ms, sample_rate, has_video, content_hash) - words(id, media_id, ordinal, text, start_ms, end_ms, confidence, speaker) · the editing surface - cuts(id, media_id, start_ms, end_ms, reason) · reason is manual, filler or silence - speakers(media_id, label, display_name) - projects(id, name, media_ids_json, created_at) - A cut is a range, never a modification. The source file is read-only for the life of the project. ### Phase 1 · Word-level timestamps Build: Transcribe with word-level timestamps rather than segment-level. Convert audio to the format the transcriber needs. Verify alignment by playing a single word and hearing that word, not the one before or after it. Trap: Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. Done when: Playing the range of a single word from the middle of a sentence plays that word cleanly. Word timestamps are monotonic with no overlaps. A ten minute file transcribes in a time you can state. The source file is unmodified, verified by checksum. Do not build yet: No editing, no playback, no rendering. ### Phase 2 · Deleting text, deleting time Build: Render the transcript as the editing surface. Selecting a span of words and deleting creates a cut from the first word's start to the last word's end. Merge adjacent cuts. Undo restores both the text and the cut. Trap: Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. Done when: Deleting three words removes exactly that audio when previewed. Adjacent cuts merge into one range rather than accumulating. Undo restores the words and removes the cut. A cut at a word boundary with no surrounding silence is flagged as potentially audible. Do not build yet: No preview playback yet, so verification is by inspecting ranges. ### Phase 3 · Preview without rendering Build: Play the source media and skip cut ranges during playback by seeking past them. Keep the transcript scrolling in sync with playback. Handle a cut that lands during a seek without stuttering. Trap: Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. Done when: An edit is audible in preview immediately after it is made, with no render step. Playback skips cut ranges and the transcript scrolls in sync. Several consecutive short cuts play smoothly rather than stuttering through each seek. Scrubbing the transcript seeks the media to that word. Do not build yet: No filler removal or rendering yet. ### Phase 4 · Filler words and silences Build: Detect filler words from the transcript and long silences from the audio. Present both as proposed cuts you review and accept or reject, individually or in bulk. Never apply them automatically. Trap: Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. Done when: Filler detection proposes cuts and applies none until accepted. Accepting all fillers is one action and is undoable in one step. Silence detection uses a threshold and minimum duration you set, and a silence shorter than the minimum is not proposed. Rejecting a proposal keeps it rejected across sessions. Do not build yet: No rendering yet. ### Phase 5 · Rendering the export Build: Build an ffmpeg filter graph from the cut list, trimming and concatenating the kept ranges for both audio and video. Keep them in sync across every cut. Show real progress and never overwrite the source. Trap: Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. Done when: An exported file matches the preview, verified by comparing a passage after several cuts. Audio and video remain in sync at the end of a file with twenty cuts. Export progress reflects real work rather than jumping. The source file is unmodified after export, verified by checksum. Do not build yet: Multi-speaker and export formats come last. ### Phase 6 · Speakers, captions and formats Build: Label speakers and let a transcript be filtered by speaker. Export captions derived from the edited transcript, with timings recomputed after cuts. Add audio-only and social aspect exports. Trap: Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file. Done when: Speaker labels are assigned and the transcript can be filtered by speaker. Exported captions are correctly timed at the end of a file with many cuts, verified by playing the export with the captions loaded. An audio-only export contains the same edit. A vertical export reframes rather than letterboxing. Do not build yet: Nothing after this. ### Out of scope (and why) - The polished nonlinear editor. Once you need layers, transitions and effects, this is a text editor pretending, and a real editor is the right tool. - Voice cloning and generated speech corrections, which are a model and a set of consent questions rather than a feature. - Real-time collaboration on an edit. - The accumulated workflow polish of the commercial product, which is most of what makes it pleasant. ### README must contain - That transcription must produce word-level timestamps, and that segment level cannot support this - That cuts are ranges and the source media is never modified - That preview skips ranges rather than rendering, and export is the only render - That filler and silence removal propose rather than apply - That caption timings are recomputed against the edited timeline
ARCHITECTURE.md
# Architecture · Descript ## Stack | Part | Choice | Why | | --- | --- | --- | | Transcription | whisper.cpp with word-level timestamps | the timestamps are the entire mechanism; a transcript without them is just text | | Runtime | TypeScript with a desktop shell | local media files, a real editor and no upload | | Preview | Playback that skips cut ranges rather than rendering | rendering on every edit makes the tool unusable; skipping is instant | | Render | ffmpeg with a filter graph built from the cut list | one render at export, from the untouched source | | Storage | The source media untouched, plus a project file | every edit is a list of ranges, so nothing is destroyed and everything is reversible | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Transcribe | Words and their timestamps. | A hosted transcription service fits the same interface and gives up the offline property. Word-level output is non-negotiable either way. | | Cuts | Ranges, merging and padding. | Pure data over time ranges, testable with no media at all. | | Preview | Playback that skips ranges. | The no-render rule is what makes the tool usable and must survive any change. | | Render | Building the filter graph and exporting. | ffmpeg is the practical choice. Single-pass concatenation rather than per-cut files is the performance decision. | | Proposals | Filler and silence suggestions. | Detection can improve; proposing rather than applying is a product commitment. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `WHISPER_MODEL_PATH` · required · The transcription model. Larger is more accurate and slower. - `OUTPUT_DIR` · required · Where renders are written. Never the source directory. - `CUT_PAD_MS` · required · How far a cut extends into surrounding silence, to avoid clicks at the edit. - `SILENCE_THRESHOLD_DB` · required · Level below which audio counts as silence for detection. - `SILENCE_MIN_MS` · required · Minimum silence length before it is proposed as a cut. A pause is not a silence to remove. - `FILLER_WORDS` · required · The filler list, editable because it is language and speaker specific. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Descript product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: whisper.cpp with word-level timestamps, TypeScript with a desktop shell, Playback that skips cut ranges rather than rendering, ffmpeg with a filter graph built from the cut list, The source media untouched, plus a project file. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. - Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file.
MILESTONES.md
# Delivery milestones · Descript Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Word-level timestamps Every word in the transcript knows exactly when it was said. ### Steps 1. Convert audio for the transcriber Mono 16 kHz 16 bit is what whisper.cpp expects. Convert from the source rather than feeding it the original. ```sh ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` 2. Request word-level timestamps Explicitly. Segment level is the default and it cannot support this product. Files: `src/transcribe.ts` ```sh whisper-cli -m models/ggml-base.en.bin -f audio.wav --output-json-full ``` 3. Verify a single word plays correctly Pick a word from the middle of a sentence, play its range, and listen. This is the test that matters. ### Done when - [ ] A single word's range plays that word cleanly - [ ] Word timestamps are monotonic with no overlaps - [ ] A ten minute file transcribes in a stated time - [ ] The source file checksum is unchanged ### Watch out - Segment-level timestamps are the default in most transcription output and they are useless here, because a segment is a sentence and you need to cut a word. Enable word-level output explicitly and check the result, since a segment timestamp will look plausible in the data and fail the moment you cut. ## M2 · Deleting text, deleting time Selecting words and pressing delete produces a cut range, and the transcript reflows. ### Steps 1. Make the transcript the editor Words as spans with their timestamps attached. Selection maps to a time range. Files: `src/editor.ts` 2. Create and merge cuts A deletion becomes a range. Adjacent or overlapping ranges merge, or the cut list grows without bound. Files: `src/cuts.ts` 3. Pad into silence where possible Extend slightly into the gap between words. Flag cuts with no gap available. Files: `src/cuts.ts` ### Done when - [ ] Deleting three words produces the corresponding cut range - [ ] Adjacent cuts merge into one - [ ] Undo restores words and removes the cut - [ ] A cut with no surrounding silence is flagged ### Watch out - Cutting exactly at word boundaries clips the natural gap between words and produces an audible click at every edit. Extend each cut slightly into the surrounding silence, and where two words are contiguous with no gap, accept the tighter cut and flag it as likely audible. ## M3 · Preview without rendering You hear the edit immediately, without waiting for anything to encode. ### Steps 1. Skip ranges during playback On approaching a cut, seek to its end. Keep a small lookahead so the seek is issued before the boundary. Files: `src/preview.ts` 2. Sync the transcript to playback Highlight the current word. This is also how you notice alignment problems. Files: `src/preview.ts` 3. Handle consecutive cuts Several short cuts in a row must not produce several audible seeks. Merge the playback plan rather than seeking repeatedly. Files: `src/preview.ts` ### Done when - [ ] An edit is audible immediately with no render - [ ] The transcript scrolls in sync with playback - [ ] Consecutive short cuts play smoothly - [ ] Clicking a word seeks the media there ### Watch out - Rendering a preview after every edit makes the tool useless, because editing is a hundred small changes and each render is seconds. Skip ranges during playback instead: seeking is instant and the result is close enough to judge the edit. ## M4 · Filler words and silences The obvious cleanup is one action, and you can see exactly what it will remove. ### Steps 1. Detect fillers from the transcript A configurable list. Language and speaker specific, so make the list editable rather than fixed. Files: `src/fillers.ts` 2. Detect silences from the audio Threshold and minimum duration, both settings. A pause is not a silence to remove. Files: `src/silence.ts` ```sh ffmpeg -i audio.wav -af silencedetect=n=-35dB:d=0.6 -f null - ``` 3. Propose, never apply Review with counts, bulk accept as one undoable action, and remembered rejections. Files: `src/proposals.ts` ### Done when - [ ] Detection proposes and applies nothing automatically - [ ] Bulk accept is one undoable action - [ ] A silence shorter than the minimum is not proposed - [ ] Rejected proposals stay rejected across sessions ### Watch out - Removing every detected filler makes speech sound unnaturally clipped, because some hesitations carry meaning and rhythm. Propose rather than apply, show the count, and let a bulk accept be a deliberate action rather than a default. ## M5 · Rendering the export The exported file matches the preview, including the video. ### Steps 1. Build the filter graph from cuts Trim and concatenate kept ranges in one ffmpeg invocation for both streams. Files: `src/render.ts` 2. Verify sync after many cuts Twenty cuts in a ten minute file, checked at the end. Drift accumulates and only shows at length. 3. Never write to the source Output alongside, checksum the source before and after as a test. Files: `src/render.test.ts` ### Done when - [ ] The export matches the preview after several cuts - [ ] Audio and video stay in sync after twenty cuts - [ ] Progress reflects real work - [ ] The source is unmodified by checksum ### Watch out - Trimming video at a non-keyframe produces either a frozen frame or a re-encode of the whole file, and doing this per cut with many cuts is very slow. Concatenate trimmed segments through a filter graph in one pass rather than cutting and joining files, and accept that a full re-encode is the honest cost of frame-accurate cuts. ## M6 · Speakers, captions and formats (production only) Multi-speaker recordings are usable and the output carries its transcript. ### Steps 1. Assign and filter by speaker Labels applied to word ranges, correctable by hand since automatic assignment will be imperfect. Files: `src/speakers.ts` 2. Recompute caption timings Against the edited timeline, not the original. Check a caption near the end of a heavily cut file. Files: `src/captions.ts` 3. Add export variants Audio only and a vertical crop. Both reuse the same cut list. Files: `src/exports.ts` ### Done when - [ ] Speaker labels are assignable and filterable - [ ] Captions are correctly timed at the end of a heavily edited file - [ ] An audio-only export carries the same edit - [ ] A vertical export reframes rather than letterboxes ### Watch out - Caption timings computed from the original transcript will be wrong after any cut, because every subsequent word has moved earlier. Recompute timings against the edited timeline, and verify by checking a caption near the end of a heavily edited file.
OPERATIONS.md
# Operations · Descript ## Backup Source media is irreplaceable and belongs in a backup. Project files are small and hold the cut list, which is the actual work. Renders are regenerable from both. ## Restore Restore sources and project files, then re-export. Because cuts are ranges over unmodified sources, a restore reproduces the edit exactly rather than approximately. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring For a personal tool, watch transcription and render times against media length. A render slower than real time by a large factor usually means per-cut file operations rather than a single filter graph. ## Incident checklist The realistic failure is a transcript misaligned with the media, which makes every cut wrong. Verify alignment on import rather than discovering it mid-edit, which is what the single word playback check in Phase 1 is for. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Single word playback confirms alignment on both fixtures - [ ] Audio and video sync is verified after twenty cuts - [ ] Caption timings are verified at the end of a heavily edited file - [ ] Source checksums are unchanged after export - [ ] Preview is confirmed to require no render step ## Launch constraint Do not market omitted Descript capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. The transcription model. Larger is more accurate and slower. WHISPER_MODEL_PATH=./models/ggml-base.en.bin # Required. Where renders are written. Never the source directory. OUTPUT_DIR=./exports # Required. How far a cut extends into surrounding silence, to avoid clicks at the edit. CUT_PAD_MS=40 # Required. Level below which audio counts as silence for detection. SILENCE_THRESHOLD_DB=-35 # Required. Minimum silence length before it is proposed as a cut. A pause is not a silence to remove. SILENCE_MIN_MS=600 # Required. The filler list, editable because it is language and speaker specific. FILLER_WORDS=um,uh,like,you know
$ choose a build depth, inspect the files, then open the complete pack in your agent
They pay to avoid building and babysitting a media editor, not just for transcription.
xpolished nonlinear editor
xoverdub/voice tools
xfiller-word workflows
xtemplates
xcloud collaboration
xrendering reliability
share on X ↗"I just replaced Descript ($24/mo) with one prompt"
Don't feel like building it? These folks already made it free.
KdenliveA real editor with local transcription and transcript-based cutting; the AI copywriter did not come along.open source↗no votes, no pay-to-list · just what's real
Descript pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| free | $0/user | $0/user | 1 media hour/editor/month; 100 AI credits one-time; 720p watermarked exports; unlimited projects |
| hobbyist | $24/user | $16/user | 10 media hours/editor/month; 400 AI credits/month; 1080p; 30 AI Speech minutes/month |
| creator | $35/user | $24/user | 30 media hours/editor/month; 800 AI credits/month; 4K; 120 AI Speech minutes/month |
| business | $65/user | $50/user | 40 media hours/editor/month; 1,500 AI credits/month; 300 AI Speech minutes/month |
| enterprise | custom | — | Custom media hours, AI credits, seats, security, support and contract terms |
free tier1 media hour/editor/month; 100 AI credits once; 720p watermarked exports; SquadCast 1 recording hour/editor/month, 1 show, 10 participants and 1 integration
billingmonthly + annual (up to 35% lower); annual billed upfront; paid editors billed separately
hidden costsExtra media hours and AI-credit top-ups are sold only on Creator and Business; adding editors multiplies the subscription.
verified 2026-08-14 · source ↗
Vibecode Descript
Kinda. The core of Descript is buildable in a weekend with the prompt on this page, but there are real gaps: polished nonlinear editor, overdub/voice tools. Read the honest list above before committing.
How much does Descript cost?
Descript costs about $24/month (Hobbyist, checked 2026-07-30), which is $288 per year.
What do I lose by replacing Descript?
Honestly: polished nonlinear editor; overdub/voice tools; filler-word workflows; templates; cloud collaboration; rendering reliability. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Descript?
Yes: Kdenlive (A real editor with local transcription and transcript-based cutting; the AI copywriter did not come along.) The prompt is for when you want it exactly your way.