Build OpusClip

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A clipping pipeline where the selection problem is honest and the rendering problem is exact: finding passages that stand alone is a judgement the tool should assist rather than claim, and reframing to vertical without decapitating the speaker is a tracking problem with a measurable answer.

estimated effort multi-daythe files for this build are in the project pack

TranscriptionLocal recognition with word timingsSelectionCandidate passages scored on stated signals, proposed not chosenFramingSubject tracking with a smoothed crop pathCaptionsTimings from the transcript, never estimatedRuntimePython 3.12 with ffmpeg

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Orchestrates transcription, selection and rendering.
Get it python.org or your package manager, then a virtual environment.
Verify python3 --version prints 3.12 or higher
installfree
Why Cutting, cropping and rendering all run through it, and reimplementing any of it is not the project.
Get it Your package manager, confirming it handles your source format and the vertical output your targets require. open ↗
Verify A clip cuts and re-encodes at the target aspect without quality loss beyond expectation
installfree, needs machine time
Why Every cut point and every caption derives from word timings, and segment-level timings are not precise enough to cut on.
Get it whisper.cpp or an equivalent producing word-level timings, and check it does before building on it. open ↗
Verify A five minute clip produces per-word timings
installfree
Why Reframing to vertical requires knowing where the speaker is in every frame, and a static crop is usually wrong.
Get it Any detector with tracking across frames, evaluated on your own footage rather than on a benchmark.
Verify A speaker moving across frame is tracked without losing them
have readyfree if you have one
Why Seam losses, speaker changes and framing failures only appear in real material of realistic length.
Get it A two hour recording with two people, including a stretch where they talk over each other and one where the speaker leaves frame.
Verify A two hour two-speaker source is available with both cases present

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- sources(id, path, duration_ms, width, height, fps, imported_at)
- words(source_id, ordinal, text, start_ms, end_ms, speaker, confidence)
- candidates(id, source_id, start_ms, end_ms, signals_json, score, rationale, accepted_by)
- clips(id, candidate_id, start_ms, end_ms, aspect, crop_path_json, rendered_path, rendered_at)
- captions(clip_id, ordinal, start_ms, end_ms, text) · derived from words, never estimated
- speakers(source_id, id, label, face_track_id) · because reframing needs to know who is talking
- A candidate stores the signals behind its score rather than only the number, so a user can see why a passage was suggested and disagree with it.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`SOURCE_DIR`	required	`./sources`	Original recordings, opened read only.
`OUTPUT_DIR`	required	`./clips`	Rendered clips and manifests. Never the source directory.
`ASR_MODEL`	required	`./models/ggml-medium.en.bin`	Timing quality matters more here than word accuracy.
`TARGET_ASPECT`	required	`9:16`	The output aspect, which drives the whole framing problem.
`CROP_SMOOTHING_MS`	required	`400`	Too little jitters, too much lags behind the speaker.
`AUTO_ACCEPT_CANDIDATES`	required	`never`	Present as a constant. Automatic clipping produces volume rather than clips.

The build, in order

1
Words with times you can cut on
A long recording transcribes with timings accurate enough to cut by.
1. The seams are where the words go missing and everything downstream drifts.
  Files src/transcribe.pytests/test_seams.py
2. A transcript ending early means every later cut point is wrong.
  Files src/verify.py
3. Guessing who spoke is worse than saying you do not know.
  Files src/speakers.py
done when · tick each as it passes
watch out
- Chunking a two hour recording without overlap loses words at every boundary, and every cut point after that is derived from a transcript that has drifted from the audio. Overlap, merge on matching text, and check the word count and final timestamp against a single-pass transcription of a shorter excerpt.
next phase
2
Candidates proposed, with reasons
Suggested passages carry the signals behind them.
1. A number with no reasons cannot be argued with.
  Files src/candidates.py
2. Automatic clipping produces volume rather than clips.
  Files src/review.py
3. A clip starting mid-sentence is discarded by whoever watches it.
  Files src/boundaries.py
done when · tick each as it passes
watch out
- Producing a score with no explanation makes disagreeing with the tool impossible, and the user either accepts everything or ignores the feature. Store which signals fired for each candidate and show them, so a suggestion is an argument the person can reject.
next phase
3
Reframing without decapitating anyone
A vertical crop follows the speaker smoothly and keeps their head in frame.
1. Frame-by-frame following is jitter, which is worse than a static crop.
  Files src/framing.py
2. A fast pan between two people is nauseating.
  Files src/framing.py
3. Checked against the output, not against the tracking confidence.
  Files tests/test_framing.py
done when · tick each as it passes
watch out
- A crop that follows the detected face position frame by frame jitters constantly and is more distracting than a static crop, while one that snaps between two speakers is nauseating. Smooth the path, move only when the subject has genuinely moved, and cut rather than pan when the speaker changes.
next phase
4
Captions from the transcript
Captions match the speech exactly at the end of a clip.
1. You already have them. Estimation drifts within a minute.
  Files src/captions.py
2. A caption breaking mid-phrase is harder to read than a longer line.
  Files src/captions.py
3. The opening seconds always look correct.
  Files tests/test_sync.py
done when · tick each as it passes
watch out
- Estimating caption timing from word count and reading speed drifts audibly within the first minute, and checking only the opening seconds always looks fine. Use the word timings you already have, and verify at the end where drift accumulates.
5
Rendering and keeping the sourceproduct builder
Clips render reproducibly and the original is untouched.
1. Reproducible, and reviewable as text before it runs.
  Files src/render.py
2. A write to an original is an irreversible loss.
  Files tests/test_sources_untouched.py
3. One bad clip must not end a thirty clip batch.
  Files src/batch.py
done when · tick each as it passes
watch out
- Any path that writes to the source file, including a helpfully added normalisation pass, damages an original recording that may be irreplaceable, and the loss is discovered when someone wants to re-clip it. Open read only, checksum before and after, and fail the build if they differ.

what this build does not replace

Predicting which clips will perform.
Publishing to platforms.
Music, effects and heavy editing.
Multi-camera switching.
professional codecs, effects, and collaboration
production codecs, rendering speed, and media templates
hosted recording reliability
large-scale rendering
distribution and podcast hosting

after v1, if you want it

Speaker-aware candidate selection, so a clip does not start mid-answer
A review interface that plays candidates back to back, which is how the accepting actually happens
Alternative aspect ratios from one render description, since targets differ

Need the files? The project pack on the verdict page hands your agent the whole brief · more audio & video.