Build Shade

KINDAreplaces $35/mosaves $420/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A local footage search index whose hard problems are identifying files that move between drives, keeping the frame index from drowning in near-identical stills, and combining picture search with speech search when the two live in different vector spaces.

estimated effort multi-daythe files for this build are in the project pack

RuntimeA Python CLI for indexing plus a small local web page for searchingIndexSQLite with a vector extension, one file per libraryFramesffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embeddingSpeechA local transcriber producing word timestamps, chunked on sentence boundariesRetrievalTwo separate indexes combined by rank position, never by comparing scores across them

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Every frame in this index comes out of it, and the scene filter is what makes the sampling worth anything.
Get it Your package manager, or a static build. open ↗
Verify ffmpeg -version prints and a scene filter run produces frames
installfree
Why Runs the indexer and holds the vectors in the same file as the metadata.
Get it python.org or your package manager, plus a SQLite vector extension.
Verify A vector query returns results against a test table
have readyfree; disk space is the cost
Why First-run downloads are large and slow, and discovering that at the start of an overnight index wastes the night.
Get it Download the weights before your first real run and note their sizes.
Verify Both models load and embed a test input offline with the network disabled
have readyfree if you own one
Why The difference between hardware here is the difference between a weekend and a fortnight, and it is better known before you start than after.
Get it Whatever you have. Run the rate measurement on one hour of footage first.
Verify A measured rate exists for one hour of your own footage
have readyfree
Why Those two clips expose the frame explosion and the chunking problems respectively, and a synthetic test will not.
Get it Pick them from the library you intend to index.
Verify Both index without the frame cap or the segment count looking absurd

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- volumes(id, label, filesystem_uuid, last_seen_at, last_mount_hint) · so an offline drive is reported as offline rather than as deleted footage
- files(id, volume_id, rel_path, bytes, mtime, quick_hash, duration_ms, first_seen_at, last_seen_at, state)
- frames(id, file_id, t_ms, thumb_path, phash, embedding) · phash is what stops a static shot from producing hundreds of identical vectors
- segments(id, file_id, start_ms, end_ms, text, embedding, mean_logprob) · one per sentence-aligned chunk of speech
- queries(id, text, token_count, was_truncated, at) · kept because a truncated query is the most common reason a search feels broken
- runs(id, root, started_at, finished_at, files_seen, files_indexed, video_seconds, wall_seconds) · the numbers that let you predict the next overnight run
- Paths are stored relative to a volume, never absolute, so remounting a drive somewhere else does not invalidate the entire index.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`INDEX_PATH`	required	`~/FootageIndex/index.db`	Vectors, metadata and run history in one file. Disposable but expensive to rebuild.
`THUMB_DIR`	required	`~/FootageIndex/thumbs`	One small image per surviving frame. Keep these; they make re-embedding cheap.
`FRAME_INTERVAL_SECONDS`	required	`5`	Baseline sampling. Scene cuts are added on top of this.
`MAX_FRAMES_PER_MINUTE`	required	`12`	The cap that stops a static shot from owning the index.
`PHASH_DISTANCE`	required	`6`	Frames closer than this to the previous one are dropped before embedding.
`SEGMENT_TARGET_SECONDS`	required	`25`	Chunking aims here but always finishes the sentence.
`MIN_SEGMENT_CONFIDENCE`	required	`-1.0`	Segments below this are excluded from search by default and remain in the index.

The build, in order

1
Knowing which file is which
The walker identifies files by content rather than by path, and an interrupted index resumes where it stopped.
1. Reorganising a library should not cost a night of GPU time.
  Files src/index/identity.pytests/test_identity.py
2. Drives get remounted and letters change.
  Files src/index/volumes.py
3. This tool reads somebody's only copy of a wedding.
  Files src/index/walk.pytests/test_readonly.py
done when · tick each as it passes
watch out
- Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column.
next phase
2
Frames worth embedding
Each video yields a manageable set of visually distinct frames rather than one every few seconds regardless.
1. Cuts are where the content actually changes.
  Files src/frames/extract.py
2. Embedding is the expensive step, so filter before it.
  Files src/frames/dedupe.pytests/test_dedupe.py
3. One boring shot should not own the index.
  Files src/frames/limits.py
done when · tick each as it passes
watch out
- Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index.
next phase
3
The words that were spoken
Speech becomes timestamped segments chunked where sentences end, with a confidence you can filter on.
1. Half a sentence is a vector for a thing nobody said.
  Files src/speech/chunk.pytests/test_chunk.py
2. The timecode is the whole point of the result.
  Files src/speech/transcribe.py
3. Low-confidence speech is where the strange results come from.
  Files src/speech/confidence.py
done when · tick each as it passes
watch out
- Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame.
next phase
4
One sentence, two indexes
A plain-English query searches pictures and speech together and returns ranked results with timecodes.
1. The most common broken search is a query nobody told you was too long.
  Files src/search/tokens.pytests/test_tokens.py
2. Two models, two spaces, two incomparable numbers.
  Files src/search/fuse.pytests/test_fuse.py
3. Knowing whether it was seen or said changes what you search next.
  Files src/web/results.html
done when · tick each as it passes
watch out
- The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results.
5
An index you can rebuild and a library you can moveproduct builder
The index is disposable, the numbers to plan a re-index are measured, and the library never depends on this tool.
1. This is the number that tells you whether a terabyte is a night or a week.
  Files src/index/rate.py
2. Extraction is the expensive half and it is model-independent.
  Files src/index/reembed.py
3. Footage on a drive in a drawer is still footage.
  Files src/web/offline.py
done when · tick each as it passes
watch out
- Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts.

what this build does not replace

Face recognition and person tagging.
Sharing links and guest access.
Review, comments and approval.
Cloud proxies and streaming.
cloud streaming of full-res files without downloading them first
guest links with per-link permissions and roles
built-in review, approval, and commenting
face recognition and shot-type tagging that improves without your involvement
team sync, so everyone searches the same index
the NLE plugins and Slack integration

after v1, if you want it

Shot type and camera movement tags, which are a second small classifier over frames you already extracted
A saved search that re-runs as the library grows, which is the closest thing to a smart folder this design allows
Colour and exposure summaries per frame, cheap to compute and genuinely useful for finding a look rather than a subject

Need the files? The project pack on the verdict page hands your agent the whole brief · more audio & video.