Vibecode Shade
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The search half is real and rebuildable. Extract keyframes with ffmpeg, embed them with CLIP, transcribe the audio with whisper, put the vectors in SQLite, and 'the drone shot over the bridge at golden hour' finds the clip on your own drives. The open-source stack for that is mature and the result is genuinely good. What does not survive the port is what Shade has grown into: cloud streaming so an editor opens full-res without waiting on a download, per-link permissions and guest access, review and approval, and a model pipeline that keeps improving without you retraining anything. Solo, on local storage, the DIY version wins outright. On a team, you are rebuilding a platform and calling it a script.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Shade. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Shade · indie build A local footage search index whose hard problems are identifying files that move between drives, keeping the frame index from drowning in near-identical stills, and combining picture search with speech search when the two live in different vector spaces. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python CLI for indexing plus a small local web page for searching | indexing runs overnight and searching happens at a keyboard, and those want different interfaces | | Index | SQLite with a vector extension, one file per library | one file you can copy, back up and delete, rather than a service to operate | | Frames | ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding | an interview locked off for ten minutes should not become a hundred vectors | | Speech | A local transcriber producing word timestamps, chunked on sentence boundaries | a chunk that starts mid-sentence embeds as nonsense | | Retrieval | Two separate indexes combined by rank position, never by comparing scores across them | picture vectors and sentence vectors live in different spaces and their numbers are not comparable | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **ffmpeg with scene detection available** · free - Why: Every frame in this index comes out of it, and the scene filter is what makes the sampling worth anything. - Get it: Your package manager, or a static build. - Verify: ffmpeg -version prints and a scene filter run produces frames - [ ] **Python 3.11 or newer with a vector-capable SQLite build** · free - Why: Runs the indexer and holds the vectors in the same file as the metadata. - Get it: python.org or your package manager, plus a SQLite vector extension. - Verify: A vector query returns results against a test table - [ ] **A picture and text embedding model and a local speech model, downloaded** · free; disk space is the cost - Why: First-run downloads are large and slow, and discovering that at the start of an overnight index wastes the night. - Get it: Download the weights before your first real run and note their sizes. - Verify: Both models load and embed a test input offline with the network disabled - [ ] **A GPU, or an honest estimate of how many nights you are prepared to spend** (optional) · free if you own one - Why: The difference between hardware here is the difference between a weekend and a fortnight, and it is better known before you start than after. - Get it: Whatever you have. Run the rate measurement on one hour of footage first. - Verify: A measured rate exists for one hour of your own footage - [ ] **One hour of your real footage including a static interview and a clip with speech** · free - Why: Those two clips expose the frame explosion and the chunking problems respectively, and a synthetic test will not. - Get it: Pick them from the library you intend to index. - Verify: Both index without the frame cap or the segment count looking absurd ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Face recognition and person tagging. - Sharing links and guest access. - Review, comments and approval. - Cloud proxies and streaming. - cloud streaming of full-res files without downloading them first - guest links with per-link permissions and roles - built-in review, approval, and commenting - face recognition and shot-type tagging that improves without your involvement - team sync, so everyone searches the same index - the NLE plugins and Slack integration If one of those is essential to you, that is the reason to keep paying for Shade, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Shade The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a search index over my own footage that finds a shot from a plain sentence and runs entirely on my machine to replace Shade. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: A Python CLI for indexing plus a small local web page for searching · indexing runs overnight and searching happens at a keyboard, and those want different interfaces - Index: SQLite with a vector extension, one file per library · one file you can copy, back up and delete, rather than a service to operate - Frames: ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding · an interview locked off for ten minutes should not become a hundred vectors - Speech: A local transcriber producing word timestamps, chunked on sentence boundaries · a chunk that starts mid-sentence embeds as nonsense - Retrieval: Two separate indexes combined by rank position, never by comparing scores across them · picture vectors and sentence vectors live in different spaces and their numbers are not comparable ### Data model (create this before Phase 1) - volumes(id, label, filesystem_uuid, last_seen_at, last_mount_hint) · so an offline drive is reported as offline rather than as deleted footage - files(id, volume_id, rel_path, bytes, mtime, quick_hash, duration_ms, first_seen_at, last_seen_at, state) - frames(id, file_id, t_ms, thumb_path, phash, embedding) · phash is what stops a static shot from producing hundreds of identical vectors - segments(id, file_id, start_ms, end_ms, text, embedding, mean_logprob) · one per sentence-aligned chunk of speech - queries(id, text, token_count, was_truncated, at) · kept because a truncated query is the most common reason a search feels broken - runs(id, root, started_at, finished_at, files_seen, files_indexed, video_seconds, wall_seconds) · the numbers that let you predict the next overnight run - Paths are stored relative to a volume, never absolute, so remounting a drive somewhere else does not invalidate the entire index. ### Phase 1 · Knowing which file is which Build: Walk a root, recording each file against its volume with a relative path, size, modification time and a quick hash of the first and last blocks plus the size. Mark work done per file so a restart skips finished ones. Open every file read-only and never move, rename or write to anything. Trap: Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. Done when: Renaming a folder and re-running the index updates paths without re-embedding anything, tested on a sample tree. Killing the indexer and restarting resumes and does not redo completed files, tested. A volume that is not mounted is reported as offline with its label and its files remain searchable as results you cannot open, tested by unmounting. No test run modifies, moves or renames any source file, asserted by comparing modification times across the tree. Do not build yet: No frames, no transcripts, no search and no web page yet. ### Phase 2 · Frames worth embedding Build: Pull keyframes at a configured interval and at every scene cut ffmpeg reports. Compute a perceptual hash per frame and drop any frame too close to the one before it. Write a small thumbnail per surviving frame and record its timestamp in milliseconds. Cap frames per minute. Trap: Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. Done when: A static clip produces a small bounded number of frames, tested with a generated video of one unchanging image. A clip with distinct scenes produces at least one frame per scene, tested with a generated cut. Thumbnails exist for every stored frame and the index refuses a frame whose thumbnail failed to write. The frames-per-minute cap is configurable and enforced, tested at the boundary. Do not build yet: No transcripts, no embeddings, no search yet. ### Phase 3 · The words that were spoken Build: Transcribe with word timestamps. Build segments by accumulating words until a sentence boundary near a target length, rather than cutting at a fixed number of seconds. Store the mean token confidence per segment and let the search filter low-confidence ones out. Trap: Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. Done when: Segments begin and end at sentence boundaries wherever the transcript provides punctuation, checked against a sample with known sentences. Every segment carries start and end timestamps in milliseconds that point at the right moment in the file, verified by opening three at random. A confidence value is stored per segment and the search can exclude below a threshold. A file with no speech produces zero segments and no errors, tested with a silent clip. Do not build yet: No search, no fusion and no web page yet. ### Phase 4 · One sentence, two indexes Build: Embed the query once for the picture index and once for the text index. Retrieve from each separately and combine by rank position rather than by score. Count the query's tokens against the picture model's context limit before embedding and say plainly when it will not fit. Results show a thumbnail, a filename, a timecode and which index matched. Trap: The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. Done when: A query over the picture model's context limit produces a visible warning rather than a crash or a silent truncation, tested with a long sentence. Picture and speech results are combined by rank position and no code path compares a picture similarity to a text similarity, verified by review. Every result carries the file, the timecode and which index produced it. Clicking a result opens the file at that frame in a player, tested on three results by hand. Do not build yet: No rebuild tooling, no export and no rate reporting yet. ### Phase 5 · An index you can rebuild and a library you can move Build: Report indexing rate per run: video seconds processed per wall second, separately for frames and speech, so the next overnight run is predictable. Export the index as a manifest of files, frames and segments with text. Support re-embedding with a new model without re-extracting frames. Handle an offline volume by showing results greyed with the drive label rather than hiding them. Trap: Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts. Done when: Each run prints video seconds per wall second for frames and speech separately, and the figures are stored per run. Re-embedding with a different model completes without re-extracting a single frame or re-transcribing a single file, tested by timing both. Export produces a manifest that reimports into an empty index with the same file, frame and segment counts. Results from an unmounted volume appear with their drive label and cannot be opened, rather than disappearing. Do not build yet: Nothing after this. ### Out of scope (and why) - Face recognition and person tagging, which is a biometric system with consent and retention questions attached and deserves a deliberate decision rather than a feature flag. - Sharing links and guest access, because the moment the index is reachable by someone else you are operating a service with permissions, and that is the half the paid product actually sells. - Review, comments and approval, which is a collaboration product sitting on top of search rather than part of it. - Cloud proxies so an editor can open full resolution without downloading, which is bandwidth and storage rather than code. ### README must contain - That everything runs locally, which models are downloaded on first run and how large they are - A measured indexing rate in video hours per wall hour on your own hardware, taken from a real run rather than estimated - That the picture model has a short query context limit, what happens when a query exceeds it, and that the interface will say so - That picture and speech results are combined by rank because their scores are not comparable - That source files are opened read-only and never moved, and how to delete the index without touching the footage ===== AGENTS.md ===== # Agent instructions · Shade indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python CLI for indexing plus a small local web page for searching, SQLite with a vector extension, one file per library, ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding, A local transcriber producing word timestamps, chunked on sentence boundaries, Two separate indexes combined by rank position, never by comparing scores across them. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. ===== BUILD_PLAN.md ===== # Build plan · Shade A local footage search index whose hard problems are identifying files that move between drives, keeping the frame index from drowning in near-identical stills, and combining picture search with speech search when the two live in different vector spaces. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Knowing which file is which The walker identifies files by content rather than by path, and an interrupted index resumes where it stopped. ### Steps 1. Identify by a content quick hash, not by path Reorganising a library should not cost a night of GPU time. Files: `src/index/identity.py`, `tests/test_identity.py` 2. Store paths relative to a volume Drives get remounted and letters change. Files: `src/index/volumes.py` 3. Open read-only and prove it This tool reads somebody's only copy of a wedding. Files: `src/index/walk.py`, `tests/test_readonly.py` ### Done when - [ ] A renamed folder does not trigger re-embedding - [ ] An interrupted index resumes cleanly - [ ] An offline volume is reported rather than treated as deleted - [ ] No source file is ever modified ### Watch out - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. ## Phase 2 · Frames worth embedding Each video yields a manageable set of visually distinct frames rather than one every few seconds regardless. ### Steps 1. Sample on interval and on scene cuts Cuts are where the content actually changes. Files: `src/frames/extract.py` 2. Deduplicate by perceptual hash before embedding Embedding is the expensive step, so filter before it. Files: `src/frames/dedupe.py`, `tests/test_dedupe.py` 3. Cap frames per minute One boring shot should not own the index. Files: `src/frames/limits.py` ### Done when - [ ] A static clip yields few frames - [ ] Every scene cut yields at least one frame - [ ] Frames without thumbnails are rejected - [ ] The frames-per-minute cap is enforced ### Watch out - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. ## Phase 3 · The words that were spoken Speech becomes timestamped segments chunked where sentences end, with a confidence you can filter on. ### Steps 1. Chunk on sentence boundaries near a target length Half a sentence is a vector for a thing nobody said. Files: `src/speech/chunk.py`, `tests/test_chunk.py` 2. Keep millisecond timestamps on every segment The timecode is the whole point of the result. Files: `src/speech/transcribe.py` 3. Store and expose a confidence per segment Low-confidence speech is where the strange results come from. Files: `src/speech/confidence.py` ### Done when - [ ] Segments respect sentence boundaries - [ ] Timestamps open at the right moment - [ ] Confidence is stored and filterable - [ ] A silent clip produces no segments ### Watch out - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. ## Phase 4 · One sentence, two indexes A plain-English query searches pictures and speech together and returns ranked results with timecodes. ### Steps 1. Count query tokens before embedding and warn The most common broken search is a query nobody told you was too long. Files: `src/search/tokens.py`, `tests/test_tokens.py` 2. Fuse by rank, never by comparing scores Two models, two spaces, two incomparable numbers. Files: `src/search/fuse.py`, `tests/test_fuse.py` 3. Show which index matched, on every result Knowing whether it was seen or said changes what you search next. Files: `src/web/results.html` ### Done when - [ ] An overlong query warns rather than truncating silently - [ ] No code path compares picture and text scores directly - [ ] Every result shows file, timecode and source index - [ ] Clicking a result opens the file at that frame ### Watch out - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. ## Not in this build - Face recognition and person tagging. - Sharing links and guest access. - Review, comments and approval. - Cloud proxies and streaming. ## After v1, if you want it - Shot type and camera movement tags, which are a second small classifier over frames you already extracted - A saved search that re-runs as the library grows, which is the closest thing to a smart folder this design allows - Colour and exposure summaries per frame, cheap to compute and genuinely useful for finding a look rather than a subject ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Vectors, metadata and run history in one file. Disposable but expensive to rebuild. INDEX_PATH=~/FootageIndex/index.db # Required. One small image per surviving frame. Keep these; they make re-embedding cheap. THUMB_DIR=~/FootageIndex/thumbs # Required. Baseline sampling. Scene cuts are added on top of this. FRAME_INTERVAL_SECONDS=5 # Required. The cap that stops a static shot from owning the index. MAX_FRAMES_PER_MINUTE=12 # Required. Frames closer than this to the previous one are dropped before embedding. PHASH_DISTANCE=6 # Required. Chunking aims here but always finishes the sentence. SEGMENT_TARGET_SECONDS=25 # Required. Segments below this are excluded from search by default and remain in the index. MIN_SEGMENT_CONFIDENCE=-1.0
You are building a lean indie version of Shade. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Shade · indie build A local footage search index whose hard problems are identifying files that move between drives, keeping the frame index from drowning in near-identical stills, and combining picture search with speech search when the two live in different vector spaces. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python CLI for indexing plus a small local web page for searching | indexing runs overnight and searching happens at a keyboard, and those want different interfaces | | Index | SQLite with a vector extension, one file per library | one file you can copy, back up and delete, rather than a service to operate | | Frames | ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding | an interview locked off for ten minutes should not become a hundred vectors | | Speech | A local transcriber producing word timestamps, chunked on sentence boundaries | a chunk that starts mid-sentence embeds as nonsense | | Retrieval | Two separate indexes combined by rank position, never by comparing scores across them | picture vectors and sentence vectors live in different spaces and their numbers are not comparable | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **ffmpeg with scene detection available** · free - Why: Every frame in this index comes out of it, and the scene filter is what makes the sampling worth anything. - Get it: Your package manager, or a static build. - Verify: ffmpeg -version prints and a scene filter run produces frames - [ ] **Python 3.11 or newer with a vector-capable SQLite build** · free - Why: Runs the indexer and holds the vectors in the same file as the metadata. - Get it: python.org or your package manager, plus a SQLite vector extension. - Verify: A vector query returns results against a test table - [ ] **A picture and text embedding model and a local speech model, downloaded** · free; disk space is the cost - Why: First-run downloads are large and slow, and discovering that at the start of an overnight index wastes the night. - Get it: Download the weights before your first real run and note their sizes. - Verify: Both models load and embed a test input offline with the network disabled - [ ] **A GPU, or an honest estimate of how many nights you are prepared to spend** (optional) · free if you own one - Why: The difference between hardware here is the difference between a weekend and a fortnight, and it is better known before you start than after. - Get it: Whatever you have. Run the rate measurement on one hour of footage first. - Verify: A measured rate exists for one hour of your own footage - [ ] **One hour of your real footage including a static interview and a clip with speech** · free - Why: Those two clips expose the frame explosion and the chunking problems respectively, and a synthetic test will not. - Get it: Pick them from the library you intend to index. - Verify: Both index without the frame cap or the segment count looking absurd ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Face recognition and person tagging. - Sharing links and guest access. - Review, comments and approval. - Cloud proxies and streaming. - cloud streaming of full-res files without downloading them first - guest links with per-link permissions and roles - built-in review, approval, and commenting - face recognition and shot-type tagging that improves without your involvement - team sync, so everyone searches the same index - the NLE plugins and Slack integration If one of those is essential to you, that is the reason to keep paying for Shade, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Shade The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a search index over my own footage that finds a shot from a plain sentence and runs entirely on my machine to replace Shade. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: A Python CLI for indexing plus a small local web page for searching · indexing runs overnight and searching happens at a keyboard, and those want different interfaces - Index: SQLite with a vector extension, one file per library · one file you can copy, back up and delete, rather than a service to operate - Frames: ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding · an interview locked off for ten minutes should not become a hundred vectors - Speech: A local transcriber producing word timestamps, chunked on sentence boundaries · a chunk that starts mid-sentence embeds as nonsense - Retrieval: Two separate indexes combined by rank position, never by comparing scores across them · picture vectors and sentence vectors live in different spaces and their numbers are not comparable ### Data model (create this before Phase 1) - volumes(id, label, filesystem_uuid, last_seen_at, last_mount_hint) · so an offline drive is reported as offline rather than as deleted footage - files(id, volume_id, rel_path, bytes, mtime, quick_hash, duration_ms, first_seen_at, last_seen_at, state) - frames(id, file_id, t_ms, thumb_path, phash, embedding) · phash is what stops a static shot from producing hundreds of identical vectors - segments(id, file_id, start_ms, end_ms, text, embedding, mean_logprob) · one per sentence-aligned chunk of speech - queries(id, text, token_count, was_truncated, at) · kept because a truncated query is the most common reason a search feels broken - runs(id, root, started_at, finished_at, files_seen, files_indexed, video_seconds, wall_seconds) · the numbers that let you predict the next overnight run - Paths are stored relative to a volume, never absolute, so remounting a drive somewhere else does not invalidate the entire index. ### Phase 1 · Knowing which file is which Build: Walk a root, recording each file against its volume with a relative path, size, modification time and a quick hash of the first and last blocks plus the size. Mark work done per file so a restart skips finished ones. Open every file read-only and never move, rename or write to anything. Trap: Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. Done when: Renaming a folder and re-running the index updates paths without re-embedding anything, tested on a sample tree. Killing the indexer and restarting resumes and does not redo completed files, tested. A volume that is not mounted is reported as offline with its label and its files remain searchable as results you cannot open, tested by unmounting. No test run modifies, moves or renames any source file, asserted by comparing modification times across the tree. Do not build yet: No frames, no transcripts, no search and no web page yet. ### Phase 2 · Frames worth embedding Build: Pull keyframes at a configured interval and at every scene cut ffmpeg reports. Compute a perceptual hash per frame and drop any frame too close to the one before it. Write a small thumbnail per surviving frame and record its timestamp in milliseconds. Cap frames per minute. Trap: Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. Done when: A static clip produces a small bounded number of frames, tested with a generated video of one unchanging image. A clip with distinct scenes produces at least one frame per scene, tested with a generated cut. Thumbnails exist for every stored frame and the index refuses a frame whose thumbnail failed to write. The frames-per-minute cap is configurable and enforced, tested at the boundary. Do not build yet: No transcripts, no embeddings, no search yet. ### Phase 3 · The words that were spoken Build: Transcribe with word timestamps. Build segments by accumulating words until a sentence boundary near a target length, rather than cutting at a fixed number of seconds. Store the mean token confidence per segment and let the search filter low-confidence ones out. Trap: Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. Done when: Segments begin and end at sentence boundaries wherever the transcript provides punctuation, checked against a sample with known sentences. Every segment carries start and end timestamps in milliseconds that point at the right moment in the file, verified by opening three at random. A confidence value is stored per segment and the search can exclude below a threshold. A file with no speech produces zero segments and no errors, tested with a silent clip. Do not build yet: No search, no fusion and no web page yet. ### Phase 4 · One sentence, two indexes Build: Embed the query once for the picture index and once for the text index. Retrieve from each separately and combine by rank position rather than by score. Count the query's tokens against the picture model's context limit before embedding and say plainly when it will not fit. Results show a thumbnail, a filename, a timecode and which index matched. Trap: The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. Done when: A query over the picture model's context limit produces a visible warning rather than a crash or a silent truncation, tested with a long sentence. Picture and speech results are combined by rank position and no code path compares a picture similarity to a text similarity, verified by review. Every result carries the file, the timecode and which index produced it. Clicking a result opens the file at that frame in a player, tested on three results by hand. Do not build yet: No rebuild tooling, no export and no rate reporting yet. ### Phase 5 · An index you can rebuild and a library you can move Build: Report indexing rate per run: video seconds processed per wall second, separately for frames and speech, so the next overnight run is predictable. Export the index as a manifest of files, frames and segments with text. Support re-embedding with a new model without re-extracting frames. Handle an offline volume by showing results greyed with the drive label rather than hiding them. Trap: Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts. Done when: Each run prints video seconds per wall second for frames and speech separately, and the figures are stored per run. Re-embedding with a different model completes without re-extracting a single frame or re-transcribing a single file, tested by timing both. Export produces a manifest that reimports into an empty index with the same file, frame and segment counts. Results from an unmounted volume appear with their drive label and cannot be opened, rather than disappearing. Do not build yet: Nothing after this. ### Out of scope (and why) - Face recognition and person tagging, which is a biometric system with consent and retention questions attached and deserves a deliberate decision rather than a feature flag. - Sharing links and guest access, because the moment the index is reachable by someone else you are operating a service with permissions, and that is the half the paid product actually sells. - Review, comments and approval, which is a collaboration product sitting on top of search rather than part of it. - Cloud proxies so an editor can open full resolution without downloading, which is bandwidth and storage rather than code. ### README must contain - That everything runs locally, which models are downloaded on first run and how large they are - A measured indexing rate in video hours per wall hour on your own hardware, taken from a real run rather than estimated - That the picture model has a short query context limit, what happens when a query exceeds it, and that the interface will say so - That picture and speech results are combined by rank because their scores are not comparable - That source files are opened read-only and never moved, and how to delete the index without touching the footage ===== AGENTS.md ===== # Agent instructions · Shade indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python CLI for indexing plus a small local web page for searching, SQLite with a vector extension, one file per library, ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding, A local transcriber producing word timestamps, chunked on sentence boundaries, Two separate indexes combined by rank position, never by comparing scores across them. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. ===== BUILD_PLAN.md ===== # Build plan · Shade A local footage search index whose hard problems are identifying files that move between drives, keeping the frame index from drowning in near-identical stills, and combining picture search with speech search when the two live in different vector spaces. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Knowing which file is which The walker identifies files by content rather than by path, and an interrupted index resumes where it stopped. ### Steps 1. Identify by a content quick hash, not by path Reorganising a library should not cost a night of GPU time. Files: `src/index/identity.py`, `tests/test_identity.py` 2. Store paths relative to a volume Drives get remounted and letters change. Files: `src/index/volumes.py` 3. Open read-only and prove it This tool reads somebody's only copy of a wedding. Files: `src/index/walk.py`, `tests/test_readonly.py` ### Done when - [ ] A renamed folder does not trigger re-embedding - [ ] An interrupted index resumes cleanly - [ ] An offline volume is reported rather than treated as deleted - [ ] No source file is ever modified ### Watch out - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. ## Phase 2 · Frames worth embedding Each video yields a manageable set of visually distinct frames rather than one every few seconds regardless. ### Steps 1. Sample on interval and on scene cuts Cuts are where the content actually changes. Files: `src/frames/extract.py` 2. Deduplicate by perceptual hash before embedding Embedding is the expensive step, so filter before it. Files: `src/frames/dedupe.py`, `tests/test_dedupe.py` 3. Cap frames per minute One boring shot should not own the index. Files: `src/frames/limits.py` ### Done when - [ ] A static clip yields few frames - [ ] Every scene cut yields at least one frame - [ ] Frames without thumbnails are rejected - [ ] The frames-per-minute cap is enforced ### Watch out - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. ## Phase 3 · The words that were spoken Speech becomes timestamped segments chunked where sentences end, with a confidence you can filter on. ### Steps 1. Chunk on sentence boundaries near a target length Half a sentence is a vector for a thing nobody said. Files: `src/speech/chunk.py`, `tests/test_chunk.py` 2. Keep millisecond timestamps on every segment The timecode is the whole point of the result. Files: `src/speech/transcribe.py` 3. Store and expose a confidence per segment Low-confidence speech is where the strange results come from. Files: `src/speech/confidence.py` ### Done when - [ ] Segments respect sentence boundaries - [ ] Timestamps open at the right moment - [ ] Confidence is stored and filterable - [ ] A silent clip produces no segments ### Watch out - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. ## Phase 4 · One sentence, two indexes A plain-English query searches pictures and speech together and returns ranked results with timecodes. ### Steps 1. Count query tokens before embedding and warn The most common broken search is a query nobody told you was too long. Files: `src/search/tokens.py`, `tests/test_tokens.py` 2. Fuse by rank, never by comparing scores Two models, two spaces, two incomparable numbers. Files: `src/search/fuse.py`, `tests/test_fuse.py` 3. Show which index matched, on every result Knowing whether it was seen or said changes what you search next. Files: `src/web/results.html` ### Done when - [ ] An overlong query warns rather than truncating silently - [ ] No code path compares picture and text scores directly - [ ] Every result shows file, timecode and source index - [ ] Clicking a result opens the file at that frame ### Watch out - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. ## Not in this build - Face recognition and person tagging. - Sharing links and guest access. - Review, comments and approval. - Cloud proxies and streaming. ## After v1, if you want it - Shot type and camera movement tags, which are a second small classifier over frames you already extracted - A saved search that re-runs as the library grows, which is the closest thing to a smart folder this design allows - Colour and exposure summaries per frame, cheap to compute and genuinely useful for finding a look rather than a subject ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Vectors, metadata and run history in one file. Disposable but expensive to rebuild. INDEX_PATH=~/FootageIndex/index.db # Required. One small image per surviving frame. Keep these; they make re-embedding cheap. THUMB_DIR=~/FootageIndex/thumbs # Required. Baseline sampling. Scene cuts are added on top of this. FRAME_INTERVAL_SECONDS=5 # Required. The cap that stops a static shot from owning the index. MAX_FRAMES_PER_MINUTE=12 # Required. Frames closer than this to the previous one are dropped before embedding. PHASH_DISTANCE=6 # Required. Chunking aims here but always finishes the sentence. SEGMENT_TARGET_SECONDS=25 # Required. Segments below this are excluded from search by default and remain in the index. MIN_SEGMENT_CONFIDENCE=-1.0
You are building a production product version of Shade. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Shade · product brief ## Problem The search half is real and rebuildable. Extract keyframes with ffmpeg, embed them with CLIP, transcribe the audio with whisper, put the vectors in SQLite, and 'the drone shot over the bridge at golden hour' finds the clip on your own drives. The open-source stack for that is mature and the result is genuinely good. What does not survive the port is what Shade has grown into: cloud streaming so an editor opens full-res without waiting on a download, per-link permissions and guest access, review and approval, and a model pipeline that keeps improving without you retraining anything. Solo, on local storage, the DIY version wins outright. On a team, you are rebuilding a platform and calling it a script. ## Product outcome A search index over your own drives where moving a folder costs nothing, a static interview cannot crowd out the shot you wanted, an overlong query says so instead of failing quietly, and a better model is a re-embed rather than a re-index. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - ffmpeg - a CLIP model via transformers.js or Python - whisper.cpp - sqlite-vec or another local vector store - a GPU, or patience measured in nights ## Explicit non-goals for v1 - Face recognition and person tagging. - Sharing links and guest access. - Review, comments and approval. - Cloud proxies and streaming. - cloud streaming of full-res files without downloading them first - guest links with per-link permissions and roles - built-in review, approval, and commenting - face recognition and shot-type tagging that improves without your involvement - team sync, so everyone searches the same index - the NLE plugins and Slack integration ## Success criteria - No source file is confirmed modified, moved or renamed - A renamed folder is confirmed not to trigger re-embedding - A static clip is confirmed to produce a bounded frame count - An overlong query is confirmed to warn rather than truncate silently - Re-embedding is confirmed to need no re-extraction ===== BRIEF.md ===== # Build brief · Shade The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a search index over my own footage that finds a shot from a plain sentence and runs entirely on my machine to replace Shade. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: A Python CLI for indexing plus a small local web page for searching · indexing runs overnight and searching happens at a keyboard, and those want different interfaces - Index: SQLite with a vector extension, one file per library · one file you can copy, back up and delete, rather than a service to operate - Frames: ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding · an interview locked off for ten minutes should not become a hundred vectors - Speech: A local transcriber producing word timestamps, chunked on sentence boundaries · a chunk that starts mid-sentence embeds as nonsense - Retrieval: Two separate indexes combined by rank position, never by comparing scores across them · picture vectors and sentence vectors live in different spaces and their numbers are not comparable ### Data model (create this before Phase 1) - volumes(id, label, filesystem_uuid, last_seen_at, last_mount_hint) · so an offline drive is reported as offline rather than as deleted footage - files(id, volume_id, rel_path, bytes, mtime, quick_hash, duration_ms, first_seen_at, last_seen_at, state) - frames(id, file_id, t_ms, thumb_path, phash, embedding) · phash is what stops a static shot from producing hundreds of identical vectors - segments(id, file_id, start_ms, end_ms, text, embedding, mean_logprob) · one per sentence-aligned chunk of speech - queries(id, text, token_count, was_truncated, at) · kept because a truncated query is the most common reason a search feels broken - runs(id, root, started_at, finished_at, files_seen, files_indexed, video_seconds, wall_seconds) · the numbers that let you predict the next overnight run - Paths are stored relative to a volume, never absolute, so remounting a drive somewhere else does not invalidate the entire index. ### Phase 1 · Knowing which file is which Build: Walk a root, recording each file against its volume with a relative path, size, modification time and a quick hash of the first and last blocks plus the size. Mark work done per file so a restart skips finished ones. Open every file read-only and never move, rename or write to anything. Trap: Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. Done when: Renaming a folder and re-running the index updates paths without re-embedding anything, tested on a sample tree. Killing the indexer and restarting resumes and does not redo completed files, tested. A volume that is not mounted is reported as offline with its label and its files remain searchable as results you cannot open, tested by unmounting. No test run modifies, moves or renames any source file, asserted by comparing modification times across the tree. Do not build yet: No frames, no transcripts, no search and no web page yet. ### Phase 2 · Frames worth embedding Build: Pull keyframes at a configured interval and at every scene cut ffmpeg reports. Compute a perceptual hash per frame and drop any frame too close to the one before it. Write a small thumbnail per surviving frame and record its timestamp in milliseconds. Cap frames per minute. Trap: Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. Done when: A static clip produces a small bounded number of frames, tested with a generated video of one unchanging image. A clip with distinct scenes produces at least one frame per scene, tested with a generated cut. Thumbnails exist for every stored frame and the index refuses a frame whose thumbnail failed to write. The frames-per-minute cap is configurable and enforced, tested at the boundary. Do not build yet: No transcripts, no embeddings, no search yet. ### Phase 3 · The words that were spoken Build: Transcribe with word timestamps. Build segments by accumulating words until a sentence boundary near a target length, rather than cutting at a fixed number of seconds. Store the mean token confidence per segment and let the search filter low-confidence ones out. Trap: Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. Done when: Segments begin and end at sentence boundaries wherever the transcript provides punctuation, checked against a sample with known sentences. Every segment carries start and end timestamps in milliseconds that point at the right moment in the file, verified by opening three at random. A confidence value is stored per segment and the search can exclude below a threshold. A file with no speech produces zero segments and no errors, tested with a silent clip. Do not build yet: No search, no fusion and no web page yet. ### Phase 4 · One sentence, two indexes Build: Embed the query once for the picture index and once for the text index. Retrieve from each separately and combine by rank position rather than by score. Count the query's tokens against the picture model's context limit before embedding and say plainly when it will not fit. Results show a thumbnail, a filename, a timecode and which index matched. Trap: The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. Done when: A query over the picture model's context limit produces a visible warning rather than a crash or a silent truncation, tested with a long sentence. Picture and speech results are combined by rank position and no code path compares a picture similarity to a text similarity, verified by review. Every result carries the file, the timecode and which index produced it. Clicking a result opens the file at that frame in a player, tested on three results by hand. Do not build yet: No rebuild tooling, no export and no rate reporting yet. ### Phase 5 · An index you can rebuild and a library you can move Build: Report indexing rate per run: video seconds processed per wall second, separately for frames and speech, so the next overnight run is predictable. Export the index as a manifest of files, frames and segments with text. Support re-embedding with a new model without re-extracting frames. Handle an offline volume by showing results greyed with the drive label rather than hiding them. Trap: Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts. Done when: Each run prints video seconds per wall second for frames and speech separately, and the figures are stored per run. Re-embedding with a different model completes without re-extracting a single frame or re-transcribing a single file, tested by timing both. Export produces a manifest that reimports into an empty index with the same file, frame and segment counts. Results from an unmounted volume appear with their drive label and cannot be opened, rather than disappearing. Do not build yet: Nothing after this. ### Out of scope (and why) - Face recognition and person tagging, which is a biometric system with consent and retention questions attached and deserves a deliberate decision rather than a feature flag. - Sharing links and guest access, because the moment the index is reachable by someone else you are operating a service with permissions, and that is the half the paid product actually sells. - Review, comments and approval, which is a collaboration product sitting on top of search rather than part of it. - Cloud proxies so an editor can open full resolution without downloading, which is bandwidth and storage rather than code. ### README must contain - That everything runs locally, which models are downloaded on first run and how large they are - A measured indexing rate in video hours per wall hour on your own hardware, taken from a real run rather than estimated - That the picture model has a short query context limit, what happens when a query exceeds it, and that the interface will say so - That picture and speech results are combined by rank because their scores are not comparable - That source files are opened read-only and never moved, and how to delete the index without touching the footage ===== ARCHITECTURE.md ===== # Architecture · Shade ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python CLI for indexing plus a small local web page for searching | indexing runs overnight and searching happens at a keyboard, and those want different interfaces | | Index | SQLite with a vector extension, one file per library | one file you can copy, back up and delete, rather than a service to operate | | Frames | ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding | an interview locked off for ten minutes should not become a hundred vectors | | Speech | A local transcriber producing word timestamps, chunked on sentence boundaries | a chunk that starts mid-sentence embeds as nonsense | | Retrieval | Two separate indexes combined by rank position, never by comparing scores across them | picture vectors and sentence vectors live in different spaces and their numbers are not comparable | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Walker | Volumes, content identity and resumable state. | Any traversal, provided identity is content-based and paths stay relative. | | Frames | Extraction, deduplication, thumbnails and caps. | Any sampler, provided deduplication happens before embedding. | | Speech | Transcription, sentence chunking and confidence. | Any transcriber with word timestamps. | | Retrieval | Two indexes and rank fusion. | Any fusion, provided scores from different models are never compared. | | Rebuild | Rate measurement, re-embedding and export. | Any format, provided frames and transcripts survive independently of the vectors. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `INDEX_PATH` · required · Vectors, metadata and run history in one file. Disposable but expensive to rebuild. - `THUMB_DIR` · required · One small image per surviving frame. Keep these; they make re-embedding cheap. - `FRAME_INTERVAL_SECONDS` · required · Baseline sampling. Scene cuts are added on top of this. - `MAX_FRAMES_PER_MINUTE` · required · The cap that stops a static shot from owning the index. - `PHASH_DISTANCE` · required · Frames closer than this to the previous one are dropped before embedding. - `SEGMENT_TARGET_SECONDS` · required · Chunking aims here but always finishes the sentence. - `MIN_SEGMENT_CONFIDENCE` · required · Segments below this are excluded from search by default and remain in the index. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Shade product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A Python CLI for indexing plus a small local web page for searching, SQLite with a vector extension, one file per library, ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding, A local transcriber producing word timestamps, chunked on sentence boundaries, Two separate indexes combined by rank position, never by comparing scores across them. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. - Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts. ===== MILESTONES.md ===== # Delivery milestones · Shade Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Knowing which file is which The walker identifies files by content rather than by path, and an interrupted index resumes where it stopped. ### Steps 1. Identify by a content quick hash, not by path Reorganising a library should not cost a night of GPU time. Files: `src/index/identity.py`, `tests/test_identity.py` 2. Store paths relative to a volume Drives get remounted and letters change. Files: `src/index/volumes.py` 3. Open read-only and prove it This tool reads somebody's only copy of a wedding. Files: `src/index/walk.py`, `tests/test_readonly.py` ### Done when - [ ] A renamed folder does not trigger re-embedding - [ ] An interrupted index resumes cleanly - [ ] An offline volume is reported rather than treated as deleted - [ ] No source file is ever modified ### Watch out - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. ## M2 · Frames worth embedding Each video yields a manageable set of visually distinct frames rather than one every few seconds regardless. ### Steps 1. Sample on interval and on scene cuts Cuts are where the content actually changes. Files: `src/frames/extract.py` 2. Deduplicate by perceptual hash before embedding Embedding is the expensive step, so filter before it. Files: `src/frames/dedupe.py`, `tests/test_dedupe.py` 3. Cap frames per minute One boring shot should not own the index. Files: `src/frames/limits.py` ### Done when - [ ] A static clip yields few frames - [ ] Every scene cut yields at least one frame - [ ] Frames without thumbnails are rejected - [ ] The frames-per-minute cap is enforced ### Watch out - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. ## M3 · The words that were spoken Speech becomes timestamped segments chunked where sentences end, with a confidence you can filter on. ### Steps 1. Chunk on sentence boundaries near a target length Half a sentence is a vector for a thing nobody said. Files: `src/speech/chunk.py`, `tests/test_chunk.py` 2. Keep millisecond timestamps on every segment The timecode is the whole point of the result. Files: `src/speech/transcribe.py` 3. Store and expose a confidence per segment Low-confidence speech is where the strange results come from. Files: `src/speech/confidence.py` ### Done when - [ ] Segments respect sentence boundaries - [ ] Timestamps open at the right moment - [ ] Confidence is stored and filterable - [ ] A silent clip produces no segments ### Watch out - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. ## M4 · One sentence, two indexes A plain-English query searches pictures and speech together and returns ranked results with timecodes. ### Steps 1. Count query tokens before embedding and warn The most common broken search is a query nobody told you was too long. Files: `src/search/tokens.py`, `tests/test_tokens.py` 2. Fuse by rank, never by comparing scores Two models, two spaces, two incomparable numbers. Files: `src/search/fuse.py`, `tests/test_fuse.py` 3. Show which index matched, on every result Knowing whether it was seen or said changes what you search next. Files: `src/web/results.html` ### Done when - [ ] An overlong query warns rather than truncating silently - [ ] No code path compares picture and text scores directly - [ ] Every result shows file, timecode and source index - [ ] Clicking a result opens the file at that frame ### Watch out - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. ## M5 · An index you can rebuild and a library you can move (production only) The index is disposable, the numbers to plan a re-index are measured, and the library never depends on this tool. ### Steps 1. Measure and print the indexing rate per run This is the number that tells you whether a terabyte is a night or a week. Files: `src/index/rate.py` 2. Keep frames and transcripts so re-embedding is cheap Extraction is the expensive half and it is model-independent. Files: `src/index/reembed.py` 3. Show offline volumes rather than hiding them Footage on a drive in a drawer is still footage. Files: `src/web/offline.py` ### Done when - [ ] Indexing rate is measured and stored per run - [ ] Re-embedding needs no re-extraction - [ ] Export and reimport preserve every count - [ ] Offline volumes appear labelled, not missing ### Watch out - Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts. ===== OPERATIONS.md ===== # Operations · Shade ## Backup The thumbnails and transcripts first, the vectors second. The vectors can be recomputed from the stored artefacts in an evening; the extraction that produced those artefacts is the part that took the week. ## Restore Restore the index file and the thumbnail directory, then run the walker against the same roots and confirm no file is re-extracted. If files are being re-extracted, the volume identifiers did not survive the restore, which is the one thing worth checking before going to bed. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring For a local tool this lives in the runs table: video seconds per wall second, the share of frames dropped by deduplication and the share of segments below the confidence threshold. A sudden change in the dropped-frame share usually means the footage changed character, not that the code did. ## Incident checklist The failure that matters is a search that returns nothing useful for footage you know is there. Check three things in order: whether the file is indexed at all, whether the query was truncated, and whether the matching frames were deduplicated away. The queries table and the frame counts answer all three. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] No source file is confirmed modified, moved or renamed - [ ] A renamed folder is confirmed not to trigger re-embedding - [ ] A static clip is confirmed to produce a bounded frame count - [ ] An overlong query is confirmed to warn rather than truncate silently - [ ] Re-embedding is confirmed to need no re-extraction ## Launch constraint Do not market omitted Shade capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Vectors, metadata and run history in one file. Disposable but expensive to rebuild. INDEX_PATH=~/FootageIndex/index.db # Required. One small image per surviving frame. Keep these; they make re-embedding cheap. THUMB_DIR=~/FootageIndex/thumbs # Required. Baseline sampling. Scene cuts are added on top of this. FRAME_INTERVAL_SECONDS=5 # Required. The cap that stops a static shot from owning the index. MAX_FRAMES_PER_MINUTE=12 # Required. Frames closer than this to the previous one are dropped before embedding. PHASH_DISTANCE=6 # Required. Chunking aims here but always finishes the sentence. SEGMENT_TARGET_SECONDS=25 # Required. Segments below this are excluded from search by default and remain in the index. MIN_SEGMENT_CONFIDENCE=-1.0
README.md
# Shade · indie build A local footage search index whose hard problems are identifying files that move between drives, keeping the frame index from drowning in near-identical stills, and combining picture search with speech search when the two live in different vector spaces. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python CLI for indexing plus a small local web page for searching | indexing runs overnight and searching happens at a keyboard, and those want different interfaces | | Index | SQLite with a vector extension, one file per library | one file you can copy, back up and delete, rather than a service to operate | | Frames | ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding | an interview locked off for ten minutes should not become a hundred vectors | | Speech | A local transcriber producing word timestamps, chunked on sentence boundaries | a chunk that starts mid-sentence embeds as nonsense | | Retrieval | Two separate indexes combined by rank position, never by comparing scores across them | picture vectors and sentence vectors live in different spaces and their numbers are not comparable | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **ffmpeg with scene detection available** · free - Why: Every frame in this index comes out of it, and the scene filter is what makes the sampling worth anything. - Get it: Your package manager, or a static build. - Verify: ffmpeg -version prints and a scene filter run produces frames - [ ] **Python 3.11 or newer with a vector-capable SQLite build** · free - Why: Runs the indexer and holds the vectors in the same file as the metadata. - Get it: python.org or your package manager, plus a SQLite vector extension. - Verify: A vector query returns results against a test table - [ ] **A picture and text embedding model and a local speech model, downloaded** · free; disk space is the cost - Why: First-run downloads are large and slow, and discovering that at the start of an overnight index wastes the night. - Get it: Download the weights before your first real run and note their sizes. - Verify: Both models load and embed a test input offline with the network disabled - [ ] **A GPU, or an honest estimate of how many nights you are prepared to spend** (optional) · free if you own one - Why: The difference between hardware here is the difference between a weekend and a fortnight, and it is better known before you start than after. - Get it: Whatever you have. Run the rate measurement on one hour of footage first. - Verify: A measured rate exists for one hour of your own footage - [ ] **One hour of your real footage including a static interview and a clip with speech** · free - Why: Those two clips expose the frame explosion and the chunking problems respectively, and a synthetic test will not. - Get it: Pick them from the library you intend to index. - Verify: Both index without the frame cap or the segment count looking absurd ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Face recognition and person tagging. - Sharing links and guest access. - Review, comments and approval. - Cloud proxies and streaming. - cloud streaming of full-res files without downloading them first - guest links with per-link permissions and roles - built-in review, approval, and commenting - face recognition and shot-type tagging that improves without your involvement - team sync, so everyone searches the same index - the NLE plugins and Slack integration If one of those is essential to you, that is the reason to keep paying for Shade, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Shade The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a search index over my own footage that finds a shot from a plain sentence and runs entirely on my machine to replace Shade. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: A Python CLI for indexing plus a small local web page for searching · indexing runs overnight and searching happens at a keyboard, and those want different interfaces - Index: SQLite with a vector extension, one file per library · one file you can copy, back up and delete, rather than a service to operate - Frames: ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding · an interview locked off for ten minutes should not become a hundred vectors - Speech: A local transcriber producing word timestamps, chunked on sentence boundaries · a chunk that starts mid-sentence embeds as nonsense - Retrieval: Two separate indexes combined by rank position, never by comparing scores across them · picture vectors and sentence vectors live in different spaces and their numbers are not comparable ### Data model (create this before Phase 1) - volumes(id, label, filesystem_uuid, last_seen_at, last_mount_hint) · so an offline drive is reported as offline rather than as deleted footage - files(id, volume_id, rel_path, bytes, mtime, quick_hash, duration_ms, first_seen_at, last_seen_at, state) - frames(id, file_id, t_ms, thumb_path, phash, embedding) · phash is what stops a static shot from producing hundreds of identical vectors - segments(id, file_id, start_ms, end_ms, text, embedding, mean_logprob) · one per sentence-aligned chunk of speech - queries(id, text, token_count, was_truncated, at) · kept because a truncated query is the most common reason a search feels broken - runs(id, root, started_at, finished_at, files_seen, files_indexed, video_seconds, wall_seconds) · the numbers that let you predict the next overnight run - Paths are stored relative to a volume, never absolute, so remounting a drive somewhere else does not invalidate the entire index. ### Phase 1 · Knowing which file is which Build: Walk a root, recording each file against its volume with a relative path, size, modification time and a quick hash of the first and last blocks plus the size. Mark work done per file so a restart skips finished ones. Open every file read-only and never move, rename or write to anything. Trap: Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. Done when: Renaming a folder and re-running the index updates paths without re-embedding anything, tested on a sample tree. Killing the indexer and restarting resumes and does not redo completed files, tested. A volume that is not mounted is reported as offline with its label and its files remain searchable as results you cannot open, tested by unmounting. No test run modifies, moves or renames any source file, asserted by comparing modification times across the tree. Do not build yet: No frames, no transcripts, no search and no web page yet. ### Phase 2 · Frames worth embedding Build: Pull keyframes at a configured interval and at every scene cut ffmpeg reports. Compute a perceptual hash per frame and drop any frame too close to the one before it. Write a small thumbnail per surviving frame and record its timestamp in milliseconds. Cap frames per minute. Trap: Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. Done when: A static clip produces a small bounded number of frames, tested with a generated video of one unchanging image. A clip with distinct scenes produces at least one frame per scene, tested with a generated cut. Thumbnails exist for every stored frame and the index refuses a frame whose thumbnail failed to write. The frames-per-minute cap is configurable and enforced, tested at the boundary. Do not build yet: No transcripts, no embeddings, no search yet. ### Phase 3 · The words that were spoken Build: Transcribe with word timestamps. Build segments by accumulating words until a sentence boundary near a target length, rather than cutting at a fixed number of seconds. Store the mean token confidence per segment and let the search filter low-confidence ones out. Trap: Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. Done when: Segments begin and end at sentence boundaries wherever the transcript provides punctuation, checked against a sample with known sentences. Every segment carries start and end timestamps in milliseconds that point at the right moment in the file, verified by opening three at random. A confidence value is stored per segment and the search can exclude below a threshold. A file with no speech produces zero segments and no errors, tested with a silent clip. Do not build yet: No search, no fusion and no web page yet. ### Phase 4 · One sentence, two indexes Build: Embed the query once for the picture index and once for the text index. Retrieve from each separately and combine by rank position rather than by score. Count the query's tokens against the picture model's context limit before embedding and say plainly when it will not fit. Results show a thumbnail, a filename, a timecode and which index matched. Trap: The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. Done when: A query over the picture model's context limit produces a visible warning rather than a crash or a silent truncation, tested with a long sentence. Picture and speech results are combined by rank position and no code path compares a picture similarity to a text similarity, verified by review. Every result carries the file, the timecode and which index produced it. Clicking a result opens the file at that frame in a player, tested on three results by hand. Do not build yet: No rebuild tooling, no export and no rate reporting yet. ### Phase 5 · An index you can rebuild and a library you can move Build: Report indexing rate per run: video seconds processed per wall second, separately for frames and speech, so the next overnight run is predictable. Export the index as a manifest of files, frames and segments with text. Support re-embedding with a new model without re-extracting frames. Handle an offline volume by showing results greyed with the drive label rather than hiding them. Trap: Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts. Done when: Each run prints video seconds per wall second for frames and speech separately, and the figures are stored per run. Re-embedding with a different model completes without re-extracting a single frame or re-transcribing a single file, tested by timing both. Export produces a manifest that reimports into an empty index with the same file, frame and segment counts. Results from an unmounted volume appear with their drive label and cannot be opened, rather than disappearing. Do not build yet: Nothing after this. ### Out of scope (and why) - Face recognition and person tagging, which is a biometric system with consent and retention questions attached and deserves a deliberate decision rather than a feature flag. - Sharing links and guest access, because the moment the index is reachable by someone else you are operating a service with permissions, and that is the half the paid product actually sells. - Review, comments and approval, which is a collaboration product sitting on top of search rather than part of it. - Cloud proxies so an editor can open full resolution without downloading, which is bandwidth and storage rather than code. ### README must contain - That everything runs locally, which models are downloaded on first run and how large they are - A measured indexing rate in video hours per wall hour on your own hardware, taken from a real run rather than estimated - That the picture model has a short query context limit, what happens when a query exceeds it, and that the interface will say so - That picture and speech results are combined by rank because their scores are not comparable - That source files are opened read-only and never moved, and how to delete the index without touching the footage
AGENTS.md
# Agent instructions · Shade indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: A Python CLI for indexing plus a small local web page for searching, SQLite with a vector extension, one file per library, ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding, A local transcriber producing word timestamps, chunked on sentence boundaries, Two separate indexes combined by rank position, never by comparing scores across them. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results.
BUILD_PLAN.md
# Build plan · Shade A local footage search index whose hard problems are identifying files that move between drives, keeping the frame index from drowning in near-identical stills, and combining picture search with speech search when the two live in different vector spaces. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Knowing which file is which The walker identifies files by content rather than by path, and an interrupted index resumes where it stopped. ### Steps 1. Identify by a content quick hash, not by path Reorganising a library should not cost a night of GPU time. Files: `src/index/identity.py`, `tests/test_identity.py` 2. Store paths relative to a volume Drives get remounted and letters change. Files: `src/index/volumes.py` 3. Open read-only and prove it This tool reads somebody's only copy of a wedding. Files: `src/index/walk.py`, `tests/test_readonly.py` ### Done when - [ ] A renamed folder does not trigger re-embedding - [ ] An interrupted index resumes cleanly - [ ] An offline volume is reported rather than treated as deleted - [ ] No source file is ever modified ### Watch out - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. ## Phase 2 · Frames worth embedding Each video yields a manageable set of visually distinct frames rather than one every few seconds regardless. ### Steps 1. Sample on interval and on scene cuts Cuts are where the content actually changes. Files: `src/frames/extract.py` 2. Deduplicate by perceptual hash before embedding Embedding is the expensive step, so filter before it. Files: `src/frames/dedupe.py`, `tests/test_dedupe.py` 3. Cap frames per minute One boring shot should not own the index. Files: `src/frames/limits.py` ### Done when - [ ] A static clip yields few frames - [ ] Every scene cut yields at least one frame - [ ] Frames without thumbnails are rejected - [ ] The frames-per-minute cap is enforced ### Watch out - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. ## Phase 3 · The words that were spoken Speech becomes timestamped segments chunked where sentences end, with a confidence you can filter on. ### Steps 1. Chunk on sentence boundaries near a target length Half a sentence is a vector for a thing nobody said. Files: `src/speech/chunk.py`, `tests/test_chunk.py` 2. Keep millisecond timestamps on every segment The timecode is the whole point of the result. Files: `src/speech/transcribe.py` 3. Store and expose a confidence per segment Low-confidence speech is where the strange results come from. Files: `src/speech/confidence.py` ### Done when - [ ] Segments respect sentence boundaries - [ ] Timestamps open at the right moment - [ ] Confidence is stored and filterable - [ ] A silent clip produces no segments ### Watch out - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. ## Phase 4 · One sentence, two indexes A plain-English query searches pictures and speech together and returns ranked results with timecodes. ### Steps 1. Count query tokens before embedding and warn The most common broken search is a query nobody told you was too long. Files: `src/search/tokens.py`, `tests/test_tokens.py` 2. Fuse by rank, never by comparing scores Two models, two spaces, two incomparable numbers. Files: `src/search/fuse.py`, `tests/test_fuse.py` 3. Show which index matched, on every result Knowing whether it was seen or said changes what you search next. Files: `src/web/results.html` ### Done when - [ ] An overlong query warns rather than truncating silently - [ ] No code path compares picture and text scores directly - [ ] Every result shows file, timecode and source index - [ ] Clicking a result opens the file at that frame ### Watch out - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. ## Not in this build - Face recognition and person tagging. - Sharing links and guest access. - Review, comments and approval. - Cloud proxies and streaming. ## After v1, if you want it - Shot type and camera movement tags, which are a second small classifier over frames you already extracted - A saved search that re-runs as the library grows, which is the closest thing to a smart folder this design allows - Colour and exposure summaries per frame, cheap to compute and genuinely useful for finding a look rather than a subject
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Vectors, metadata and run history in one file. Disposable but expensive to rebuild. INDEX_PATH=~/FootageIndex/index.db # Required. One small image per surviving frame. Keep these; they make re-embedding cheap. THUMB_DIR=~/FootageIndex/thumbs # Required. Baseline sampling. Scene cuts are added on top of this. FRAME_INTERVAL_SECONDS=5 # Required. The cap that stops a static shot from owning the index. MAX_FRAMES_PER_MINUTE=12 # Required. Frames closer than this to the previous one are dropped before embedding. PHASH_DISTANCE=6 # Required. Chunking aims here but always finishes the sentence. SEGMENT_TARGET_SECONDS=25 # Required. Segments below this are excluded from search by default and remain in the index. MIN_SEGMENT_CONFIDENCE=-1.0
PRODUCT.md
# Shade · product brief ## Problem The search half is real and rebuildable. Extract keyframes with ffmpeg, embed them with CLIP, transcribe the audio with whisper, put the vectors in SQLite, and 'the drone shot over the bridge at golden hour' finds the clip on your own drives. The open-source stack for that is mature and the result is genuinely good. What does not survive the port is what Shade has grown into: cloud streaming so an editor opens full-res without waiting on a download, per-link permissions and guest access, review and approval, and a model pipeline that keeps improving without you retraining anything. Solo, on local storage, the DIY version wins outright. On a team, you are rebuilding a platform and calling it a script. ## Product outcome A search index over your own drives where moving a folder costs nothing, a static interview cannot crowd out the shot you wanted, an overlong query says so instead of failing quietly, and a better model is a re-embed rather than a re-index. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - ffmpeg - a CLIP model via transformers.js or Python - whisper.cpp - sqlite-vec or another local vector store - a GPU, or patience measured in nights ## Explicit non-goals for v1 - Face recognition and person tagging. - Sharing links and guest access. - Review, comments and approval. - Cloud proxies and streaming. - cloud streaming of full-res files without downloading them first - guest links with per-link permissions and roles - built-in review, approval, and commenting - face recognition and shot-type tagging that improves without your involvement - team sync, so everyone searches the same index - the NLE plugins and Slack integration ## Success criteria - No source file is confirmed modified, moved or renamed - A renamed folder is confirmed not to trigger re-embedding - A static clip is confirmed to produce a bounded frame count - An overlong query is confirmed to warn rather than truncate silently - Re-embedding is confirmed to need no re-extraction
BRIEF.md
# Build brief · Shade The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a search index over my own footage that finds a shot from a plain sentence and runs entirely on my machine to replace Shade. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: A Python CLI for indexing plus a small local web page for searching · indexing runs overnight and searching happens at a keyboard, and those want different interfaces - Index: SQLite with a vector extension, one file per library · one file you can copy, back up and delete, rather than a service to operate - Frames: ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding · an interview locked off for ten minutes should not become a hundred vectors - Speech: A local transcriber producing word timestamps, chunked on sentence boundaries · a chunk that starts mid-sentence embeds as nonsense - Retrieval: Two separate indexes combined by rank position, never by comparing scores across them · picture vectors and sentence vectors live in different spaces and their numbers are not comparable ### Data model (create this before Phase 1) - volumes(id, label, filesystem_uuid, last_seen_at, last_mount_hint) · so an offline drive is reported as offline rather than as deleted footage - files(id, volume_id, rel_path, bytes, mtime, quick_hash, duration_ms, first_seen_at, last_seen_at, state) - frames(id, file_id, t_ms, thumb_path, phash, embedding) · phash is what stops a static shot from producing hundreds of identical vectors - segments(id, file_id, start_ms, end_ms, text, embedding, mean_logprob) · one per sentence-aligned chunk of speech - queries(id, text, token_count, was_truncated, at) · kept because a truncated query is the most common reason a search feels broken - runs(id, root, started_at, finished_at, files_seen, files_indexed, video_seconds, wall_seconds) · the numbers that let you predict the next overnight run - Paths are stored relative to a volume, never absolute, so remounting a drive somewhere else does not invalidate the entire index. ### Phase 1 · Knowing which file is which Build: Walk a root, recording each file against its volume with a relative path, size, modification time and a quick hash of the first and last blocks plus the size. Mark work done per file so a restart skips finished ones. Open every file read-only and never move, rename or write to anything. Trap: Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. Done when: Renaming a folder and re-running the index updates paths without re-embedding anything, tested on a sample tree. Killing the indexer and restarting resumes and does not redo completed files, tested. A volume that is not mounted is reported as offline with its label and its files remain searchable as results you cannot open, tested by unmounting. No test run modifies, moves or renames any source file, asserted by comparing modification times across the tree. Do not build yet: No frames, no transcripts, no search and no web page yet. ### Phase 2 · Frames worth embedding Build: Pull keyframes at a configured interval and at every scene cut ffmpeg reports. Compute a perceptual hash per frame and drop any frame too close to the one before it. Write a small thumbnail per surviving frame and record its timestamp in milliseconds. Cap frames per minute. Trap: Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. Done when: A static clip produces a small bounded number of frames, tested with a generated video of one unchanging image. A clip with distinct scenes produces at least one frame per scene, tested with a generated cut. Thumbnails exist for every stored frame and the index refuses a frame whose thumbnail failed to write. The frames-per-minute cap is configurable and enforced, tested at the boundary. Do not build yet: No transcripts, no embeddings, no search yet. ### Phase 3 · The words that were spoken Build: Transcribe with word timestamps. Build segments by accumulating words until a sentence boundary near a target length, rather than cutting at a fixed number of seconds. Store the mean token confidence per segment and let the search filter low-confidence ones out. Trap: Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. Done when: Segments begin and end at sentence boundaries wherever the transcript provides punctuation, checked against a sample with known sentences. Every segment carries start and end timestamps in milliseconds that point at the right moment in the file, verified by opening three at random. A confidence value is stored per segment and the search can exclude below a threshold. A file with no speech produces zero segments and no errors, tested with a silent clip. Do not build yet: No search, no fusion and no web page yet. ### Phase 4 · One sentence, two indexes Build: Embed the query once for the picture index and once for the text index. Retrieve from each separately and combine by rank position rather than by score. Count the query's tokens against the picture model's context limit before embedding and say plainly when it will not fit. Results show a thumbnail, a filename, a timecode and which index matched. Trap: The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. Done when: A query over the picture model's context limit produces a visible warning rather than a crash or a silent truncation, tested with a long sentence. Picture and speech results are combined by rank position and no code path compares a picture similarity to a text similarity, verified by review. Every result carries the file, the timecode and which index produced it. Clicking a result opens the file at that frame in a player, tested on three results by hand. Do not build yet: No rebuild tooling, no export and no rate reporting yet. ### Phase 5 · An index you can rebuild and a library you can move Build: Report indexing rate per run: video seconds processed per wall second, separately for frames and speech, so the next overnight run is predictable. Export the index as a manifest of files, frames and segments with text. Support re-embedding with a new model without re-extracting frames. Handle an offline volume by showing results greyed with the drive label rather than hiding them. Trap: Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts. Done when: Each run prints video seconds per wall second for frames and speech separately, and the figures are stored per run. Re-embedding with a different model completes without re-extracting a single frame or re-transcribing a single file, tested by timing both. Export produces a manifest that reimports into an empty index with the same file, frame and segment counts. Results from an unmounted volume appear with their drive label and cannot be opened, rather than disappearing. Do not build yet: Nothing after this. ### Out of scope (and why) - Face recognition and person tagging, which is a biometric system with consent and retention questions attached and deserves a deliberate decision rather than a feature flag. - Sharing links and guest access, because the moment the index is reachable by someone else you are operating a service with permissions, and that is the half the paid product actually sells. - Review, comments and approval, which is a collaboration product sitting on top of search rather than part of it. - Cloud proxies so an editor can open full resolution without downloading, which is bandwidth and storage rather than code. ### README must contain - That everything runs locally, which models are downloaded on first run and how large they are - A measured indexing rate in video hours per wall hour on your own hardware, taken from a real run rather than estimated - That the picture model has a short query context limit, what happens when a query exceeds it, and that the interface will say so - That picture and speech results are combined by rank because their scores are not comparable - That source files are opened read-only and never moved, and how to delete the index without touching the footage
ARCHITECTURE.md
# Architecture · Shade ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | A Python CLI for indexing plus a small local web page for searching | indexing runs overnight and searching happens at a keyboard, and those want different interfaces | | Index | SQLite with a vector extension, one file per library | one file you can copy, back up and delete, rather than a service to operate | | Frames | ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding | an interview locked off for ten minutes should not become a hundred vectors | | Speech | A local transcriber producing word timestamps, chunked on sentence boundaries | a chunk that starts mid-sentence embeds as nonsense | | Retrieval | Two separate indexes combined by rank position, never by comparing scores across them | picture vectors and sentence vectors live in different spaces and their numbers are not comparable | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Walker | Volumes, content identity and resumable state. | Any traversal, provided identity is content-based and paths stay relative. | | Frames | Extraction, deduplication, thumbnails and caps. | Any sampler, provided deduplication happens before embedding. | | Speech | Transcription, sentence chunking and confidence. | Any transcriber with word timestamps. | | Retrieval | Two indexes and rank fusion. | Any fusion, provided scores from different models are never compared. | | Rebuild | Rate measurement, re-embedding and export. | Any format, provided frames and transcripts survive independently of the vectors. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `INDEX_PATH` · required · Vectors, metadata and run history in one file. Disposable but expensive to rebuild. - `THUMB_DIR` · required · One small image per surviving frame. Keep these; they make re-embedding cheap. - `FRAME_INTERVAL_SECONDS` · required · Baseline sampling. Scene cuts are added on top of this. - `MAX_FRAMES_PER_MINUTE` · required · The cap that stops a static shot from owning the index. - `PHASH_DISTANCE` · required · Frames closer than this to the previous one are dropped before embedding. - `SEGMENT_TARGET_SECONDS` · required · Chunking aims here but always finishes the sentence. - `MIN_SEGMENT_CONFIDENCE` · required · Segments below this are excluded from search by default and remain in the index. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Shade product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: A Python CLI for indexing plus a small local web page for searching, SQLite with a vector extension, one file per library, ffmpeg keyframes at a fixed interval plus scene cut detection, deduplicated before embedding, A local transcriber producing word timestamps, chunked on sentence boundaries, Two separate indexes combined by rank position, never by comparing scores across them. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. - Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts.
MILESTONES.md
# Delivery milestones · Shade Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Knowing which file is which The walker identifies files by content rather than by path, and an interrupted index resumes where it stopped. ### Steps 1. Identify by a content quick hash, not by path Reorganising a library should not cost a night of GPU time. Files: `src/index/identity.py`, `tests/test_identity.py` 2. Store paths relative to a volume Drives get remounted and letters change. Files: `src/index/volumes.py` 3. Open read-only and prove it This tool reads somebody's only copy of a wedding. Files: `src/index/walk.py`, `tests/test_readonly.py` ### Done when - [ ] A renamed folder does not trigger re-embedding - [ ] An interrupted index resumes cleanly - [ ] An offline volume is reported rather than treated as deleted - [ ] No source file is ever modified ### Watch out - Keying the index on the absolute path means the day you reorganise a folder or mount the drive at a different letter, every file looks new, and you re-index a terabyte to discover you already had it. Key on a content quick hash and store paths relative to a volume identifier, so a move is an update to one column. ## M2 · Frames worth embedding Each video yields a manageable set of visually distinct frames rather than one every few seconds regardless. ### Steps 1. Sample on interval and on scene cuts Cuts are where the content actually changes. Files: `src/frames/extract.py` 2. Deduplicate by perceptual hash before embedding Embedding is the expensive step, so filter before it. Files: `src/frames/dedupe.py`, `tests/test_dedupe.py` 3. Cap frames per minute One boring shot should not own the index. Files: `src/frames/limits.py` ### Done when - [ ] A static clip yields few frames - [ ] Every scene cut yields at least one frame - [ ] Frames without thumbnails are rejected - [ ] The frames-per-minute cap is enforced ### Watch out - Embedding a frame every few seconds regardless of content means a locked-off ten-minute interview contributes hundreds of nearly identical vectors, and those vectors then crowd out the one drone shot you were actually looking for because they are individually decent matches for almost any query. Deduplicate by perceptual hash before embedding, and cap frames per minute so a static shot cannot dominate the index. ## M3 · The words that were spoken Speech becomes timestamped segments chunked where sentences end, with a confidence you can filter on. ### Steps 1. Chunk on sentence boundaries near a target length Half a sentence is a vector for a thing nobody said. Files: `src/speech/chunk.py`, `tests/test_chunk.py` 2. Keep millisecond timestamps on every segment The timecode is the whole point of the result. Files: `src/speech/transcribe.py` 3. Store and expose a confidence per segment Low-confidence speech is where the strange results come from. Files: `src/speech/confidence.py` ### Done when - [ ] Segments respect sentence boundaries - [ ] Timestamps open at the right moment - [ ] Confidence is stored and filterable - [ ] A silent clip produces no segments ### Watch out - Chunking the transcript into fixed windows cuts sentences in half, and half a sentence embeds as something that means neither what the speaker said nor anything else, so the segment sits in the index matching queries it has nothing to do with. Accumulate words to a sentence boundary near your target length, and keep the boundary timestamps so a result still opens at the right frame. ## M4 · One sentence, two indexes A plain-English query searches pictures and speech together and returns ranked results with timecodes. ### Steps 1. Count query tokens before embedding and warn The most common broken search is a query nobody told you was too long. Files: `src/search/tokens.py`, `tests/test_tokens.py` 2. Fuse by rank, never by comparing scores Two models, two spaces, two incomparable numbers. Files: `src/search/fuse.py`, `tests/test_fuse.py` 3. Show which index matched, on every result Knowing whether it was seen or said changes what you search next. Files: `src/web/results.html` ### Done when - [ ] An overlong query warns rather than truncating silently - [ ] No code path compares picture and text scores directly - [ ] Every result shows file, timecode and source index - [ ] Clicking a result opens the file at that frame ### Watch out - The reference CLIP tokeniser defaults to a context length of 77 and its tokenize function raises rather than truncating unless you ask it to truncate, in which case the tail is silently dropped and the last token replaced with the end marker, so a paragraph-long query either crashes the search or quietly searches only its opening clause while the interface shows no sign of it. Count the tokens first, tell the user the query is too long, and offer to split it into clauses and merge the results. ## M5 · An index you can rebuild and a library you can move (production only) The index is disposable, the numbers to plan a re-index are measured, and the library never depends on this tool. ### Steps 1. Measure and print the indexing rate per run This is the number that tells you whether a terabyte is a night or a week. Files: `src/index/rate.py` 2. Keep frames and transcripts so re-embedding is cheap Extraction is the expensive half and it is model-independent. Files: `src/index/reembed.py` 3. Show offline volumes rather than hiding them Footage on a drive in a drawer is still footage. Files: `src/web/offline.py` ### Done when - [ ] Indexing rate is measured and stored per run - [ ] Re-embedding needs no re-extraction - [ ] Export and reimport preserve every count - [ ] Offline volumes appear labelled, not missing ### Watch out - Storing embeddings without keeping the extracted frames and transcripts means the first time you want a better model you re-extract every frame and re-transcribe every hour of audio, which is by far the most expensive part of the whole pipeline and the part that had nothing to do with the model. Keep frames, thumbnails and transcript text on disk, and make re-embedding a pass over stored artefacts.
OPERATIONS.md
# Operations · Shade ## Backup The thumbnails and transcripts first, the vectors second. The vectors can be recomputed from the stored artefacts in an evening; the extraction that produced those artefacts is the part that took the week. ## Restore Restore the index file and the thumbnail directory, then run the walker against the same roots and confirm no file is re-extracted. If files are being re-extracted, the volume identifiers did not survive the restore, which is the one thing worth checking before going to bed. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring For a local tool this lives in the runs table: video seconds per wall second, the share of frames dropped by deduplication and the share of segments below the confidence threshold. A sudden change in the dropped-frame share usually means the footage changed character, not that the code did. ## Incident checklist The failure that matters is a search that returns nothing useful for footage you know is there. Check three things in order: whether the file is indexed at all, whether the query was truncated, and whether the matching frames were deduplicated away. The queries table and the frame counts answer all three. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] No source file is confirmed modified, moved or renamed - [ ] A renamed folder is confirmed not to trigger re-embedding - [ ] A static clip is confirmed to produce a bounded frame count - [ ] An overlong query is confirmed to warn rather than truncate silently - [ ] Re-embedding is confirmed to need no re-extraction ## Launch constraint Do not market omitted Shade capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Vectors, metadata and run history in one file. Disposable but expensive to rebuild. INDEX_PATH=~/FootageIndex/index.db # Required. One small image per surviving frame. Keep these; they make re-embedding cheap. THUMB_DIR=~/FootageIndex/thumbs # Required. Baseline sampling. Scene cuts are added on top of this. FRAME_INTERVAL_SECONDS=5 # Required. The cap that stops a static shot from owning the index. MAX_FRAMES_PER_MINUTE=12 # Required. Frames closer than this to the previous one are dropped before embedding. PHASH_DISTANCE=6 # Required. Chunking aims here but always finishes the sentence. SEGMENT_TARGET_SECONDS=25 # Required. Segments below this are excluded from search by default and remain in the index. MIN_SEGMENT_CONFIDENCE=-1.0
$ choose a build depth, inspect the files, then open the complete pack in your agent
They pay because the search only matters if the whole team gets it. A local index that only lives on the editor's machine solves the editor's problem and nobody else's, and the person who most needs to find the clip is usually the one furthest from the storage. Shade sells the index plus the delivery of what the index found, and the second half is the expensive one.
xcloud streaming of full-res files without downloading them first
xguest links with per-link permissions and roles
xbuilt-in review, approval, and commenting
xface recognition and shot-type tagging that improves without your involvement
xteam sync, so everyone searches the same index
xthe NLE plugins and Slack integration
share on X ↗"I just replaced Shade ($35/mo) with one prompt"
Don't feel like building it? These folks already made it free.
ResourceSpaceFaces, sentence search and local transcripts in an old-school DAM; installation has not discovered joy.open source↗no votes, no pay-to-list · just what's real
Shade pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| growth | $35/user | $29.75/user | 1 workspace; up to 15 paid seats; 150 guests; 500 GB active storage/seat |
| enterprise | custom | — | Unlimited workspaces and seats; 250 guests; 1 TB active storage/seat; 1 TB bring-your-own S3 storage/seat |
free tierno free tier; trial length and numeric trial caps are not publicly disclosed
billingmonthly + annual (15% lower)
hidden costsGrowth is capped at one workspace and 15 paid seats; storage expansion and enterprise bring-your-own-storage workflows require a higher plan or add-on.
verified 2026-08-14 · source ↗
Vibecode Shade
Kinda. The core of Shade is buildable in a weekend with the prompt on this page, but there are real gaps: cloud streaming of full-res files without downloading them first, guest links with per-link permissions and roles. Read the honest list above before committing.
How much does Shade cost?
Shade costs about $35/month (Growth, checked 2026-07-30), which is $420 per year.
What do I lose by replacing Shade?
Honestly: cloud streaming of full-res files without downloading them first; guest links with per-link permissions and roles; built-in review, approval, and commenting; face recognition and shot-type tagging that improves without your involvement; team sync, so everyone searches the same index; the NLE plugins and Slack integration. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Shade?
Yes: ResourceSpace (Faces, sentence search and local transcripts in an old-school DAM; installation has not discovered joy.) The prompt is for when you want it exactly your way.