Vibecode TranscriptAPI
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The happy path is genuinely a one-sitting build: the open-source youtube-transcript-api library fetches a caption track in a few lines, and wrapping it in FastAPI gives you a working personal endpoint. The honest gap is reliability. YouTube rate-limits and IP-blocks caption scraping at any real volume, so the DIY version works until it suddenly does not, and there is no fix without a rotating proxy pool you must rent and operate. The paid product is not selling the parsing; it is selling the unblocked pipe, plus search and playlist endpoints on top.
the project pack5 files · written for this build, step by step
You are building a lean indie version of TranscriptAPI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # TranscriptAPI · indie build A personal caption store whose hard problems are that the only supported fetch needs edit permission on the video, that automatic caption segments are phrase fragments rather than sentences, and that the API bills quota units per call so a playlist loop can exhaust the day before lunch. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and uvicorn, one process, no worker fleet | the work is network waiting and parsing, and a personal load fits in one process | | Storage | SQLite beside the app with a full text index over the sentence layer | a caption store is small, append heavy and read mostly | | Sources | A source interface with the official captions API and a caption file importer behind it | these are the two things you are actually entitled to read, and the seam takes a licensed third | | Authorisation | OAuth for the official API, local keys if you serve anyone but yourself | captions.download needs edit permission on the video and there is no route around it | | Units | Segment times as integer milliseconds, never floating point seconds | floats drift, compare badly, and ruin alignment when you join tracks | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the service and the importer. - Get it: python.org or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A Google Cloud project with the YouTube Data API enabled and an OAuth client** · free within the default quota - Why: The only supported caption fetch runs through it, and the project is also where the daily quota lives. - Get it: Create a project, enable the API, configure the consent screen and create an OAuth client for a desktop application. - Verify: A consent flow completes and captions.list returns tracks for a video you own - [ ] **At least one video you can edit, with a caption track** · free - Why: Without it you cannot exercise the official source at all, and the whole authorisation path stays untested. - Get it: Upload an unlisted test video to a channel you control and add captions to it. - Verify: The official source returns its track and segments - [ ] **A set of caption files in SRT, WebVTT and TTML** · free - Why: These are the importer's real inputs and the sentence heuristic needs an automatic-caption fixture with no punctuation to be tested honestly. - Get it: Export captions from a video you control in each format, and keep one deliberately unpunctuated. - Verify: All three import to identical row shapes - [ ] **Your written source policy** · free - Why: Deciding what the service may fetch after you have built the fetcher means deciding it from whatever already works. - Get it: Write down which sources are enabled, what each one requires, and that no evasion layer will be added. - Verify: The policy endpoint and the README carry the same text ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A proxy pool or any other blocking evasion. - Channel, playlist and search over videos you do not control. - Speech recognition over downloaded audio. - Any availability promise. - staying unblocked when YouTube rate-limits caption fetches - search, channel and playlist endpoints - reliability at bulk volume - an SLA and support when YouTube changes something If one of those is essential to you, that is the reason to keep paying for TranscriptAPI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · TranscriptAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal transcript service that stores, searches and serves caption tracks from sources you are entitled to read to replace TranscriptAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Read this before you install a library, because it decides what the service is allowed to do. The official route is closed for other people's videos. The YouTube Data API captions.download method states plainly that it 'requires the user to have permission to edit the video', and captions.list returns track metadata only, under the same authorisation. Checked against Google's reference on 15 September 2026. So there is no supported way to pull the caption track of a video you do not control. Every three-line library that appears to do it anyway reads an endpoint YouTube does not publish. The YouTube API Services Developer Policies, section III.E, say you 'must not, and must not encourage, enable, or require others to, directly or indirectly, scrape YouTube Applications', and the main Terms prohibit accessing the Service by automated means. That is the whole moat of the paid product: a rented proxy fleet that keeps the scrape working after YouTube blocks it. This build does not do that. Renting residential addresses to keep reading an endpoint after the platform has cut you off is evasion, and no amount of retry logic makes it engineering. So build the honest version: a source interface with the official API behind it for videos you control, a caption file importer for everything else, and a documented seam for a provider you have a licence with. The cache, the sentence layer, the search and the serving are all genuinely yours, and they are most of the code. What you do not get is other people's videos on demand. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet · the work is network waiting and parsing, and a personal load fits in one process - Storage: SQLite beside the app with a full text index over the sentence layer · a caption store is small, append heavy and read mostly - Sources: A source interface with the official captions API and a caption file importer behind it · these are the two things you are actually entitled to read, and the seam takes a licensed third - Authorisation: OAuth for the official API, local keys if you serve anyone but yourself · captions.download needs edit permission on the video and there is no route around it - Units: Segment times as integer milliseconds, never floating point seconds · floats drift, compare badly, and ruin alignment when you join tracks ### Data model (create this before Phase 1) - videos(id, title, channel_id, duration_ms, first_seen_at, added_by_source) - tracks(id, video_id, language, kind, is_default, source, fetched_at, revision) · kind is manual or automatic, and one language can have both - segments(track_id, ordinal, start_ms, duration_ms, text) · the fetched truth, integers only, never rewritten - sentences(track_id, ordinal, start_ms, end_ms, text, first_segment, last_segment) · derived, with offsets back into segments so a search hit maps to a timestamp - fetches(id, track_id, at, source, outcome, quota_units, http_status, retry_after_s) · every attempt including the refusals, because the refusals are the interesting ones - quota(day, units_used, units_limit) · counted locally before the call rather than discovered by the failure - clients(id, label, created_at, daily_request_limit, revoked_at) · only if this ever serves someone other than you - There is no column anywhere holding 'the transcript'. A video has tracks, a track has segments, and every response names the track it came from. ### Phase 1 · A source interface, and the two sources you actually have Build: Define a source protocol returning a normalised track plus segments. Implement two: the official captions API using OAuth for videos you can edit, and an importer for SRT, WebVTT and TTML files. Record which source produced every track. Leave a third implementation stubbed and documented. Trap: Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. Done when: A track imported from a WebVTT file and a track fetched through the official API produce identical row shapes, asserted in tests. No module outside the source package makes an outbound request, checked by review. Every track row records its source and the time it was fetched. Importing the same file twice creates one track, not two. Do not build yet: No sentence layer, search, caching policy or serving to other clients. ### Phase 2 · Segments are the truth, sentences are derived Build: Store segments exactly as fetched. Build a sentence layer over them with a punctuation and gap heuristic, keeping the first and last segment ordinal on every sentence so a hit resolves to a start time. Index the sentences for full text search. Mark whether a track is automatic. Trap: Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. Done when: A search for a phrase that spans two caption fragments returns a hit with a start time inside a second of the spoken phrase, tested against a fixture track. Rebuilding the sentence layer from segments produces byte-identical rows, tested. Segment rows are never updated after insert, enforced by a trigger or asserted in tests. Automatic tracks are flagged and the flag appears in every response. Do not build yet: No quota accounting, caching policy, error taxonomy or serving. ### Phase 3 · Quota you count, and errors that name the reason Build: Count quota units locally before each call and refuse when the day's budget is spent. Map every failure to a named outcome: captions disabled, language not present, not authorised for this video, upstream refused, rate limited with a retry hint. Store the outcome on the fetch row. Trap: The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. Done when: Exhausting the local unit budget returns a distinct status with the time the budget resets, tested with a stubbed counter. Every documented failure maps to its own outcome value and none of them produces a five hundred, tested against a fake source. A rate limited response records the retry hint and the client is told it. The fetch table contains a row for every attempt, successful or not, checked after a test run. Do not build yet: No caching policy, deep links or multi-client serving. ### Phase 4 · A cache keyed on what actually varies Build: Cache tracks keyed on video, language and kind together, with a configurable age after which a track is refetched. Serve from cache by default and refetch only on an explicit parameter. Emit deep links at whole seconds alongside the millisecond values. Trap: Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. Done when: Requesting two languages for one video stores two tracks and returns the right one for each, tested. A repeat request inside the freshness window makes no outbound call, verified by counting fetch rows. The refresh parameter forces a new fetch and creates a new revision rather than overwriting the old segments. Every response names the video, language, kind and fetch time of the track it served. Do not build yet: No client keys, budgets or export yet. ### Phase 5 · Serving it to something other than yourself Build: Add client keys with per-key daily limits and a revoke path. Export every table as JSONL, and import the export into an empty database. Publish the source policy at an endpoint so a caller can read what this service is and is not allowed to fetch. Log slow and failed requests. Trap: A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal. Done when: A key that exceeds its own limit is refused while other keys keep working, tested with two keys. Revoking a key takes effect on the next request with no restart, tested. The export reimports into an empty database and the search index rebuilds with identical results, tested. The source policy endpoint states which sources are enabled and what each requires, and the README says the same thing. Do not build yet: Nothing after this. ### Out of scope (and why) - A rotating proxy pool, or any other means of continuing to read an endpoint after YouTube has blocked you. That is the paid product's actual business and it is evasion, not engineering. - Channel, playlist and search endpoints over videos you do not control, which need the same unsupported fetch at higher volume. - Running speech recognition over downloaded audio, which is a separate licence question about the audio as well as a much larger build. - An availability promise. This service is as reliable as a source it does not operate, and saying otherwise in a README is how you end up supporting it. ### README must contain - That captions.download requires permission to edit the video, quoted from Google's reference with the date checked - That the Developer Policies prohibit scraping YouTube applications, and that this build therefore has no fetcher for videos you do not control - The two working sources, what each needs, and how to plug in a provider you hold a licence with - That quota is counted locally, what the per-call unit costs were when this was written, and where to check the current daily limit - That segments are stored untouched and sentences are derived, so a search hit resolves to a real timestamp ===== AGENTS.md ===== # Agent instructions · TranscriptAPI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet, SQLite beside the app with a full text index over the sentence layer, A source interface with the official captions API and a caption file importer behind it, OAuth for the official API, local keys if you serve anyone but yourself, Segment times as integer milliseconds, never floating point seconds. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. ===== BUILD_PLAN.md ===== # Build plan · TranscriptAPI A personal caption store whose hard problems are that the only supported fetch needs edit permission on the video, that automatic caption segments are phrase fragments rather than sentences, and that the API bills quota units per call so a playlist loop can exhaust the day before lunch. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A source interface, and the two sources you actually have Captions enter the store through one interface, with the official API and a file importer behind it. ### Steps 1. Write the source protocol before either implementation Two implementations written against one interface is the point of the exercise. Files: `src/sources/base.py`, `src/models.py` 2. Implement the file importer first It has no credentials and no quota, so the schema gets settled cheaply. Files: `src/sources/captionfile.py`, `tests/test_import.py` 3. Add the official API source behind OAuth It only works on videos you can edit, and the code should say so in its docstring. Files: `src/sources/youtube_api.py`, `docs/authorisation.md` ### Done when - [ ] Both sources produce identical row shapes - [ ] No outbound request exists outside the source package - [ ] Every track records its source and fetch time - [ ] Re-importing a file does not duplicate a track ### Watch out - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. ## Phase 2 · Segments are the truth, sentences are derived Automatic caption fragments become searchable sentences without losing the timing. ### Steps 1. Never modify a segment after insert It is the only record of what the source actually said. Files: `src/store/segments.py`, `tests/test_immutable.py` 2. Derive sentences with a gap and punctuation rule And keep the segment range so a hit is a timestamp, not a video. Files: `src/sentences.py`, `tests/test_sentences.py` 3. Index sentences, not segments Fragment-level indexing cannot match a phrase a human would search for. Files: `src/search.py`, `migrations/002_fts.sql` ### Done when - [ ] A cross-fragment phrase search returns a correct start time - [ ] The sentence layer rebuilds identically from segments - [ ] Segments are never updated after insert - [ ] Automatic tracks are flagged in every response ### Watch out - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. ## Phase 3 · Quota you count, and errors that name the reason A refusal is a distinct, documented outcome and never a bare five hundred. ### Steps 1. Count units before the call, not after the failure The failure arrives too late to be a budget. Files: `src/quota.py`, `tests/test_quota.py` 2. Give every failure its own named outcome 'Upstream error' is not a thing a caller can act on. Files: `src/errors.py`, `src/api/routes.py` 3. Record every attempt, including the refusals The refusals are the ones you will want to read later. Files: `src/store/fetches.py` ### Done when - [ ] A spent budget returns a distinct status with a reset time - [ ] No failure path produces a bare five hundred - [ ] Rate limit hints are stored and passed to the caller - [ ] Every attempt has a fetch row ### Watch out - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. ## Phase 4 · A cache keyed on what actually varies A repeat request costs nothing and a different language does not collide with the first one. ### Steps 1. Key on video, language and kind together Any one of the three is enough to make the answer wrong. Files: `src/cache.py`, `tests/test_cache_key.py` 2. Refresh creates a revision, never an overwrite Old segments are how you explain an old answer. Files: `src/store/tracks.py` 3. Put track identity in every response body So a caller can see it got something other than what it asked for. Files: `src/api/schemas.py` ### Done when - [ ] Two languages for one video do not collide - [ ] A cached hit makes no outbound call - [ ] Refresh creates a revision rather than overwriting - [ ] Every response names the track it served ### Watch out - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. ## Not in this build - A proxy pool or any other blocking evasion. - Channel, playlist and search over videos you do not control. - Speech recognition over downloaded audio. - Any availability promise. ## After v1, if you want it - A diff view between two revisions of the same track, which is how you notice a caption edit - Speaker labels imported from formats that carry them, kept out of the derived sentence text - A small client library that fails loudly on the named outcomes rather than retrying them ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Videos, tracks, segments, sentences and the fetch log. Back this up; refetching is not always possible. DATABASE_PATH=./data/transcripts.db # Optional. The OAuth client for the official captions source. Omit and only the file importer runs. GOOGLE_CLIENT_ID=1234-abcd.apps.googleusercontent.com # Optional · secret. Paired with the client id. Never in the repository. GOOGLE_CLIENT_SECRET=GOCSPX-...redacted # Optional · secret. Stored refresh token for the account that owns the videos. GOOGLE_TOKEN_PATH=./data/oauth-token.json # Required. Your project's published daily limit. Check it in the console rather than trusting this default. DAILY_QUOTA_UNITS=10000 # Required. How stale a cached track may be before a refresh is allowed. TRACK_MAX_AGE_DAYS=30 # Required. Turn on when something other than you starts calling this. CLIENT_KEYS_ENABLED=false
You are building a lean indie version of TranscriptAPI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # TranscriptAPI · indie build A personal caption store whose hard problems are that the only supported fetch needs edit permission on the video, that automatic caption segments are phrase fragments rather than sentences, and that the API bills quota units per call so a playlist loop can exhaust the day before lunch. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and uvicorn, one process, no worker fleet | the work is network waiting and parsing, and a personal load fits in one process | | Storage | SQLite beside the app with a full text index over the sentence layer | a caption store is small, append heavy and read mostly | | Sources | A source interface with the official captions API and a caption file importer behind it | these are the two things you are actually entitled to read, and the seam takes a licensed third | | Authorisation | OAuth for the official API, local keys if you serve anyone but yourself | captions.download needs edit permission on the video and there is no route around it | | Units | Segment times as integer milliseconds, never floating point seconds | floats drift, compare badly, and ruin alignment when you join tracks | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the service and the importer. - Get it: python.org or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A Google Cloud project with the YouTube Data API enabled and an OAuth client** · free within the default quota - Why: The only supported caption fetch runs through it, and the project is also where the daily quota lives. - Get it: Create a project, enable the API, configure the consent screen and create an OAuth client for a desktop application. - Verify: A consent flow completes and captions.list returns tracks for a video you own - [ ] **At least one video you can edit, with a caption track** · free - Why: Without it you cannot exercise the official source at all, and the whole authorisation path stays untested. - Get it: Upload an unlisted test video to a channel you control and add captions to it. - Verify: The official source returns its track and segments - [ ] **A set of caption files in SRT, WebVTT and TTML** · free - Why: These are the importer's real inputs and the sentence heuristic needs an automatic-caption fixture with no punctuation to be tested honestly. - Get it: Export captions from a video you control in each format, and keep one deliberately unpunctuated. - Verify: All three import to identical row shapes - [ ] **Your written source policy** · free - Why: Deciding what the service may fetch after you have built the fetcher means deciding it from whatever already works. - Get it: Write down which sources are enabled, what each one requires, and that no evasion layer will be added. - Verify: The policy endpoint and the README carry the same text ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A proxy pool or any other blocking evasion. - Channel, playlist and search over videos you do not control. - Speech recognition over downloaded audio. - Any availability promise. - staying unblocked when YouTube rate-limits caption fetches - search, channel and playlist endpoints - reliability at bulk volume - an SLA and support when YouTube changes something If one of those is essential to you, that is the reason to keep paying for TranscriptAPI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · TranscriptAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal transcript service that stores, searches and serves caption tracks from sources you are entitled to read to replace TranscriptAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Read this before you install a library, because it decides what the service is allowed to do. The official route is closed for other people's videos. The YouTube Data API captions.download method states plainly that it 'requires the user to have permission to edit the video', and captions.list returns track metadata only, under the same authorisation. Checked against Google's reference on 15 September 2026. So there is no supported way to pull the caption track of a video you do not control. Every three-line library that appears to do it anyway reads an endpoint YouTube does not publish. The YouTube API Services Developer Policies, section III.E, say you 'must not, and must not encourage, enable, or require others to, directly or indirectly, scrape YouTube Applications', and the main Terms prohibit accessing the Service by automated means. That is the whole moat of the paid product: a rented proxy fleet that keeps the scrape working after YouTube blocks it. This build does not do that. Renting residential addresses to keep reading an endpoint after the platform has cut you off is evasion, and no amount of retry logic makes it engineering. So build the honest version: a source interface with the official API behind it for videos you control, a caption file importer for everything else, and a documented seam for a provider you have a licence with. The cache, the sentence layer, the search and the serving are all genuinely yours, and they are most of the code. What you do not get is other people's videos on demand. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet · the work is network waiting and parsing, and a personal load fits in one process - Storage: SQLite beside the app with a full text index over the sentence layer · a caption store is small, append heavy and read mostly - Sources: A source interface with the official captions API and a caption file importer behind it · these are the two things you are actually entitled to read, and the seam takes a licensed third - Authorisation: OAuth for the official API, local keys if you serve anyone but yourself · captions.download needs edit permission on the video and there is no route around it - Units: Segment times as integer milliseconds, never floating point seconds · floats drift, compare badly, and ruin alignment when you join tracks ### Data model (create this before Phase 1) - videos(id, title, channel_id, duration_ms, first_seen_at, added_by_source) - tracks(id, video_id, language, kind, is_default, source, fetched_at, revision) · kind is manual or automatic, and one language can have both - segments(track_id, ordinal, start_ms, duration_ms, text) · the fetched truth, integers only, never rewritten - sentences(track_id, ordinal, start_ms, end_ms, text, first_segment, last_segment) · derived, with offsets back into segments so a search hit maps to a timestamp - fetches(id, track_id, at, source, outcome, quota_units, http_status, retry_after_s) · every attempt including the refusals, because the refusals are the interesting ones - quota(day, units_used, units_limit) · counted locally before the call rather than discovered by the failure - clients(id, label, created_at, daily_request_limit, revoked_at) · only if this ever serves someone other than you - There is no column anywhere holding 'the transcript'. A video has tracks, a track has segments, and every response names the track it came from. ### Phase 1 · A source interface, and the two sources you actually have Build: Define a source protocol returning a normalised track plus segments. Implement two: the official captions API using OAuth for videos you can edit, and an importer for SRT, WebVTT and TTML files. Record which source produced every track. Leave a third implementation stubbed and documented. Trap: Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. Done when: A track imported from a WebVTT file and a track fetched through the official API produce identical row shapes, asserted in tests. No module outside the source package makes an outbound request, checked by review. Every track row records its source and the time it was fetched. Importing the same file twice creates one track, not two. Do not build yet: No sentence layer, search, caching policy or serving to other clients. ### Phase 2 · Segments are the truth, sentences are derived Build: Store segments exactly as fetched. Build a sentence layer over them with a punctuation and gap heuristic, keeping the first and last segment ordinal on every sentence so a hit resolves to a start time. Index the sentences for full text search. Mark whether a track is automatic. Trap: Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. Done when: A search for a phrase that spans two caption fragments returns a hit with a start time inside a second of the spoken phrase, tested against a fixture track. Rebuilding the sentence layer from segments produces byte-identical rows, tested. Segment rows are never updated after insert, enforced by a trigger or asserted in tests. Automatic tracks are flagged and the flag appears in every response. Do not build yet: No quota accounting, caching policy, error taxonomy or serving. ### Phase 3 · Quota you count, and errors that name the reason Build: Count quota units locally before each call and refuse when the day's budget is spent. Map every failure to a named outcome: captions disabled, language not present, not authorised for this video, upstream refused, rate limited with a retry hint. Store the outcome on the fetch row. Trap: The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. Done when: Exhausting the local unit budget returns a distinct status with the time the budget resets, tested with a stubbed counter. Every documented failure maps to its own outcome value and none of them produces a five hundred, tested against a fake source. A rate limited response records the retry hint and the client is told it. The fetch table contains a row for every attempt, successful or not, checked after a test run. Do not build yet: No caching policy, deep links or multi-client serving. ### Phase 4 · A cache keyed on what actually varies Build: Cache tracks keyed on video, language and kind together, with a configurable age after which a track is refetched. Serve from cache by default and refetch only on an explicit parameter. Emit deep links at whole seconds alongside the millisecond values. Trap: Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. Done when: Requesting two languages for one video stores two tracks and returns the right one for each, tested. A repeat request inside the freshness window makes no outbound call, verified by counting fetch rows. The refresh parameter forces a new fetch and creates a new revision rather than overwriting the old segments. Every response names the video, language, kind and fetch time of the track it served. Do not build yet: No client keys, budgets or export yet. ### Phase 5 · Serving it to something other than yourself Build: Add client keys with per-key daily limits and a revoke path. Export every table as JSONL, and import the export into an empty database. Publish the source policy at an endpoint so a caller can read what this service is and is not allowed to fetch. Log slow and failed requests. Trap: A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal. Done when: A key that exceeds its own limit is refused while other keys keep working, tested with two keys. Revoking a key takes effect on the next request with no restart, tested. The export reimports into an empty database and the search index rebuilds with identical results, tested. The source policy endpoint states which sources are enabled and what each requires, and the README says the same thing. Do not build yet: Nothing after this. ### Out of scope (and why) - A rotating proxy pool, or any other means of continuing to read an endpoint after YouTube has blocked you. That is the paid product's actual business and it is evasion, not engineering. - Channel, playlist and search endpoints over videos you do not control, which need the same unsupported fetch at higher volume. - Running speech recognition over downloaded audio, which is a separate licence question about the audio as well as a much larger build. - An availability promise. This service is as reliable as a source it does not operate, and saying otherwise in a README is how you end up supporting it. ### README must contain - That captions.download requires permission to edit the video, quoted from Google's reference with the date checked - That the Developer Policies prohibit scraping YouTube applications, and that this build therefore has no fetcher for videos you do not control - The two working sources, what each needs, and how to plug in a provider you hold a licence with - That quota is counted locally, what the per-call unit costs were when this was written, and where to check the current daily limit - That segments are stored untouched and sentences are derived, so a search hit resolves to a real timestamp ===== AGENTS.md ===== # Agent instructions · TranscriptAPI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet, SQLite beside the app with a full text index over the sentence layer, A source interface with the official captions API and a caption file importer behind it, OAuth for the official API, local keys if you serve anyone but yourself, Segment times as integer milliseconds, never floating point seconds. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. ===== BUILD_PLAN.md ===== # Build plan · TranscriptAPI A personal caption store whose hard problems are that the only supported fetch needs edit permission on the video, that automatic caption segments are phrase fragments rather than sentences, and that the API bills quota units per call so a playlist loop can exhaust the day before lunch. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A source interface, and the two sources you actually have Captions enter the store through one interface, with the official API and a file importer behind it. ### Steps 1. Write the source protocol before either implementation Two implementations written against one interface is the point of the exercise. Files: `src/sources/base.py`, `src/models.py` 2. Implement the file importer first It has no credentials and no quota, so the schema gets settled cheaply. Files: `src/sources/captionfile.py`, `tests/test_import.py` 3. Add the official API source behind OAuth It only works on videos you can edit, and the code should say so in its docstring. Files: `src/sources/youtube_api.py`, `docs/authorisation.md` ### Done when - [ ] Both sources produce identical row shapes - [ ] No outbound request exists outside the source package - [ ] Every track records its source and fetch time - [ ] Re-importing a file does not duplicate a track ### Watch out - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. ## Phase 2 · Segments are the truth, sentences are derived Automatic caption fragments become searchable sentences without losing the timing. ### Steps 1. Never modify a segment after insert It is the only record of what the source actually said. Files: `src/store/segments.py`, `tests/test_immutable.py` 2. Derive sentences with a gap and punctuation rule And keep the segment range so a hit is a timestamp, not a video. Files: `src/sentences.py`, `tests/test_sentences.py` 3. Index sentences, not segments Fragment-level indexing cannot match a phrase a human would search for. Files: `src/search.py`, `migrations/002_fts.sql` ### Done when - [ ] A cross-fragment phrase search returns a correct start time - [ ] The sentence layer rebuilds identically from segments - [ ] Segments are never updated after insert - [ ] Automatic tracks are flagged in every response ### Watch out - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. ## Phase 3 · Quota you count, and errors that name the reason A refusal is a distinct, documented outcome and never a bare five hundred. ### Steps 1. Count units before the call, not after the failure The failure arrives too late to be a budget. Files: `src/quota.py`, `tests/test_quota.py` 2. Give every failure its own named outcome 'Upstream error' is not a thing a caller can act on. Files: `src/errors.py`, `src/api/routes.py` 3. Record every attempt, including the refusals The refusals are the ones you will want to read later. Files: `src/store/fetches.py` ### Done when - [ ] A spent budget returns a distinct status with a reset time - [ ] No failure path produces a bare five hundred - [ ] Rate limit hints are stored and passed to the caller - [ ] Every attempt has a fetch row ### Watch out - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. ## Phase 4 · A cache keyed on what actually varies A repeat request costs nothing and a different language does not collide with the first one. ### Steps 1. Key on video, language and kind together Any one of the three is enough to make the answer wrong. Files: `src/cache.py`, `tests/test_cache_key.py` 2. Refresh creates a revision, never an overwrite Old segments are how you explain an old answer. Files: `src/store/tracks.py` 3. Put track identity in every response body So a caller can see it got something other than what it asked for. Files: `src/api/schemas.py` ### Done when - [ ] Two languages for one video do not collide - [ ] A cached hit makes no outbound call - [ ] Refresh creates a revision rather than overwriting - [ ] Every response names the track it served ### Watch out - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. ## Not in this build - A proxy pool or any other blocking evasion. - Channel, playlist and search over videos you do not control. - Speech recognition over downloaded audio. - Any availability promise. ## After v1, if you want it - A diff view between two revisions of the same track, which is how you notice a caption edit - Speaker labels imported from formats that carry them, kept out of the derived sentence text - A small client library that fails loudly on the named outcomes rather than retrying them ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Videos, tracks, segments, sentences and the fetch log. Back this up; refetching is not always possible. DATABASE_PATH=./data/transcripts.db # Optional. The OAuth client for the official captions source. Omit and only the file importer runs. GOOGLE_CLIENT_ID=1234-abcd.apps.googleusercontent.com # Optional · secret. Paired with the client id. Never in the repository. GOOGLE_CLIENT_SECRET=GOCSPX-...redacted # Optional · secret. Stored refresh token for the account that owns the videos. GOOGLE_TOKEN_PATH=./data/oauth-token.json # Required. Your project's published daily limit. Check it in the console rather than trusting this default. DAILY_QUOTA_UNITS=10000 # Required. How stale a cached track may be before a refresh is allowed. TRACK_MAX_AGE_DAYS=30 # Required. Turn on when something other than you starts calling this. CLIENT_KEYS_ENABLED=false
You are building a production product version of TranscriptAPI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # TranscriptAPI · product brief ## Problem The happy path is genuinely a one-sitting build: the open-source youtube-transcript-api library fetches a caption track in a few lines, and wrapping it in FastAPI gives you a working personal endpoint. The honest gap is reliability. YouTube rate-limits and IP-blocks caption scraping at any real volume, so the DIY version works until it suddenly does not, and there is no fix without a rotating proxy pool you must rent and operate. The paid product is not selling the parsing; it is selling the unblocked pipe, plus search and playlist endpoints on top. ## Product outcome A caption store that is honest about what it may fetch, keeps the fetched segments untouched, resolves a phrase search to a timestamp, counts its quota before spending it, and exports everything including the record of every refusal. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Python 3.11 - a rotating proxy pool if you use it at any volume ## Explicit non-goals for v1 - A proxy pool or any other blocking evasion. - Channel, playlist and search over videos you do not control. - Speech recognition over downloaded audio. - Any availability promise. - staying unblocked when YouTube rate-limits caption fetches - search, channel and playlist endpoints - reliability at bulk volume - an SLA and support when YouTube changes something ## Success criteria - No fetcher exists for videos the authorised account cannot edit - Segments are confirmed immutable after insert - A cross-fragment phrase search is confirmed to return a correct timestamp - The quota counter is confirmed to refuse before the call, not after - Export and reimport are confirmed to round-trip with a rebuilt index ===== BRIEF.md ===== # Build brief · TranscriptAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal transcript service that stores, searches and serves caption tracks from sources you are entitled to read to replace TranscriptAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Read this before you install a library, because it decides what the service is allowed to do. The official route is closed for other people's videos. The YouTube Data API captions.download method states plainly that it 'requires the user to have permission to edit the video', and captions.list returns track metadata only, under the same authorisation. Checked against Google's reference on 15 September 2026. So there is no supported way to pull the caption track of a video you do not control. Every three-line library that appears to do it anyway reads an endpoint YouTube does not publish. The YouTube API Services Developer Policies, section III.E, say you 'must not, and must not encourage, enable, or require others to, directly or indirectly, scrape YouTube Applications', and the main Terms prohibit accessing the Service by automated means. That is the whole moat of the paid product: a rented proxy fleet that keeps the scrape working after YouTube blocks it. This build does not do that. Renting residential addresses to keep reading an endpoint after the platform has cut you off is evasion, and no amount of retry logic makes it engineering. So build the honest version: a source interface with the official API behind it for videos you control, a caption file importer for everything else, and a documented seam for a provider you have a licence with. The cache, the sentence layer, the search and the serving are all genuinely yours, and they are most of the code. What you do not get is other people's videos on demand. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet · the work is network waiting and parsing, and a personal load fits in one process - Storage: SQLite beside the app with a full text index over the sentence layer · a caption store is small, append heavy and read mostly - Sources: A source interface with the official captions API and a caption file importer behind it · these are the two things you are actually entitled to read, and the seam takes a licensed third - Authorisation: OAuth for the official API, local keys if you serve anyone but yourself · captions.download needs edit permission on the video and there is no route around it - Units: Segment times as integer milliseconds, never floating point seconds · floats drift, compare badly, and ruin alignment when you join tracks ### Data model (create this before Phase 1) - videos(id, title, channel_id, duration_ms, first_seen_at, added_by_source) - tracks(id, video_id, language, kind, is_default, source, fetched_at, revision) · kind is manual or automatic, and one language can have both - segments(track_id, ordinal, start_ms, duration_ms, text) · the fetched truth, integers only, never rewritten - sentences(track_id, ordinal, start_ms, end_ms, text, first_segment, last_segment) · derived, with offsets back into segments so a search hit maps to a timestamp - fetches(id, track_id, at, source, outcome, quota_units, http_status, retry_after_s) · every attempt including the refusals, because the refusals are the interesting ones - quota(day, units_used, units_limit) · counted locally before the call rather than discovered by the failure - clients(id, label, created_at, daily_request_limit, revoked_at) · only if this ever serves someone other than you - There is no column anywhere holding 'the transcript'. A video has tracks, a track has segments, and every response names the track it came from. ### Phase 1 · A source interface, and the two sources you actually have Build: Define a source protocol returning a normalised track plus segments. Implement two: the official captions API using OAuth for videos you can edit, and an importer for SRT, WebVTT and TTML files. Record which source produced every track. Leave a third implementation stubbed and documented. Trap: Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. Done when: A track imported from a WebVTT file and a track fetched through the official API produce identical row shapes, asserted in tests. No module outside the source package makes an outbound request, checked by review. Every track row records its source and the time it was fetched. Importing the same file twice creates one track, not two. Do not build yet: No sentence layer, search, caching policy or serving to other clients. ### Phase 2 · Segments are the truth, sentences are derived Build: Store segments exactly as fetched. Build a sentence layer over them with a punctuation and gap heuristic, keeping the first and last segment ordinal on every sentence so a hit resolves to a start time. Index the sentences for full text search. Mark whether a track is automatic. Trap: Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. Done when: A search for a phrase that spans two caption fragments returns a hit with a start time inside a second of the spoken phrase, tested against a fixture track. Rebuilding the sentence layer from segments produces byte-identical rows, tested. Segment rows are never updated after insert, enforced by a trigger or asserted in tests. Automatic tracks are flagged and the flag appears in every response. Do not build yet: No quota accounting, caching policy, error taxonomy or serving. ### Phase 3 · Quota you count, and errors that name the reason Build: Count quota units locally before each call and refuse when the day's budget is spent. Map every failure to a named outcome: captions disabled, language not present, not authorised for this video, upstream refused, rate limited with a retry hint. Store the outcome on the fetch row. Trap: The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. Done when: Exhausting the local unit budget returns a distinct status with the time the budget resets, tested with a stubbed counter. Every documented failure maps to its own outcome value and none of them produces a five hundred, tested against a fake source. A rate limited response records the retry hint and the client is told it. The fetch table contains a row for every attempt, successful or not, checked after a test run. Do not build yet: No caching policy, deep links or multi-client serving. ### Phase 4 · A cache keyed on what actually varies Build: Cache tracks keyed on video, language and kind together, with a configurable age after which a track is refetched. Serve from cache by default and refetch only on an explicit parameter. Emit deep links at whole seconds alongside the millisecond values. Trap: Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. Done when: Requesting two languages for one video stores two tracks and returns the right one for each, tested. A repeat request inside the freshness window makes no outbound call, verified by counting fetch rows. The refresh parameter forces a new fetch and creates a new revision rather than overwriting the old segments. Every response names the video, language, kind and fetch time of the track it served. Do not build yet: No client keys, budgets or export yet. ### Phase 5 · Serving it to something other than yourself Build: Add client keys with per-key daily limits and a revoke path. Export every table as JSONL, and import the export into an empty database. Publish the source policy at an endpoint so a caller can read what this service is and is not allowed to fetch. Log slow and failed requests. Trap: A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal. Done when: A key that exceeds its own limit is refused while other keys keep working, tested with two keys. Revoking a key takes effect on the next request with no restart, tested. The export reimports into an empty database and the search index rebuilds with identical results, tested. The source policy endpoint states which sources are enabled and what each requires, and the README says the same thing. Do not build yet: Nothing after this. ### Out of scope (and why) - A rotating proxy pool, or any other means of continuing to read an endpoint after YouTube has blocked you. That is the paid product's actual business and it is evasion, not engineering. - Channel, playlist and search endpoints over videos you do not control, which need the same unsupported fetch at higher volume. - Running speech recognition over downloaded audio, which is a separate licence question about the audio as well as a much larger build. - An availability promise. This service is as reliable as a source it does not operate, and saying otherwise in a README is how you end up supporting it. ### README must contain - That captions.download requires permission to edit the video, quoted from Google's reference with the date checked - That the Developer Policies prohibit scraping YouTube applications, and that this build therefore has no fetcher for videos you do not control - The two working sources, what each needs, and how to plug in a provider you hold a licence with - That quota is counted locally, what the per-call unit costs were when this was written, and where to check the current daily limit - That segments are stored untouched and sentences are derived, so a search hit resolves to a real timestamp ===== ARCHITECTURE.md ===== # Architecture · TranscriptAPI ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and uvicorn, one process, no worker fleet | the work is network waiting and parsing, and a personal load fits in one process | | Storage | SQLite beside the app with a full text index over the sentence layer | a caption store is small, append heavy and read mostly | | Sources | A source interface with the official captions API and a caption file importer behind it | these are the two things you are actually entitled to read, and the seam takes a licensed third | | Authorisation | OAuth for the official API, local keys if you serve anyone but yourself | captions.download needs edit permission on the video and there is no route around it | | Units | Segment times as integer milliseconds, never floating point seconds | floats drift, compare badly, and ruin alignment when you join tracks | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Sources | The official API client and the file importer. | Any source you hold a licence for. Never a scraper, whatever the retry logic looks like. | | Store | Videos, tracks, immutable segments, revisions. | Any database, provided segments stay immutable and revisions accumulate. | | Sentences | The derived layer and the search index. | Any heuristic, provided the segment range survives on every sentence. | | Quota | Local unit counting and per-key limits. | Any counter, provided it decides before the call rather than after. | | Serving | Routes, client keys, export and the policy endpoint. | Any framework. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Videos, tracks, segments, sentences and the fetch log. Back this up; refetching is not always possible. - `GOOGLE_CLIENT_ID` · optional · The OAuth client for the official captions source. Omit and only the file importer runs. - `GOOGLE_CLIENT_SECRET` · optional, secret · Paired with the client id. Never in the repository. - `GOOGLE_TOKEN_PATH` · optional, secret · Stored refresh token for the account that owns the videos. - `DAILY_QUOTA_UNITS` · required · Your project's published daily limit. Check it in the console rather than trusting this default. - `TRACK_MAX_AGE_DAYS` · required · How stale a cached track may be before a refresh is allowed. - `CLIENT_KEYS_ENABLED` · required · Turn on when something other than you starts calling this. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · TranscriptAPI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet, SQLite beside the app with a full text index over the sentence layer, A source interface with the official captions API and a caption file importer behind it, OAuth for the official API, local keys if you serve anyone but yourself, Segment times as integer milliseconds, never floating point seconds. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. - A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal. ===== MILESTONES.md ===== # Delivery milestones · TranscriptAPI Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A source interface, and the two sources you actually have Captions enter the store through one interface, with the official API and a file importer behind it. ### Steps 1. Write the source protocol before either implementation Two implementations written against one interface is the point of the exercise. Files: `src/sources/base.py`, `src/models.py` 2. Implement the file importer first It has no credentials and no quota, so the schema gets settled cheaply. Files: `src/sources/captionfile.py`, `tests/test_import.py` 3. Add the official API source behind OAuth It only works on videos you can edit, and the code should say so in its docstring. Files: `src/sources/youtube_api.py`, `docs/authorisation.md` ### Done when - [ ] Both sources produce identical row shapes - [ ] No outbound request exists outside the source package - [ ] Every track records its source and fetch time - [ ] Re-importing a file does not duplicate a track ### Watch out - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. ## M2 · Segments are the truth, sentences are derived Automatic caption fragments become searchable sentences without losing the timing. ### Steps 1. Never modify a segment after insert It is the only record of what the source actually said. Files: `src/store/segments.py`, `tests/test_immutable.py` 2. Derive sentences with a gap and punctuation rule And keep the segment range so a hit is a timestamp, not a video. Files: `src/sentences.py`, `tests/test_sentences.py` 3. Index sentences, not segments Fragment-level indexing cannot match a phrase a human would search for. Files: `src/search.py`, `migrations/002_fts.sql` ### Done when - [ ] A cross-fragment phrase search returns a correct start time - [ ] The sentence layer rebuilds identically from segments - [ ] Segments are never updated after insert - [ ] Automatic tracks are flagged in every response ### Watch out - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. ## M3 · Quota you count, and errors that name the reason A refusal is a distinct, documented outcome and never a bare five hundred. ### Steps 1. Count units before the call, not after the failure The failure arrives too late to be a budget. Files: `src/quota.py`, `tests/test_quota.py` 2. Give every failure its own named outcome 'Upstream error' is not a thing a caller can act on. Files: `src/errors.py`, `src/api/routes.py` 3. Record every attempt, including the refusals The refusals are the ones you will want to read later. Files: `src/store/fetches.py` ### Done when - [ ] A spent budget returns a distinct status with a reset time - [ ] No failure path produces a bare five hundred - [ ] Rate limit hints are stored and passed to the caller - [ ] Every attempt has a fetch row ### Watch out - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. ## M4 · A cache keyed on what actually varies A repeat request costs nothing and a different language does not collide with the first one. ### Steps 1. Key on video, language and kind together Any one of the three is enough to make the answer wrong. Files: `src/cache.py`, `tests/test_cache_key.py` 2. Refresh creates a revision, never an overwrite Old segments are how you explain an old answer. Files: `src/store/tracks.py` 3. Put track identity in every response body So a caller can see it got something other than what it asked for. Files: `src/api/schemas.py` ### Done when - [ ] Two languages for one video do not collide - [ ] A cached hit makes no outbound call - [ ] Refresh creates a revision rather than overwriting - [ ] Every response names the track it served ### Watch out - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. ## M5 · Serving it to something other than yourself (production only) The service survives a second consumer, states its source policy, and exports everything. ### Steps 1. Check the per-key limit before spending global quota One noisy caller should not be able to take the day from everyone. Files: `src/limits.py`, `tests/test_limits.py` 2. Export as JSONL and prove the reimport Including the fetch log, which is the history of the refusals. Files: `src/export.py`, `tests/test_roundtrip.py` 3. Publish the source policy as an endpoint It is the honest answer to 'why can it not fetch that video'. Files: `src/api/policy.py`, `README.md` ### Done when - [ ] A key over its own limit is refused, others unaffected - [ ] Revocation takes effect without a restart - [ ] Export and reimport round-trips and the index rebuilds - [ ] The source policy is published and matches the README ### Watch out - A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal. ===== OPERATIONS.md ===== # Operations · TranscriptAPI ## Backup The database, because a track fetched from a video that has since been deleted or had its captions turned off cannot be fetched again. Nightly copy plus the export, which is the format that survives a schema change. ## Restore Restore the database, rebuild the search index from sentences, and confirm a known phrase search returns the same timestamp it did before. A restore that leaves the index stale looks healthy and answers wrongly. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Daily units spent against the limit, the proportion of fetches ending in each named outcome, and the age of the oldest served track. A sudden rise in one outcome value is the signal that something upstream changed. ## Incident checklist The realistic incident is not an outage, it is the official source starting to refuse. Because every attempt is logged with its outcome and every response names its track, you can tell when it started and which answers were served from cache afterwards. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] No fetcher exists for videos the authorised account cannot edit - [ ] Segments are confirmed immutable after insert - [ ] A cross-fragment phrase search is confirmed to return a correct timestamp - [ ] The quota counter is confirmed to refuse before the call, not after - [ ] Export and reimport are confirmed to round-trip with a rebuilt index ## Launch constraint Do not market omitted TranscriptAPI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Videos, tracks, segments, sentences and the fetch log. Back this up; refetching is not always possible. DATABASE_PATH=./data/transcripts.db # Optional. The OAuth client for the official captions source. Omit and only the file importer runs. GOOGLE_CLIENT_ID=1234-abcd.apps.googleusercontent.com # Optional · secret. Paired with the client id. Never in the repository. GOOGLE_CLIENT_SECRET=GOCSPX-...redacted # Optional · secret. Stored refresh token for the account that owns the videos. GOOGLE_TOKEN_PATH=./data/oauth-token.json # Required. Your project's published daily limit. Check it in the console rather than trusting this default. DAILY_QUOTA_UNITS=10000 # Required. How stale a cached track may be before a refresh is allowed. TRACK_MAX_AGE_DAYS=30 # Required. Turn on when something other than you starts calling this. CLIENT_KEYS_ENABLED=false
README.md
# TranscriptAPI · indie build A personal caption store whose hard problems are that the only supported fetch needs edit permission on the video, that automatic caption segments are phrase fragments rather than sentences, and that the API bills quota units per call so a playlist loop can exhaust the day before lunch. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and uvicorn, one process, no worker fleet | the work is network waiting and parsing, and a personal load fits in one process | | Storage | SQLite beside the app with a full text index over the sentence layer | a caption store is small, append heavy and read mostly | | Sources | A source interface with the official captions API and a caption file importer behind it | these are the two things you are actually entitled to read, and the seam takes a licensed third | | Authorisation | OAuth for the official API, local keys if you serve anyone but yourself | captions.download needs edit permission on the video and there is no route around it | | Units | Segment times as integer milliseconds, never floating point seconds | floats drift, compare badly, and ruin alignment when you join tracks | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the service and the importer. - Get it: python.org or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A Google Cloud project with the YouTube Data API enabled and an OAuth client** · free within the default quota - Why: The only supported caption fetch runs through it, and the project is also where the daily quota lives. - Get it: Create a project, enable the API, configure the consent screen and create an OAuth client for a desktop application. - Verify: A consent flow completes and captions.list returns tracks for a video you own - [ ] **At least one video you can edit, with a caption track** · free - Why: Without it you cannot exercise the official source at all, and the whole authorisation path stays untested. - Get it: Upload an unlisted test video to a channel you control and add captions to it. - Verify: The official source returns its track and segments - [ ] **A set of caption files in SRT, WebVTT and TTML** · free - Why: These are the importer's real inputs and the sentence heuristic needs an automatic-caption fixture with no punctuation to be tested honestly. - Get it: Export captions from a video you control in each format, and keep one deliberately unpunctuated. - Verify: All three import to identical row shapes - [ ] **Your written source policy** · free - Why: Deciding what the service may fetch after you have built the fetcher means deciding it from whatever already works. - Get it: Write down which sources are enabled, what each one requires, and that no evasion layer will be added. - Verify: The policy endpoint and the README carry the same text ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A proxy pool or any other blocking evasion. - Channel, playlist and search over videos you do not control. - Speech recognition over downloaded audio. - Any availability promise. - staying unblocked when YouTube rate-limits caption fetches - search, channel and playlist endpoints - reliability at bulk volume - an SLA and support when YouTube changes something If one of those is essential to you, that is the reason to keep paying for TranscriptAPI, and the README should say so rather than pretend.
BRIEF.md
# Build brief · TranscriptAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal transcript service that stores, searches and serves caption tracks from sources you are entitled to read to replace TranscriptAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Read this before you install a library, because it decides what the service is allowed to do. The official route is closed for other people's videos. The YouTube Data API captions.download method states plainly that it 'requires the user to have permission to edit the video', and captions.list returns track metadata only, under the same authorisation. Checked against Google's reference on 15 September 2026. So there is no supported way to pull the caption track of a video you do not control. Every three-line library that appears to do it anyway reads an endpoint YouTube does not publish. The YouTube API Services Developer Policies, section III.E, say you 'must not, and must not encourage, enable, or require others to, directly or indirectly, scrape YouTube Applications', and the main Terms prohibit accessing the Service by automated means. That is the whole moat of the paid product: a rented proxy fleet that keeps the scrape working after YouTube blocks it. This build does not do that. Renting residential addresses to keep reading an endpoint after the platform has cut you off is evasion, and no amount of retry logic makes it engineering. So build the honest version: a source interface with the official API behind it for videos you control, a caption file importer for everything else, and a documented seam for a provider you have a licence with. The cache, the sentence layer, the search and the serving are all genuinely yours, and they are most of the code. What you do not get is other people's videos on demand. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet · the work is network waiting and parsing, and a personal load fits in one process - Storage: SQLite beside the app with a full text index over the sentence layer · a caption store is small, append heavy and read mostly - Sources: A source interface with the official captions API and a caption file importer behind it · these are the two things you are actually entitled to read, and the seam takes a licensed third - Authorisation: OAuth for the official API, local keys if you serve anyone but yourself · captions.download needs edit permission on the video and there is no route around it - Units: Segment times as integer milliseconds, never floating point seconds · floats drift, compare badly, and ruin alignment when you join tracks ### Data model (create this before Phase 1) - videos(id, title, channel_id, duration_ms, first_seen_at, added_by_source) - tracks(id, video_id, language, kind, is_default, source, fetched_at, revision) · kind is manual or automatic, and one language can have both - segments(track_id, ordinal, start_ms, duration_ms, text) · the fetched truth, integers only, never rewritten - sentences(track_id, ordinal, start_ms, end_ms, text, first_segment, last_segment) · derived, with offsets back into segments so a search hit maps to a timestamp - fetches(id, track_id, at, source, outcome, quota_units, http_status, retry_after_s) · every attempt including the refusals, because the refusals are the interesting ones - quota(day, units_used, units_limit) · counted locally before the call rather than discovered by the failure - clients(id, label, created_at, daily_request_limit, revoked_at) · only if this ever serves someone other than you - There is no column anywhere holding 'the transcript'. A video has tracks, a track has segments, and every response names the track it came from. ### Phase 1 · A source interface, and the two sources you actually have Build: Define a source protocol returning a normalised track plus segments. Implement two: the official captions API using OAuth for videos you can edit, and an importer for SRT, WebVTT and TTML files. Record which source produced every track. Leave a third implementation stubbed and documented. Trap: Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. Done when: A track imported from a WebVTT file and a track fetched through the official API produce identical row shapes, asserted in tests. No module outside the source package makes an outbound request, checked by review. Every track row records its source and the time it was fetched. Importing the same file twice creates one track, not two. Do not build yet: No sentence layer, search, caching policy or serving to other clients. ### Phase 2 · Segments are the truth, sentences are derived Build: Store segments exactly as fetched. Build a sentence layer over them with a punctuation and gap heuristic, keeping the first and last segment ordinal on every sentence so a hit resolves to a start time. Index the sentences for full text search. Mark whether a track is automatic. Trap: Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. Done when: A search for a phrase that spans two caption fragments returns a hit with a start time inside a second of the spoken phrase, tested against a fixture track. Rebuilding the sentence layer from segments produces byte-identical rows, tested. Segment rows are never updated after insert, enforced by a trigger or asserted in tests. Automatic tracks are flagged and the flag appears in every response. Do not build yet: No quota accounting, caching policy, error taxonomy or serving. ### Phase 3 · Quota you count, and errors that name the reason Build: Count quota units locally before each call and refuse when the day's budget is spent. Map every failure to a named outcome: captions disabled, language not present, not authorised for this video, upstream refused, rate limited with a retry hint. Store the outcome on the fetch row. Trap: The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. Done when: Exhausting the local unit budget returns a distinct status with the time the budget resets, tested with a stubbed counter. Every documented failure maps to its own outcome value and none of them produces a five hundred, tested against a fake source. A rate limited response records the retry hint and the client is told it. The fetch table contains a row for every attempt, successful or not, checked after a test run. Do not build yet: No caching policy, deep links or multi-client serving. ### Phase 4 · A cache keyed on what actually varies Build: Cache tracks keyed on video, language and kind together, with a configurable age after which a track is refetched. Serve from cache by default and refetch only on an explicit parameter. Emit deep links at whole seconds alongside the millisecond values. Trap: Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. Done when: Requesting two languages for one video stores two tracks and returns the right one for each, tested. A repeat request inside the freshness window makes no outbound call, verified by counting fetch rows. The refresh parameter forces a new fetch and creates a new revision rather than overwriting the old segments. Every response names the video, language, kind and fetch time of the track it served. Do not build yet: No client keys, budgets or export yet. ### Phase 5 · Serving it to something other than yourself Build: Add client keys with per-key daily limits and a revoke path. Export every table as JSONL, and import the export into an empty database. Publish the source policy at an endpoint so a caller can read what this service is and is not allowed to fetch. Log slow and failed requests. Trap: A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal. Done when: A key that exceeds its own limit is refused while other keys keep working, tested with two keys. Revoking a key takes effect on the next request with no restart, tested. The export reimports into an empty database and the search index rebuilds with identical results, tested. The source policy endpoint states which sources are enabled and what each requires, and the README says the same thing. Do not build yet: Nothing after this. ### Out of scope (and why) - A rotating proxy pool, or any other means of continuing to read an endpoint after YouTube has blocked you. That is the paid product's actual business and it is evasion, not engineering. - Channel, playlist and search endpoints over videos you do not control, which need the same unsupported fetch at higher volume. - Running speech recognition over downloaded audio, which is a separate licence question about the audio as well as a much larger build. - An availability promise. This service is as reliable as a source it does not operate, and saying otherwise in a README is how you end up supporting it. ### README must contain - That captions.download requires permission to edit the video, quoted from Google's reference with the date checked - That the Developer Policies prohibit scraping YouTube applications, and that this build therefore has no fetcher for videos you do not control - The two working sources, what each needs, and how to plug in a provider you hold a licence with - That quota is counted locally, what the per-call unit costs were when this was written, and where to check the current daily limit - That segments are stored untouched and sentences are derived, so a search hit resolves to a real timestamp
AGENTS.md
# Agent instructions · TranscriptAPI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet, SQLite beside the app with a full text index over the sentence layer, A source interface with the official captions API and a caption file importer behind it, OAuth for the official API, local keys if you serve anyone but yourself, Segment times as integer milliseconds, never floating point seconds. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible.
BUILD_PLAN.md
# Build plan · TranscriptAPI A personal caption store whose hard problems are that the only supported fetch needs edit permission on the video, that automatic caption segments are phrase fragments rather than sentences, and that the API bills quota units per call so a playlist loop can exhaust the day before lunch. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · A source interface, and the two sources you actually have Captions enter the store through one interface, with the official API and a file importer behind it. ### Steps 1. Write the source protocol before either implementation Two implementations written against one interface is the point of the exercise. Files: `src/sources/base.py`, `src/models.py` 2. Implement the file importer first It has no credentials and no quota, so the schema gets settled cheaply. Files: `src/sources/captionfile.py`, `tests/test_import.py` 3. Add the official API source behind OAuth It only works on videos you can edit, and the code should say so in its docstring. Files: `src/sources/youtube_api.py`, `docs/authorisation.md` ### Done when - [ ] Both sources produce identical row shapes - [ ] No outbound request exists outside the source package - [ ] Every track records its source and fetch time - [ ] Re-importing a file does not duplicate a track ### Watch out - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. ## Phase 2 · Segments are the truth, sentences are derived Automatic caption fragments become searchable sentences without losing the timing. ### Steps 1. Never modify a segment after insert It is the only record of what the source actually said. Files: `src/store/segments.py`, `tests/test_immutable.py` 2. Derive sentences with a gap and punctuation rule And keep the segment range so a hit is a timestamp, not a video. Files: `src/sentences.py`, `tests/test_sentences.py` 3. Index sentences, not segments Fragment-level indexing cannot match a phrase a human would search for. Files: `src/search.py`, `migrations/002_fts.sql` ### Done when - [ ] A cross-fragment phrase search returns a correct start time - [ ] The sentence layer rebuilds identically from segments - [ ] Segments are never updated after insert - [ ] Automatic tracks are flagged in every response ### Watch out - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. ## Phase 3 · Quota you count, and errors that name the reason A refusal is a distinct, documented outcome and never a bare five hundred. ### Steps 1. Count units before the call, not after the failure The failure arrives too late to be a budget. Files: `src/quota.py`, `tests/test_quota.py` 2. Give every failure its own named outcome 'Upstream error' is not a thing a caller can act on. Files: `src/errors.py`, `src/api/routes.py` 3. Record every attempt, including the refusals The refusals are the ones you will want to read later. Files: `src/store/fetches.py` ### Done when - [ ] A spent budget returns a distinct status with a reset time - [ ] No failure path produces a bare five hundred - [ ] Rate limit hints are stored and passed to the caller - [ ] Every attempt has a fetch row ### Watch out - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. ## Phase 4 · A cache keyed on what actually varies A repeat request costs nothing and a different language does not collide with the first one. ### Steps 1. Key on video, language and kind together Any one of the three is enough to make the answer wrong. Files: `src/cache.py`, `tests/test_cache_key.py` 2. Refresh creates a revision, never an overwrite Old segments are how you explain an old answer. Files: `src/store/tracks.py` 3. Put track identity in every response body So a caller can see it got something other than what it asked for. Files: `src/api/schemas.py` ### Done when - [ ] Two languages for one video do not collide - [ ] A cached hit makes no outbound call - [ ] Refresh creates a revision rather than overwriting - [ ] Every response names the track it served ### Watch out - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. ## Not in this build - A proxy pool or any other blocking evasion. - Channel, playlist and search over videos you do not control. - Speech recognition over downloaded audio. - Any availability promise. ## After v1, if you want it - A diff view between two revisions of the same track, which is how you notice a caption edit - Speaker labels imported from formats that carry them, kept out of the derived sentence text - A small client library that fails loudly on the named outcomes rather than retrying them
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Videos, tracks, segments, sentences and the fetch log. Back this up; refetching is not always possible. DATABASE_PATH=./data/transcripts.db # Optional. The OAuth client for the official captions source. Omit and only the file importer runs. GOOGLE_CLIENT_ID=1234-abcd.apps.googleusercontent.com # Optional · secret. Paired with the client id. Never in the repository. GOOGLE_CLIENT_SECRET=GOCSPX-...redacted # Optional · secret. Stored refresh token for the account that owns the videos. GOOGLE_TOKEN_PATH=./data/oauth-token.json # Required. Your project's published daily limit. Check it in the console rather than trusting this default. DAILY_QUOTA_UNITS=10000 # Required. How stale a cached track may be before a refresh is allowed. TRACK_MAX_AGE_DAYS=30 # Required. Turn on when something other than you starts calling this. CLIENT_KEYS_ENABLED=false
PRODUCT.md
# TranscriptAPI · product brief ## Problem The happy path is genuinely a one-sitting build: the open-source youtube-transcript-api library fetches a caption track in a few lines, and wrapping it in FastAPI gives you a working personal endpoint. The honest gap is reliability. YouTube rate-limits and IP-blocks caption scraping at any real volume, so the DIY version works until it suddenly does not, and there is no fix without a rotating proxy pool you must rent and operate. The paid product is not selling the parsing; it is selling the unblocked pipe, plus search and playlist endpoints on top. ## Product outcome A caption store that is honest about what it may fetch, keeps the fetched segments untouched, resolves a phrase search to a timestamp, counts its quota before spending it, and exports everything including the record of every refusal. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Python 3.11 - a rotating proxy pool if you use it at any volume ## Explicit non-goals for v1 - A proxy pool or any other blocking evasion. - Channel, playlist and search over videos you do not control. - Speech recognition over downloaded audio. - Any availability promise. - staying unblocked when YouTube rate-limits caption fetches - search, channel and playlist endpoints - reliability at bulk volume - an SLA and support when YouTube changes something ## Success criteria - No fetcher exists for videos the authorised account cannot edit - Segments are confirmed immutable after insert - A cross-fragment phrase search is confirmed to return a correct timestamp - The quota counter is confirmed to refuse before the call, not after - Export and reimport are confirmed to round-trip with a rebuilt index
BRIEF.md
# Build brief · TranscriptAPI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal transcript service that stores, searches and serves caption tracks from sources you are entitled to read to replace TranscriptAPI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Read this before you install a library, because it decides what the service is allowed to do. The official route is closed for other people's videos. The YouTube Data API captions.download method states plainly that it 'requires the user to have permission to edit the video', and captions.list returns track metadata only, under the same authorisation. Checked against Google's reference on 15 September 2026. So there is no supported way to pull the caption track of a video you do not control. Every three-line library that appears to do it anyway reads an endpoint YouTube does not publish. The YouTube API Services Developer Policies, section III.E, say you 'must not, and must not encourage, enable, or require others to, directly or indirectly, scrape YouTube Applications', and the main Terms prohibit accessing the Service by automated means. That is the whole moat of the paid product: a rented proxy fleet that keeps the scrape working after YouTube blocks it. This build does not do that. Renting residential addresses to keep reading an endpoint after the platform has cut you off is evasion, and no amount of retry logic makes it engineering. So build the honest version: a source interface with the official API behind it for videos you control, a caption file importer for everything else, and a documented seam for a provider you have a licence with. The cache, the sentence layer, the search and the serving are all genuinely yours, and they are most of the code. What you do not get is other people's videos on demand. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet · the work is network waiting and parsing, and a personal load fits in one process - Storage: SQLite beside the app with a full text index over the sentence layer · a caption store is small, append heavy and read mostly - Sources: A source interface with the official captions API and a caption file importer behind it · these are the two things you are actually entitled to read, and the seam takes a licensed third - Authorisation: OAuth for the official API, local keys if you serve anyone but yourself · captions.download needs edit permission on the video and there is no route around it - Units: Segment times as integer milliseconds, never floating point seconds · floats drift, compare badly, and ruin alignment when you join tracks ### Data model (create this before Phase 1) - videos(id, title, channel_id, duration_ms, first_seen_at, added_by_source) - tracks(id, video_id, language, kind, is_default, source, fetched_at, revision) · kind is manual or automatic, and one language can have both - segments(track_id, ordinal, start_ms, duration_ms, text) · the fetched truth, integers only, never rewritten - sentences(track_id, ordinal, start_ms, end_ms, text, first_segment, last_segment) · derived, with offsets back into segments so a search hit maps to a timestamp - fetches(id, track_id, at, source, outcome, quota_units, http_status, retry_after_s) · every attempt including the refusals, because the refusals are the interesting ones - quota(day, units_used, units_limit) · counted locally before the call rather than discovered by the failure - clients(id, label, created_at, daily_request_limit, revoked_at) · only if this ever serves someone other than you - There is no column anywhere holding 'the transcript'. A video has tracks, a track has segments, and every response names the track it came from. ### Phase 1 · A source interface, and the two sources you actually have Build: Define a source protocol returning a normalised track plus segments. Implement two: the official captions API using OAuth for videos you can edit, and an importer for SRT, WebVTT and TTML files. Record which source produced every track. Leave a third implementation stubbed and documented. Trap: Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. Done when: A track imported from a WebVTT file and a track fetched through the official API produce identical row shapes, asserted in tests. No module outside the source package makes an outbound request, checked by review. Every track row records its source and the time it was fetched. Importing the same file twice creates one track, not two. Do not build yet: No sentence layer, search, caching policy or serving to other clients. ### Phase 2 · Segments are the truth, sentences are derived Build: Store segments exactly as fetched. Build a sentence layer over them with a punctuation and gap heuristic, keeping the first and last segment ordinal on every sentence so a hit resolves to a start time. Index the sentences for full text search. Mark whether a track is automatic. Trap: Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. Done when: A search for a phrase that spans two caption fragments returns a hit with a start time inside a second of the spoken phrase, tested against a fixture track. Rebuilding the sentence layer from segments produces byte-identical rows, tested. Segment rows are never updated after insert, enforced by a trigger or asserted in tests. Automatic tracks are flagged and the flag appears in every response. Do not build yet: No quota accounting, caching policy, error taxonomy or serving. ### Phase 3 · Quota you count, and errors that name the reason Build: Count quota units locally before each call and refuse when the day's budget is spent. Map every failure to a named outcome: captions disabled, language not present, not authorised for this video, upstream refused, rate limited with a retry hint. Store the outcome on the fetch row. Trap: The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. Done when: Exhausting the local unit budget returns a distinct status with the time the budget resets, tested with a stubbed counter. Every documented failure maps to its own outcome value and none of them produces a five hundred, tested against a fake source. A rate limited response records the retry hint and the client is told it. The fetch table contains a row for every attempt, successful or not, checked after a test run. Do not build yet: No caching policy, deep links or multi-client serving. ### Phase 4 · A cache keyed on what actually varies Build: Cache tracks keyed on video, language and kind together, with a configurable age after which a track is refetched. Serve from cache by default and refetch only on an explicit parameter. Emit deep links at whole seconds alongside the millisecond values. Trap: Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. Done when: Requesting two languages for one video stores two tracks and returns the right one for each, tested. A repeat request inside the freshness window makes no outbound call, verified by counting fetch rows. The refresh parameter forces a new fetch and creates a new revision rather than overwriting the old segments. Every response names the video, language, kind and fetch time of the track it served. Do not build yet: No client keys, budgets or export yet. ### Phase 5 · Serving it to something other than yourself Build: Add client keys with per-key daily limits and a revoke path. Export every table as JSONL, and import the export into an empty database. Publish the source policy at an endpoint so a caller can read what this service is and is not allowed to fetch. Log slow and failed requests. Trap: A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal. Done when: A key that exceeds its own limit is refused while other keys keep working, tested with two keys. Revoking a key takes effect on the next request with no restart, tested. The export reimports into an empty database and the search index rebuilds with identical results, tested. The source policy endpoint states which sources are enabled and what each requires, and the README says the same thing. Do not build yet: Nothing after this. ### Out of scope (and why) - A rotating proxy pool, or any other means of continuing to read an endpoint after YouTube has blocked you. That is the paid product's actual business and it is evasion, not engineering. - Channel, playlist and search endpoints over videos you do not control, which need the same unsupported fetch at higher volume. - Running speech recognition over downloaded audio, which is a separate licence question about the audio as well as a much larger build. - An availability promise. This service is as reliable as a source it does not operate, and saying otherwise in a README is how you end up supporting it. ### README must contain - That captions.download requires permission to edit the video, quoted from Google's reference with the date checked - That the Developer Policies prohibit scraping YouTube applications, and that this build therefore has no fetcher for videos you do not control - The two working sources, what each needs, and how to plug in a provider you hold a licence with - That quota is counted locally, what the per-call unit costs were when this was written, and where to check the current daily limit - That segments are stored untouched and sentences are derived, so a search hit resolves to a real timestamp
ARCHITECTURE.md
# Architecture · TranscriptAPI ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and uvicorn, one process, no worker fleet | the work is network waiting and parsing, and a personal load fits in one process | | Storage | SQLite beside the app with a full text index over the sentence layer | a caption store is small, append heavy and read mostly | | Sources | A source interface with the official captions API and a caption file importer behind it | these are the two things you are actually entitled to read, and the seam takes a licensed third | | Authorisation | OAuth for the official API, local keys if you serve anyone but yourself | captions.download needs edit permission on the video and there is no route around it | | Units | Segment times as integer milliseconds, never floating point seconds | floats drift, compare badly, and ruin alignment when you join tracks | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Sources | The official API client and the file importer. | Any source you hold a licence for. Never a scraper, whatever the retry logic looks like. | | Store | Videos, tracks, immutable segments, revisions. | Any database, provided segments stay immutable and revisions accumulate. | | Sentences | The derived layer and the search index. | Any heuristic, provided the segment range survives on every sentence. | | Quota | Local unit counting and per-key limits. | Any counter, provided it decides before the call rather than after. | | Serving | Routes, client keys, export and the policy endpoint. | Any framework. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_PATH` · required · Videos, tracks, segments, sentences and the fetch log. Back this up; refetching is not always possible. - `GOOGLE_CLIENT_ID` · optional · The OAuth client for the official captions source. Omit and only the file importer runs. - `GOOGLE_CLIENT_SECRET` · optional, secret · Paired with the client id. Never in the repository. - `GOOGLE_TOKEN_PATH` · optional, secret · Stored refresh token for the account that owns the videos. - `DAILY_QUOTA_UNITS` · required · Your project's published daily limit. Check it in the console rather than trusting this default. - `TRACK_MAX_AGE_DAYS` · required · How stale a cached track may be before a refresh is allowed. - `CLIENT_KEYS_ENABLED` · required · Turn on when something other than you starts calling this. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · TranscriptAPI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with FastAPI and uvicorn, one process, no worker fleet, SQLite beside the app with a full text index over the sentence layer, A source interface with the official captions API and a caption file importer behind it, OAuth for the official API, local keys if you serve anyone but yourself, Segment times as integer milliseconds, never floating point seconds. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. - A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal.
MILESTONES.md
# Delivery milestones · TranscriptAPI Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · A source interface, and the two sources you actually have Captions enter the store through one interface, with the official API and a file importer behind it. ### Steps 1. Write the source protocol before either implementation Two implementations written against one interface is the point of the exercise. Files: `src/sources/base.py`, `src/models.py` 2. Implement the file importer first It has no credentials and no quota, so the schema gets settled cheaply. Files: `src/sources/captionfile.py`, `tests/test_import.py` 3. Add the official API source behind OAuth It only works on videos you can edit, and the code should say so in its docstring. Files: `src/sources/youtube_api.py`, `docs/authorisation.md` ### Done when - [ ] Both sources produce identical row shapes - [ ] No outbound request exists outside the source package - [ ] Every track records its source and fetch time - [ ] Re-importing a file does not duplicate a track ### Watch out - Calling a scraping library directly from your request handler feels like it saves a day, and it hard-wires both the legal exposure and the technical fragility into every code path you will later write, so when YouTube blocks the endpoint there is nothing to swap. Put every fetch behind the interface from the first commit, and let the stub carry the note about what a licensed provider would need to satisfy. ## M2 · Segments are the truth, sentences are derived Automatic caption fragments become searchable sentences without losing the timing. ### Steps 1. Never modify a segment after insert It is the only record of what the source actually said. Files: `src/store/segments.py`, `tests/test_immutable.py` 2. Derive sentences with a gap and punctuation rule And keep the segment range so a hit is a timestamp, not a video. Files: `src/sentences.py`, `tests/test_sentences.py` 3. Index sentences, not segments Fragment-level indexing cannot match a phrase a human would search for. Files: `src/search.py`, `migrations/002_fts.sql` ### Done when - [ ] A cross-fragment phrase search returns a correct start time - [ ] The sentence layer rebuilds identically from segments - [ ] Segments are never updated after insert - [ ] Automatic tracks are flagged in every response ### Watch out - Automatic captions arrive as short unpunctuated fragments that split mid-sentence, so joining them with spaces and indexing the result gives you a body of text in which no phrase search ever matches what a person would type, and every hit resolves to the whole video. Keep segments untouched, derive sentences on top, and store the segment range on each sentence so a search result is a timestamp. ## M3 · Quota you count, and errors that name the reason A refusal is a distinct, documented outcome and never a bare five hundred. ### Steps 1. Count units before the call, not after the failure The failure arrives too late to be a budget. Files: `src/quota.py`, `tests/test_quota.py` 2. Give every failure its own named outcome 'Upstream error' is not a thing a caller can act on. Files: `src/errors.py`, `src/api/routes.py` 3. Record every attempt, including the refusals The refusals are the ones you will want to read later. Files: `src/store/fetches.py` ### Done when - [ ] A spent budget returns a distinct status with a reset time - [ ] No failure path produces a bare five hundred - [ ] Rate limit hints are stored and passed to the caller - [ ] Every attempt has a fetch row ### Watch out - The API bills quota units per call, and on 15 September 2026 Google's reference listed captions.list at fifty units and captions.download at two hundred, so a loop over a playlist burns a day's project budget in minutes and every later call then fails in a way that looks exactly like a network error. Check your project's current daily limit in the console, count units locally before the call, and return a distinct status when the budget is gone rather than a generic upstream failure. ## M4 · A cache keyed on what actually varies A repeat request costs nothing and a different language does not collide with the first one. ### Steps 1. Key on video, language and kind together Any one of the three is enough to make the answer wrong. Files: `src/cache.py`, `tests/test_cache_key.py` 2. Refresh creates a revision, never an overwrite Old segments are how you explain an old answer. Files: `src/store/tracks.py` 3. Put track identity in every response body So a caller can see it got something other than what it asked for. Files: `src/api/schemas.py` ### Done when - [ ] Two languages for one video do not collide - [ ] A cached hit makes no outbound call - [ ] Refresh creates a revision rather than overwriting - [ ] Every response names the track it served ### Watch out - Keying the cache on the video alone means the first request decides the language and kind for every request that follows, so a caller asking for the manual German track quietly receives the automatic English one and nothing in the response contradicts it. Key on video, language and kind together, and put the track identity in the response body so a wrong answer is visible. ## M5 · Serving it to something other than yourself (production only) The service survives a second consumer, states its source policy, and exports everything. ### Steps 1. Check the per-key limit before spending global quota One noisy caller should not be able to take the day from everyone. Files: `src/limits.py`, `tests/test_limits.py` 2. Export as JSONL and prove the reimport Including the fetch log, which is the history of the refusals. Files: `src/export.py`, `tests/test_roundtrip.py` 3. Publish the source policy as an endpoint It is the honest answer to 'why can it not fetch that video'. Files: `src/api/policy.py`, `README.md` ### Done when - [ ] A key over its own limit is refused, others unaffected - [ ] Revocation takes effect without a restart - [ ] Export and reimport round-trips and the index rebuilds - [ ] The source policy is published and matches the README ### Watch out - A personal service that quietly gets a second consumer will exhaust the shared daily quota on whichever caller loops hardest, and because the quota is global the other caller sees identical failures with no explanation. Give every key its own daily limit, spend from the global budget only after the per-key check passes, and say which limit was hit in the refusal.
OPERATIONS.md
# Operations · TranscriptAPI ## Backup The database, because a track fetched from a video that has since been deleted or had its captions turned off cannot be fetched again. Nightly copy plus the export, which is the format that survives a schema change. ## Restore Restore the database, rebuild the search index from sentences, and confirm a known phrase search returns the same timestamp it did before. A restore that leaves the index stale looks healthy and answers wrongly. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Daily units spent against the limit, the proportion of fetches ending in each named outcome, and the age of the oldest served track. A sudden rise in one outcome value is the signal that something upstream changed. ## Incident checklist The realistic incident is not an outage, it is the official source starting to refuse. Because every attempt is logged with its outcome and every response names its track, you can tell when it started and which answers were served from cache afterwards. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] No fetcher exists for videos the authorised account cannot edit - [ ] Segments are confirmed immutable after insert - [ ] A cross-fragment phrase search is confirmed to return a correct timestamp - [ ] The quota counter is confirmed to refuse before the call, not after - [ ] Export and reimport are confirmed to round-trip with a rebuilt index ## Launch constraint Do not market omitted TranscriptAPI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Videos, tracks, segments, sentences and the fetch log. Back this up; refetching is not always possible. DATABASE_PATH=./data/transcripts.db # Optional. The OAuth client for the official captions source. Omit and only the file importer runs. GOOGLE_CLIENT_ID=1234-abcd.apps.googleusercontent.com # Optional · secret. Paired with the client id. Never in the repository. GOOGLE_CLIENT_SECRET=GOCSPX-...redacted # Optional · secret. Stored refresh token for the account that owns the videos. GOOGLE_TOKEN_PATH=./data/oauth-token.json # Required. Your project's published daily limit. Check it in the console rather than trusting this default. DAILY_QUOTA_UNITS=10000 # Required. How stale a cached track may be before a refresh is allowed. TRACK_MAX_AGE_DAYS=30 # Required. Turn on when something other than you starts calling this. CLIENT_KEYS_ENABLED=false
$ choose a build depth, inspect the files, then open the complete pack in your agent
The parse was never the hard part. Holding a durable, unblocked pipe into YouTube captions at volume is an operations problem that costs real money to run, and buying it for 5 USD a month is cheaper than renting proxies and babysitting them.
xstaying unblocked when YouTube rate-limits caption fetches
xsearch, channel and playlist endpoints
xreliability at bulk volume
xan SLA and support when YouTube changes something
share on X ↗"I just replaced TranscriptAPI ($5/mo) with one prompt"
TranscriptAPI pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| free signup | $0 | $0 | 100 credits; no card; credits expire after 90 days. |
| starter | $5 | $4.50 | 1,000 credits/month; 200 requests/minute on monthly billing or 300 requests/minute on annual billing. |
| custom | custom | — | Custom credits, rate limits and support. |
free tier100 signup credits; no card; expire after 90 days
billingmonthly or annual; annual Starter is $54/year
hidden costsBase subscription credits do not roll over. Top-ups cost $2.50/1,000 credits on monthly Starter or $1.50/1,000 on annual Starter, require an active subscription, and pause if the subscription lapses.
verified 2026-08-14 · source ↗
Vibecode TranscriptAPI
Kinda. The core of TranscriptAPI is buildable in a weekend with the prompt on this page, but there are real gaps: staying unblocked when YouTube rate-limits caption fetches, search, channel and playlist endpoints. Read the honest list above before committing.
How much does TranscriptAPI cost?
TranscriptAPI costs about $5/month (Starter, checked 2026-07-31), which is $60 per year.
What do I lose by replacing TranscriptAPI?
Honestly: staying unblocked when YouTube rate-limits caption fetches; search, channel and playlist endpoints; reliability at bulk volume; an SLA and support when YouTube changes something. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to TranscriptAPI?
Yes: youtube-transcript-api (open-source Python library for the core caption fetch). Using prior art is also vibecoding; the prompt is for when you want it exactly your way.