Vibecode AudioFetcher
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The core loop is genuinely one-shottable: a small local UI, yt-dlp for the fetch, ffmpeg for the encode, files land in a folder. What is not a one sitting is everything that makes a public converter feel reliable week after week · YouTube bot checks, cookies/PO tokens, geo blocks, fair queues, abuse protection, and a hosted path so non-technical people never open a terminal. Build the personal script; do not expect one prompt to replace a maintained public service.
the project pack5 files · written for this build, step by step
You are building a lean indie version of AudioFetcher. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # AudioFetcher · indie build A personal audio library whose hard problems are fetching politely and only once from sources that publish downloads, refusing to re-encode audio that is already in the format you asked for, and keeping an honest record of the permission claim behind anything fetched through the optional extractor. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with a small local web interface bound to the loopback address | this is one person's library and it should not be reachable from anywhere else | | Fetching | Conditional requests with stored validators, one connection per host at a time | a feed poller that ignores caching headers is a poller that gets blocked | | Encoding | ffmpeg, with stream copy preferred over transcoding wherever the format already matches | re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse | | Storage | SQLite for rows, audio files on disk keyed by checksum | the checksum is the identity of a file and the filename is only a convenience | | Extractors | An optional plugin interface, disabled unless configured, with an attestation record | the honest way to offer this is as a deliberate act with a name and a date attached | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the local interface and the fetch queue. - Get it: python.org or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **ffmpeg with ffprobe** · free - Why: Probing before encoding is what makes the stream-copy path possible, and ffprobe is how you know whether to encode at all. - Get it: Your package manager. - Verify: ffprobe -version prints a version and ffmpeg -codecs lists an MP3 encoder - [ ] **The published terms of every host you intend to fetch from, read and recorded** · free - Why: Rule zero rests on what those documents say today, and they change, so reading them is part of setup rather than part of the argument afterwards. - Get it: Read each host's terms and record the URL and the date in your configuration. - Verify: Every configured host has a terms URL and a date recorded - [ ] **Disk space with a number you have decided in advance** · free - Why: An audio library grows until something stops it, and the thing that stops it should be a ceiling you chose rather than a full filesystem. - Get it: Decide the ceiling and the path, and put both in configuration. - Verify: Filling to the ceiling refuses new fetches - [ ] **Whether any extractor will be enabled, and on what basis** · free - Why: This decision changes what the tool is, and making it deliberately at the start is the difference between a library and a liability. - Get it: Write down the answer, and if it is yes, write the attestation statement you are prepared to sign. - Verify: The README and the configuration agree, and the attestation row exists if enabled ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A hosted service other people can use. - Anything that works around a protection measure. - A shipped, enabled extractor for any platform. - Loudness normalisation and audio repair. - hosted UI for non-technical users - playlist preview with per-track checkboxes - priority queue and 320 kbps Pro path - someone else fighting YouTube bot checks, PO tokens, and geo blocks - email magic-link Pro auth and fair multi-user queueing If one of those is essential to you, that is the reason to keep paying for AudioFetcher, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · AudioFetcher The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal audio fetcher that downloads what publishers offer for download, encodes it once, and refuses the rest to replace AudioFetcher. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This product downloads audio from somewhere else, so the platform terms and the copyright position are the design rather than a paragraph at the end of the README. YouTube's Terms of Service, currently effective 17 March 2025, forbid accessing, reproducing, downloading, distributing or otherwise using any part of the Service or Content except as expressly permitted by the Service, with prior written permission, or as permitted by applicable law. A second restriction forbids circumventing anything that prevents or restricts copying, and a third forbids automated access except by public search engines following robots.txt. In the United States, 17 U.S.C. 1201(a)(1)(A) says that no person shall circumvent a technological measure that effectively controls access to a work. And the private copying exception many people assume exists does not exist in the United Kingdom: section 28B of the 1988 Act was inserted in 2014, the regulations that inserted it were quashed by the High Court on 17 July 2015, and the Intellectual Property Office's own notice says those acts now constitute acts of infringement. So the default build is not a video ripper. It fetches what the publisher offers for download: podcast enclosures from a feed, direct file addresses, archives that publish downloadable media, and files you already hold. A site-specific extractor is optional and off by default, and turning it on records who attested what and when. Nothing here circumvents anything, and no checkbox here is pretending to be a licence. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with a small local web interface bound to the loopback address · this is one person's library and it should not be reachable from anywhere else - Fetching: Conditional requests with stored validators, one connection per host at a time · a feed poller that ignores caching headers is a poller that gets blocked - Encoding: ffmpeg, with stream copy preferred over transcoding wherever the format already matches · re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse - Storage: SQLite for rows, audio files on disk keyed by checksum · the checksum is the identity of a file and the filename is only a convenience - Extractors: An optional plugin interface, disabled unless configured, with an attestation record · the honest way to offer this is as a deliberate act with a name and a date attached ### Data model (create this before Phase 1) - sources(id, kind, url, added_at, etag, last_modified, licence_note, attestation_id) · kind is feed, direct, archive or extractor - items(id, source_id, remote_guid, title, published_at, media_url, bytes, mime, duration_ms, checksum, state) - fetches(id, item_id, attempt, http_status, bytes, started_at, ended_at, error_excerpt) · every attempt including the refused ones - files(item_id, path, format, bitrate_kbps, channels, sample_rate, transcoded_from, checksum) · transcoded_from records the chain, or is null when the stream was copied - attestations(id, made_at, who, statement, source_url) · a claim a person made with a date, which is a different and more honest thing than a permission - host_policy(host, robots_fetched_at, crawl_delay_seconds, max_concurrency, backoff_until) · one row per host you touch - No row here says permission granted. The nearest thing is a recorded claim, and the interface says so wherever it shows one. ### Phase 1 · Sources that publish downloads Build: Add a feed by address, parse its enclosures, and record each item with the publisher's own identifier. Store the entity tag and last-modified value from every response and send them back as conditional request headers on the next poll. Read the host's robots file before crawling it and record what it said. Never fetch the same media address twice for the same item. Trap: Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. Done when: A second poll of an unchanged feed sends the stored validators and transfers no body, verified against a local server that asserts the headers arrived. An item already fetched is never fetched again even when the feed reorders or republishes it, tested with a feed fixture that does both. A Retry-After header delays the next request by at least the stated interval, tested. The robots response for each host is recorded with the time it was read, and a disallowed path is not fetched. Do not build yet: No encoding, no extractor, no library interface yet. ### Phase 2 · Encode once, or not at all Build: Probe the source stream first. If the codec and bitrate already satisfy the request, copy the stream and write files.transcoded_from as null. Otherwise transcode once, with the parameters recorded, and store the chain. Write tags from the feed metadata including the source address and the fetch date. Keep the original file until an explicit purge. Trap: Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. Done when: A source already matching the requested format produces a file whose audio stream is bit-identical to the source, tested by comparing stream checksums. An upward bitrate request is refused with an explanation rather than performed, tested. Every transcoded file records its source format and the encoder parameters in transcoded_from, and no file has a null chain unless it was a stream copy, asserted in the suite. Tags include the source address and the fetch date and survive a round trip through a tag reader. Do not build yet: No extractor, no queue, no library or export yet. ### Phase 3 · The extractor boundary Build: Define a plugin interface and ship no extractor enabled by default. Enabling one requires configuration plus an attestation row naming the person, the statement and the date. Refuse to run an extractor against a host whose published terms the operator has not recorded reading. Never implement anything that descrambles, decrypts or otherwise bypasses a protection measure, whatever a library offers. Trap: An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. Done when: With no extractor configured, the interface offers only feeds, direct addresses and local files, and the extractor code path is unreachable, tested. Enabling an extractor without an attestation row fails with a message naming what is missing, tested. Every item fetched through an extractor displays its attestation, its date and who made it, checked in the interface test. No code path in the repository performs decryption or signature evasion of any kind, verified by review and stated in the README. Do not build yet: No queue policy, no library interface, no ceilings yet. ### Phase 4 · A queue that is polite to strangers Build: Queue fetches with at most the configured concurrency per host and a delay between requests taken from the host policy. Back off on a 429 or 503 and store backoff_until so a restart does not reset it. Refuse media longer than a configured ceiling and refuse anything reporting itself as a live stream. Show the queue with the reason each item is waiting. Trap: Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. Done when: Two queued items for the same host never download concurrently at the default configuration, tested against a server that fails on overlap. A 429 response sets backoff_until and a process restart preserves it, tested by restarting mid-backoff. Media exceeding the duration ceiling is refused before any bytes are transferred, tested using the feed's declared duration and a probe. A live stream is refused with a named reason rather than downloaded indefinitely, tested. Do not build yet: No disk ceilings, no export, no retention yet. ### Phase 5 · Disk, retention and an honest README Build: Enforce a total size ceiling and refuse new fetches when it is reached rather than deleting to make room. Add retention by source with an explicit keep list. Export the whole library as a manifest of checksums, titles, sources, attestations and file paths, so the audio and the record can be moved together. Write the README so a stranger can tell in thirty seconds what is in scope and what is not. Trap: A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all. Done when: Reaching the size ceiling refuses new fetches with a named reason and deletes nothing, tested by filling to the boundary. Retention deletes only sources with an explicit policy, and an item on the keep list survives a retention run, tested. The exported manifest reimports into an empty database and every file resolves by checksum, tested. The README states the terms position, the copyright position and the extractor default in its first section, checked by review. Do not build yet: Nothing after this. ### Out of scope (and why) - A hosted service other people can paste addresses into, because you would then be operating the thing the terms above are written about, at your own risk and on your own domain. - Anything that descrambles, decrypts or works around a protection measure, which is a separate prohibition from the terms and is not negotiable here. - Shipping an enabled extractor for any platform, since the entire attestation design exists because that choice belongs to the operator and not to the author. - Loudness normalisation and audio repair, which are a genuinely interesting project and have nothing to do with the problems this build is about. ### README must contain - The platform terms position and the copyright position, stated in the first section with the sources and the dates they were read - That the private copying exception people assume exists was quashed in the United Kingdom in 2015 and that the position differs by jurisdiction - That no extractor is enabled by default, what enabling one requires, and that an attestation is a recorded claim rather than a permission - That audio already in the requested format is copied rather than re-encoded, and why an upward bitrate request is refused - Where the files live, what the size ceiling does when reached, and how to export the manifest with the audio ===== AGENTS.md ===== # Agent instructions · AudioFetcher indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with a small local web interface bound to the loopback address, Conditional requests with stored validators, one connection per host at a time, ffmpeg, with stream copy preferred over transcoding wherever the format already matches, SQLite for rows, audio files on disk keyed by checksum, An optional plugin interface, disabled unless configured, with an attestation record. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. ===== BUILD_PLAN.md ===== # Build plan · AudioFetcher A personal audio library whose hard problems are fetching politely and only once from sources that publish downloads, refusing to re-encode audio that is already in the format you asked for, and keeping an honest record of the permission claim behind anything fetched through the optional extractor. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sources that publish downloads Feeds and direct addresses are fetched once each, politely, with validators stored so a poll is usually free. ### Steps 1. Store validators and send them back A not-modified response should be the common case. Files: `src/fetch/conditional.py`, `tests/test_conditional.py` 2. Key items on the publisher's identifier Feeds reorder, republish and change titles. Files: `src/feeds/parse.py`, `tests/test_feed_dedupe.py` 3. Read robots and honour Retry-After Both are instructions, and ignoring them is how access ends. Files: `src/fetch/policy.py`, `tests/test_retry_after.py` ### Done when - [ ] An unchanged feed transfers no body on the second poll - [ ] A reordered or republished item is not refetched - [ ] Retry-After delays the next request - [ ] Robots responses are recorded and honoured ### Watch out - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. ## Phase 2 · Encode once, or not at all Audio already in the requested format is copied rather than re-encoded, and any transcode records what it came from. ### Steps 1. Probe before deciding to encode Most fetches do not need an encoder at all. Files: `src/encode/probe.py`, `tests/test_streamcopy.py` 2. Refuse an upward bitrate request There is nothing to put back and the file gets worse. Files: `src/encode/plan.py` 3. Record the chain and tag the source Six months later this is the only provenance you have. Files: `src/encode/run.py`, `src/encode/tags.py` ### Done when - [ ] A matching source is stream-copied bit-identically - [ ] An upward bitrate request is refused with a reason - [ ] Every transcode records its chain and parameters - [ ] Tags carry the source address and fetch date ### Watch out - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. ## Phase 3 · The extractor boundary Site-specific extraction is off unless someone deliberately turns it on and records why. ### Steps 1. Ship with no extractor enabled The default build is the one most people will run. Files: `src/extractors/__init__.py`, `tests/test_default_disabled.py` 2. Require a stored attestation, not a checkbox A claim with a name and a date is the only honest artefact here. Files: `src/extractors/attest.py`, `migrations/003_attestations.sql` 3. Refuse anything that bypasses a protection measure This is a line, not a configuration option. Files: `docs/scope.md` ### Done when - [ ] The default build exposes no extractor - [ ] Enabling without an attestation fails with a reason - [ ] Extractor items display their attestation - [ ] No circumvention code exists in the repository ### Watch out - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. ## Phase 4 · A queue that is polite to strangers Downloads run one host at a time with recorded backoff, and a long or live source is refused before it starts. ### Steps 1. Limit concurrency per host, not globally One slow publisher should not be hammered by your whole queue. Files: `src/queue/claim.py`, `tests/test_host_concurrency.py` 2. Persist backoff across restarts In-memory backoff resets exactly when you least want it to. Files: `src/queue/backoff.py`, `tests/test_backoff_persist.py` 3. Refuse live and over-long media up front Before bytes, not after two hours of them. Files: `src/queue/guard.py` ### Done when - [ ] One host is never downloaded from concurrently by default - [ ] Backoff survives a restart - [ ] Over-long media is refused before transfer - [ ] Live streams are refused with a reason ### Watch out - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. ## Not in this build - A hosted service other people can use. - Anything that works around a protection measure. - A shipped, enabled extractor for any platform. - Loudness normalisation and audio repair. ## After v1, if you want it - Chapter and transcript handling for feeds that publish them, which is metadata the publisher already offers - A local search over titles and descriptions, which costs nothing once the rows are already there - A player that reads the manifest directly, so the library works even if this application stops running ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Audio files, keyed by checksum. The largest thing on the disk. LIBRARY_DIR=./library # Required. Sources, items, fetches and attestations. Back this up; the audio is refetchable and the record is not. DATABASE_PATH=./data/library.db # Required. New fetches are refused at this size. Nothing is ever evicted to make room. LIBRARY_CEILING_GB=50 # Required. Media longer than this is refused before any bytes move. MAX_DURATION_MINUTES=240 # Required. Simultaneous downloads per host. Raise it only with a reason. HOST_CONCURRENCY=1 # Required. Off unless deliberately set, and setting it needs an attestation row to exist. EXTRACTORS_ENABLED=false # Required. Identifies you honestly to the hosts you fetch from, with a way to reach you. USER_AGENT=my-audio-library/1.0 (contact: me@example.com)
You are building a lean indie version of AudioFetcher. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # AudioFetcher · indie build A personal audio library whose hard problems are fetching politely and only once from sources that publish downloads, refusing to re-encode audio that is already in the format you asked for, and keeping an honest record of the permission claim behind anything fetched through the optional extractor. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with a small local web interface bound to the loopback address | this is one person's library and it should not be reachable from anywhere else | | Fetching | Conditional requests with stored validators, one connection per host at a time | a feed poller that ignores caching headers is a poller that gets blocked | | Encoding | ffmpeg, with stream copy preferred over transcoding wherever the format already matches | re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse | | Storage | SQLite for rows, audio files on disk keyed by checksum | the checksum is the identity of a file and the filename is only a convenience | | Extractors | An optional plugin interface, disabled unless configured, with an attestation record | the honest way to offer this is as a deliberate act with a name and a date attached | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the local interface and the fetch queue. - Get it: python.org or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **ffmpeg with ffprobe** · free - Why: Probing before encoding is what makes the stream-copy path possible, and ffprobe is how you know whether to encode at all. - Get it: Your package manager. - Verify: ffprobe -version prints a version and ffmpeg -codecs lists an MP3 encoder - [ ] **The published terms of every host you intend to fetch from, read and recorded** · free - Why: Rule zero rests on what those documents say today, and they change, so reading them is part of setup rather than part of the argument afterwards. - Get it: Read each host's terms and record the URL and the date in your configuration. - Verify: Every configured host has a terms URL and a date recorded - [ ] **Disk space with a number you have decided in advance** · free - Why: An audio library grows until something stops it, and the thing that stops it should be a ceiling you chose rather than a full filesystem. - Get it: Decide the ceiling and the path, and put both in configuration. - Verify: Filling to the ceiling refuses new fetches - [ ] **Whether any extractor will be enabled, and on what basis** · free - Why: This decision changes what the tool is, and making it deliberately at the start is the difference between a library and a liability. - Get it: Write down the answer, and if it is yes, write the attestation statement you are prepared to sign. - Verify: The README and the configuration agree, and the attestation row exists if enabled ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A hosted service other people can use. - Anything that works around a protection measure. - A shipped, enabled extractor for any platform. - Loudness normalisation and audio repair. - hosted UI for non-technical users - playlist preview with per-track checkboxes - priority queue and 320 kbps Pro path - someone else fighting YouTube bot checks, PO tokens, and geo blocks - email magic-link Pro auth and fair multi-user queueing If one of those is essential to you, that is the reason to keep paying for AudioFetcher, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · AudioFetcher The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal audio fetcher that downloads what publishers offer for download, encodes it once, and refuses the rest to replace AudioFetcher. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This product downloads audio from somewhere else, so the platform terms and the copyright position are the design rather than a paragraph at the end of the README. YouTube's Terms of Service, currently effective 17 March 2025, forbid accessing, reproducing, downloading, distributing or otherwise using any part of the Service or Content except as expressly permitted by the Service, with prior written permission, or as permitted by applicable law. A second restriction forbids circumventing anything that prevents or restricts copying, and a third forbids automated access except by public search engines following robots.txt. In the United States, 17 U.S.C. 1201(a)(1)(A) says that no person shall circumvent a technological measure that effectively controls access to a work. And the private copying exception many people assume exists does not exist in the United Kingdom: section 28B of the 1988 Act was inserted in 2014, the regulations that inserted it were quashed by the High Court on 17 July 2015, and the Intellectual Property Office's own notice says those acts now constitute acts of infringement. So the default build is not a video ripper. It fetches what the publisher offers for download: podcast enclosures from a feed, direct file addresses, archives that publish downloadable media, and files you already hold. A site-specific extractor is optional and off by default, and turning it on records who attested what and when. Nothing here circumvents anything, and no checkbox here is pretending to be a licence. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with a small local web interface bound to the loopback address · this is one person's library and it should not be reachable from anywhere else - Fetching: Conditional requests with stored validators, one connection per host at a time · a feed poller that ignores caching headers is a poller that gets blocked - Encoding: ffmpeg, with stream copy preferred over transcoding wherever the format already matches · re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse - Storage: SQLite for rows, audio files on disk keyed by checksum · the checksum is the identity of a file and the filename is only a convenience - Extractors: An optional plugin interface, disabled unless configured, with an attestation record · the honest way to offer this is as a deliberate act with a name and a date attached ### Data model (create this before Phase 1) - sources(id, kind, url, added_at, etag, last_modified, licence_note, attestation_id) · kind is feed, direct, archive or extractor - items(id, source_id, remote_guid, title, published_at, media_url, bytes, mime, duration_ms, checksum, state) - fetches(id, item_id, attempt, http_status, bytes, started_at, ended_at, error_excerpt) · every attempt including the refused ones - files(item_id, path, format, bitrate_kbps, channels, sample_rate, transcoded_from, checksum) · transcoded_from records the chain, or is null when the stream was copied - attestations(id, made_at, who, statement, source_url) · a claim a person made with a date, which is a different and more honest thing than a permission - host_policy(host, robots_fetched_at, crawl_delay_seconds, max_concurrency, backoff_until) · one row per host you touch - No row here says permission granted. The nearest thing is a recorded claim, and the interface says so wherever it shows one. ### Phase 1 · Sources that publish downloads Build: Add a feed by address, parse its enclosures, and record each item with the publisher's own identifier. Store the entity tag and last-modified value from every response and send them back as conditional request headers on the next poll. Read the host's robots file before crawling it and record what it said. Never fetch the same media address twice for the same item. Trap: Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. Done when: A second poll of an unchanged feed sends the stored validators and transfers no body, verified against a local server that asserts the headers arrived. An item already fetched is never fetched again even when the feed reorders or republishes it, tested with a feed fixture that does both. A Retry-After header delays the next request by at least the stated interval, tested. The robots response for each host is recorded with the time it was read, and a disallowed path is not fetched. Do not build yet: No encoding, no extractor, no library interface yet. ### Phase 2 · Encode once, or not at all Build: Probe the source stream first. If the codec and bitrate already satisfy the request, copy the stream and write files.transcoded_from as null. Otherwise transcode once, with the parameters recorded, and store the chain. Write tags from the feed metadata including the source address and the fetch date. Keep the original file until an explicit purge. Trap: Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. Done when: A source already matching the requested format produces a file whose audio stream is bit-identical to the source, tested by comparing stream checksums. An upward bitrate request is refused with an explanation rather than performed, tested. Every transcoded file records its source format and the encoder parameters in transcoded_from, and no file has a null chain unless it was a stream copy, asserted in the suite. Tags include the source address and the fetch date and survive a round trip through a tag reader. Do not build yet: No extractor, no queue, no library or export yet. ### Phase 3 · The extractor boundary Build: Define a plugin interface and ship no extractor enabled by default. Enabling one requires configuration plus an attestation row naming the person, the statement and the date. Refuse to run an extractor against a host whose published terms the operator has not recorded reading. Never implement anything that descrambles, decrypts or otherwise bypasses a protection measure, whatever a library offers. Trap: An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. Done when: With no extractor configured, the interface offers only feeds, direct addresses and local files, and the extractor code path is unreachable, tested. Enabling an extractor without an attestation row fails with a message naming what is missing, tested. Every item fetched through an extractor displays its attestation, its date and who made it, checked in the interface test. No code path in the repository performs decryption or signature evasion of any kind, verified by review and stated in the README. Do not build yet: No queue policy, no library interface, no ceilings yet. ### Phase 4 · A queue that is polite to strangers Build: Queue fetches with at most the configured concurrency per host and a delay between requests taken from the host policy. Back off on a 429 or 503 and store backoff_until so a restart does not reset it. Refuse media longer than a configured ceiling and refuse anything reporting itself as a live stream. Show the queue with the reason each item is waiting. Trap: Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. Done when: Two queued items for the same host never download concurrently at the default configuration, tested against a server that fails on overlap. A 429 response sets backoff_until and a process restart preserves it, tested by restarting mid-backoff. Media exceeding the duration ceiling is refused before any bytes are transferred, tested using the feed's declared duration and a probe. A live stream is refused with a named reason rather than downloaded indefinitely, tested. Do not build yet: No disk ceilings, no export, no retention yet. ### Phase 5 · Disk, retention and an honest README Build: Enforce a total size ceiling and refuse new fetches when it is reached rather than deleting to make room. Add retention by source with an explicit keep list. Export the whole library as a manifest of checksums, titles, sources, attestations and file paths, so the audio and the record can be moved together. Write the README so a stranger can tell in thirty seconds what is in scope and what is not. Trap: A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all. Done when: Reaching the size ceiling refuses new fetches with a named reason and deletes nothing, tested by filling to the boundary. Retention deletes only sources with an explicit policy, and an item on the keep list survives a retention run, tested. The exported manifest reimports into an empty database and every file resolves by checksum, tested. The README states the terms position, the copyright position and the extractor default in its first section, checked by review. Do not build yet: Nothing after this. ### Out of scope (and why) - A hosted service other people can paste addresses into, because you would then be operating the thing the terms above are written about, at your own risk and on your own domain. - Anything that descrambles, decrypts or works around a protection measure, which is a separate prohibition from the terms and is not negotiable here. - Shipping an enabled extractor for any platform, since the entire attestation design exists because that choice belongs to the operator and not to the author. - Loudness normalisation and audio repair, which are a genuinely interesting project and have nothing to do with the problems this build is about. ### README must contain - The platform terms position and the copyright position, stated in the first section with the sources and the dates they were read - That the private copying exception people assume exists was quashed in the United Kingdom in 2015 and that the position differs by jurisdiction - That no extractor is enabled by default, what enabling one requires, and that an attestation is a recorded claim rather than a permission - That audio already in the requested format is copied rather than re-encoded, and why an upward bitrate request is refused - Where the files live, what the size ceiling does when reached, and how to export the manifest with the audio ===== AGENTS.md ===== # Agent instructions · AudioFetcher indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with a small local web interface bound to the loopback address, Conditional requests with stored validators, one connection per host at a time, ffmpeg, with stream copy preferred over transcoding wherever the format already matches, SQLite for rows, audio files on disk keyed by checksum, An optional plugin interface, disabled unless configured, with an attestation record. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. ===== BUILD_PLAN.md ===== # Build plan · AudioFetcher A personal audio library whose hard problems are fetching politely and only once from sources that publish downloads, refusing to re-encode audio that is already in the format you asked for, and keeping an honest record of the permission claim behind anything fetched through the optional extractor. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sources that publish downloads Feeds and direct addresses are fetched once each, politely, with validators stored so a poll is usually free. ### Steps 1. Store validators and send them back A not-modified response should be the common case. Files: `src/fetch/conditional.py`, `tests/test_conditional.py` 2. Key items on the publisher's identifier Feeds reorder, republish and change titles. Files: `src/feeds/parse.py`, `tests/test_feed_dedupe.py` 3. Read robots and honour Retry-After Both are instructions, and ignoring them is how access ends. Files: `src/fetch/policy.py`, `tests/test_retry_after.py` ### Done when - [ ] An unchanged feed transfers no body on the second poll - [ ] A reordered or republished item is not refetched - [ ] Retry-After delays the next request - [ ] Robots responses are recorded and honoured ### Watch out - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. ## Phase 2 · Encode once, or not at all Audio already in the requested format is copied rather than re-encoded, and any transcode records what it came from. ### Steps 1. Probe before deciding to encode Most fetches do not need an encoder at all. Files: `src/encode/probe.py`, `tests/test_streamcopy.py` 2. Refuse an upward bitrate request There is nothing to put back and the file gets worse. Files: `src/encode/plan.py` 3. Record the chain and tag the source Six months later this is the only provenance you have. Files: `src/encode/run.py`, `src/encode/tags.py` ### Done when - [ ] A matching source is stream-copied bit-identically - [ ] An upward bitrate request is refused with a reason - [ ] Every transcode records its chain and parameters - [ ] Tags carry the source address and fetch date ### Watch out - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. ## Phase 3 · The extractor boundary Site-specific extraction is off unless someone deliberately turns it on and records why. ### Steps 1. Ship with no extractor enabled The default build is the one most people will run. Files: `src/extractors/__init__.py`, `tests/test_default_disabled.py` 2. Require a stored attestation, not a checkbox A claim with a name and a date is the only honest artefact here. Files: `src/extractors/attest.py`, `migrations/003_attestations.sql` 3. Refuse anything that bypasses a protection measure This is a line, not a configuration option. Files: `docs/scope.md` ### Done when - [ ] The default build exposes no extractor - [ ] Enabling without an attestation fails with a reason - [ ] Extractor items display their attestation - [ ] No circumvention code exists in the repository ### Watch out - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. ## Phase 4 · A queue that is polite to strangers Downloads run one host at a time with recorded backoff, and a long or live source is refused before it starts. ### Steps 1. Limit concurrency per host, not globally One slow publisher should not be hammered by your whole queue. Files: `src/queue/claim.py`, `tests/test_host_concurrency.py` 2. Persist backoff across restarts In-memory backoff resets exactly when you least want it to. Files: `src/queue/backoff.py`, `tests/test_backoff_persist.py` 3. Refuse live and over-long media up front Before bytes, not after two hours of them. Files: `src/queue/guard.py` ### Done when - [ ] One host is never downloaded from concurrently by default - [ ] Backoff survives a restart - [ ] Over-long media is refused before transfer - [ ] Live streams are refused with a reason ### Watch out - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. ## Not in this build - A hosted service other people can use. - Anything that works around a protection measure. - A shipped, enabled extractor for any platform. - Loudness normalisation and audio repair. ## After v1, if you want it - Chapter and transcript handling for feeds that publish them, which is metadata the publisher already offers - A local search over titles and descriptions, which costs nothing once the rows are already there - A player that reads the manifest directly, so the library works even if this application stops running ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Audio files, keyed by checksum. The largest thing on the disk. LIBRARY_DIR=./library # Required. Sources, items, fetches and attestations. Back this up; the audio is refetchable and the record is not. DATABASE_PATH=./data/library.db # Required. New fetches are refused at this size. Nothing is ever evicted to make room. LIBRARY_CEILING_GB=50 # Required. Media longer than this is refused before any bytes move. MAX_DURATION_MINUTES=240 # Required. Simultaneous downloads per host. Raise it only with a reason. HOST_CONCURRENCY=1 # Required. Off unless deliberately set, and setting it needs an attestation row to exist. EXTRACTORS_ENABLED=false # Required. Identifies you honestly to the hosts you fetch from, with a way to reach you. USER_AGENT=my-audio-library/1.0 (contact: me@example.com)
You are building a production product version of AudioFetcher. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # AudioFetcher · product brief ## Problem The core loop is genuinely one-shottable: a small local UI, yt-dlp for the fetch, ffmpeg for the encode, files land in a folder. What is not a one sitting is everything that makes a public converter feel reliable week after week · YouTube bot checks, cookies/PO tokens, geo blocks, fair queues, abuse protection, and a hosted path so non-technical people never open a terminal. Build the personal script; do not expect one prompt to replace a maintained public service. ## Product outcome A library that fetches only what publishers offer for download, transfers each thing once, copies the stream when no encoding is needed, records a dated claim behind anything fetched through the optional extractor, and stops rather than evicting when the disk ceiling is reached. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Python or Node - yt-dlp - ffmpeg - optional cookies or residential proxy if YouTube blocks datacenter IPs ## Explicit non-goals for v1 - A hosted service other people can use. - Anything that works around a protection measure. - A shipped, enabled extractor for any platform. - Loudness normalisation and audio repair. - hosted UI for non-technical users - playlist preview with per-track checkboxes - priority queue and 320 kbps Pro path - someone else fighting YouTube bot checks, PO tokens, and geo blocks - email magic-link Pro auth and fair multi-user queueing ## Success criteria - An unchanged feed is confirmed to transfer no body on a second poll - A matching source is confirmed stream-copied rather than re-encoded - The default build is confirmed to expose no extractor - Persisted backoff is confirmed to survive a restart - The size ceiling is confirmed to refuse fetches and delete nothing ===== BRIEF.md ===== # Build brief · AudioFetcher The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal audio fetcher that downloads what publishers offer for download, encodes it once, and refuses the rest to replace AudioFetcher. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This product downloads audio from somewhere else, so the platform terms and the copyright position are the design rather than a paragraph at the end of the README. YouTube's Terms of Service, currently effective 17 March 2025, forbid accessing, reproducing, downloading, distributing or otherwise using any part of the Service or Content except as expressly permitted by the Service, with prior written permission, or as permitted by applicable law. A second restriction forbids circumventing anything that prevents or restricts copying, and a third forbids automated access except by public search engines following robots.txt. In the United States, 17 U.S.C. 1201(a)(1)(A) says that no person shall circumvent a technological measure that effectively controls access to a work. And the private copying exception many people assume exists does not exist in the United Kingdom: section 28B of the 1988 Act was inserted in 2014, the regulations that inserted it were quashed by the High Court on 17 July 2015, and the Intellectual Property Office's own notice says those acts now constitute acts of infringement. So the default build is not a video ripper. It fetches what the publisher offers for download: podcast enclosures from a feed, direct file addresses, archives that publish downloadable media, and files you already hold. A site-specific extractor is optional and off by default, and turning it on records who attested what and when. Nothing here circumvents anything, and no checkbox here is pretending to be a licence. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with a small local web interface bound to the loopback address · this is one person's library and it should not be reachable from anywhere else - Fetching: Conditional requests with stored validators, one connection per host at a time · a feed poller that ignores caching headers is a poller that gets blocked - Encoding: ffmpeg, with stream copy preferred over transcoding wherever the format already matches · re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse - Storage: SQLite for rows, audio files on disk keyed by checksum · the checksum is the identity of a file and the filename is only a convenience - Extractors: An optional plugin interface, disabled unless configured, with an attestation record · the honest way to offer this is as a deliberate act with a name and a date attached ### Data model (create this before Phase 1) - sources(id, kind, url, added_at, etag, last_modified, licence_note, attestation_id) · kind is feed, direct, archive or extractor - items(id, source_id, remote_guid, title, published_at, media_url, bytes, mime, duration_ms, checksum, state) - fetches(id, item_id, attempt, http_status, bytes, started_at, ended_at, error_excerpt) · every attempt including the refused ones - files(item_id, path, format, bitrate_kbps, channels, sample_rate, transcoded_from, checksum) · transcoded_from records the chain, or is null when the stream was copied - attestations(id, made_at, who, statement, source_url) · a claim a person made with a date, which is a different and more honest thing than a permission - host_policy(host, robots_fetched_at, crawl_delay_seconds, max_concurrency, backoff_until) · one row per host you touch - No row here says permission granted. The nearest thing is a recorded claim, and the interface says so wherever it shows one. ### Phase 1 · Sources that publish downloads Build: Add a feed by address, parse its enclosures, and record each item with the publisher's own identifier. Store the entity tag and last-modified value from every response and send them back as conditional request headers on the next poll. Read the host's robots file before crawling it and record what it said. Never fetch the same media address twice for the same item. Trap: Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. Done when: A second poll of an unchanged feed sends the stored validators and transfers no body, verified against a local server that asserts the headers arrived. An item already fetched is never fetched again even when the feed reorders or republishes it, tested with a feed fixture that does both. A Retry-After header delays the next request by at least the stated interval, tested. The robots response for each host is recorded with the time it was read, and a disallowed path is not fetched. Do not build yet: No encoding, no extractor, no library interface yet. ### Phase 2 · Encode once, or not at all Build: Probe the source stream first. If the codec and bitrate already satisfy the request, copy the stream and write files.transcoded_from as null. Otherwise transcode once, with the parameters recorded, and store the chain. Write tags from the feed metadata including the source address and the fetch date. Keep the original file until an explicit purge. Trap: Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. Done when: A source already matching the requested format produces a file whose audio stream is bit-identical to the source, tested by comparing stream checksums. An upward bitrate request is refused with an explanation rather than performed, tested. Every transcoded file records its source format and the encoder parameters in transcoded_from, and no file has a null chain unless it was a stream copy, asserted in the suite. Tags include the source address and the fetch date and survive a round trip through a tag reader. Do not build yet: No extractor, no queue, no library or export yet. ### Phase 3 · The extractor boundary Build: Define a plugin interface and ship no extractor enabled by default. Enabling one requires configuration plus an attestation row naming the person, the statement and the date. Refuse to run an extractor against a host whose published terms the operator has not recorded reading. Never implement anything that descrambles, decrypts or otherwise bypasses a protection measure, whatever a library offers. Trap: An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. Done when: With no extractor configured, the interface offers only feeds, direct addresses and local files, and the extractor code path is unreachable, tested. Enabling an extractor without an attestation row fails with a message naming what is missing, tested. Every item fetched through an extractor displays its attestation, its date and who made it, checked in the interface test. No code path in the repository performs decryption or signature evasion of any kind, verified by review and stated in the README. Do not build yet: No queue policy, no library interface, no ceilings yet. ### Phase 4 · A queue that is polite to strangers Build: Queue fetches with at most the configured concurrency per host and a delay between requests taken from the host policy. Back off on a 429 or 503 and store backoff_until so a restart does not reset it. Refuse media longer than a configured ceiling and refuse anything reporting itself as a live stream. Show the queue with the reason each item is waiting. Trap: Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. Done when: Two queued items for the same host never download concurrently at the default configuration, tested against a server that fails on overlap. A 429 response sets backoff_until and a process restart preserves it, tested by restarting mid-backoff. Media exceeding the duration ceiling is refused before any bytes are transferred, tested using the feed's declared duration and a probe. A live stream is refused with a named reason rather than downloaded indefinitely, tested. Do not build yet: No disk ceilings, no export, no retention yet. ### Phase 5 · Disk, retention and an honest README Build: Enforce a total size ceiling and refuse new fetches when it is reached rather than deleting to make room. Add retention by source with an explicit keep list. Export the whole library as a manifest of checksums, titles, sources, attestations and file paths, so the audio and the record can be moved together. Write the README so a stranger can tell in thirty seconds what is in scope and what is not. Trap: A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all. Done when: Reaching the size ceiling refuses new fetches with a named reason and deletes nothing, tested by filling to the boundary. Retention deletes only sources with an explicit policy, and an item on the keep list survives a retention run, tested. The exported manifest reimports into an empty database and every file resolves by checksum, tested. The README states the terms position, the copyright position and the extractor default in its first section, checked by review. Do not build yet: Nothing after this. ### Out of scope (and why) - A hosted service other people can paste addresses into, because you would then be operating the thing the terms above are written about, at your own risk and on your own domain. - Anything that descrambles, decrypts or works around a protection measure, which is a separate prohibition from the terms and is not negotiable here. - Shipping an enabled extractor for any platform, since the entire attestation design exists because that choice belongs to the operator and not to the author. - Loudness normalisation and audio repair, which are a genuinely interesting project and have nothing to do with the problems this build is about. ### README must contain - The platform terms position and the copyright position, stated in the first section with the sources and the dates they were read - That the private copying exception people assume exists was quashed in the United Kingdom in 2015 and that the position differs by jurisdiction - That no extractor is enabled by default, what enabling one requires, and that an attestation is a recorded claim rather than a permission - That audio already in the requested format is copied rather than re-encoded, and why an upward bitrate request is refused - Where the files live, what the size ceiling does when reached, and how to export the manifest with the audio ===== ARCHITECTURE.md ===== # Architecture · AudioFetcher ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with a small local web interface bound to the loopback address | this is one person's library and it should not be reachable from anywhere else | | Fetching | Conditional requests with stored validators, one connection per host at a time | a feed poller that ignores caching headers is a poller that gets blocked | | Encoding | ffmpeg, with stream copy preferred over transcoding wherever the format already matches | re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse | | Storage | SQLite for rows, audio files on disk keyed by checksum | the checksum is the identity of a file and the filename is only a convenience | | Extractors | An optional plugin interface, disabled unless configured, with an attestation record | the honest way to offer this is as a deliberate act with a name and a date attached | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Sources | Feeds, direct addresses and validators. | Any parser, provided conditional requests and publisher identifiers are kept. | | Queue | Per-host concurrency, persisted backoff and guards. | Any runner, provided backoff survives a restart. | | Encode | Probing, stream copy and recorded transcode chains. | Any encoder, provided a matching source is copied rather than re-encoded. | | Extractors | The optional plugin boundary and attestations. | Not swappable for an always-on extractor, which changes what the tool is. | | Library | Checksums, ceiling, retention and the manifest. | Any store, provided the manifest exports with the attestations attached. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `LIBRARY_DIR` · required · Audio files, keyed by checksum. The largest thing on the disk. - `DATABASE_PATH` · required · Sources, items, fetches and attestations. Back this up; the audio is refetchable and the record is not. - `LIBRARY_CEILING_GB` · required · New fetches are refused at this size. Nothing is ever evicted to make room. - `MAX_DURATION_MINUTES` · required · Media longer than this is refused before any bytes move. - `HOST_CONCURRENCY` · required · Simultaneous downloads per host. Raise it only with a reason. - `EXTRACTORS_ENABLED` · required · Off unless deliberately set, and setting it needs an attestation row to exist. - `USER_AGENT` · required · Identifies you honestly to the hosts you fetch from, with a way to reach you. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · AudioFetcher product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with a small local web interface bound to the loopback address, Conditional requests with stored validators, one connection per host at a time, ffmpeg, with stream copy preferred over transcoding wherever the format already matches, SQLite for rows, audio files on disk keyed by checksum, An optional plugin interface, disabled unless configured, with an attestation record. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. - A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all. ===== MILESTONES.md ===== # Delivery milestones · AudioFetcher Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Sources that publish downloads Feeds and direct addresses are fetched once each, politely, with validators stored so a poll is usually free. ### Steps 1. Store validators and send them back A not-modified response should be the common case. Files: `src/fetch/conditional.py`, `tests/test_conditional.py` 2. Key items on the publisher's identifier Feeds reorder, republish and change titles. Files: `src/feeds/parse.py`, `tests/test_feed_dedupe.py` 3. Read robots and honour Retry-After Both are instructions, and ignoring them is how access ends. Files: `src/fetch/policy.py`, `tests/test_retry_after.py` ### Done when - [ ] An unchanged feed transfers no body on the second poll - [ ] A reordered or republished item is not refetched - [ ] Retry-After delays the next request - [ ] Robots responses are recorded and honoured ### Watch out - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. ## M2 · Encode once, or not at all Audio already in the requested format is copied rather than re-encoded, and any transcode records what it came from. ### Steps 1. Probe before deciding to encode Most fetches do not need an encoder at all. Files: `src/encode/probe.py`, `tests/test_streamcopy.py` 2. Refuse an upward bitrate request There is nothing to put back and the file gets worse. Files: `src/encode/plan.py` 3. Record the chain and tag the source Six months later this is the only provenance you have. Files: `src/encode/run.py`, `src/encode/tags.py` ### Done when - [ ] A matching source is stream-copied bit-identically - [ ] An upward bitrate request is refused with a reason - [ ] Every transcode records its chain and parameters - [ ] Tags carry the source address and fetch date ### Watch out - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. ## M3 · The extractor boundary Site-specific extraction is off unless someone deliberately turns it on and records why. ### Steps 1. Ship with no extractor enabled The default build is the one most people will run. Files: `src/extractors/__init__.py`, `tests/test_default_disabled.py` 2. Require a stored attestation, not a checkbox A claim with a name and a date is the only honest artefact here. Files: `src/extractors/attest.py`, `migrations/003_attestations.sql` 3. Refuse anything that bypasses a protection measure This is a line, not a configuration option. Files: `docs/scope.md` ### Done when - [ ] The default build exposes no extractor - [ ] Enabling without an attestation fails with a reason - [ ] Extractor items display their attestation - [ ] No circumvention code exists in the repository ### Watch out - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. ## M4 · A queue that is polite to strangers Downloads run one host at a time with recorded backoff, and a long or live source is refused before it starts. ### Steps 1. Limit concurrency per host, not globally One slow publisher should not be hammered by your whole queue. Files: `src/queue/claim.py`, `tests/test_host_concurrency.py` 2. Persist backoff across restarts In-memory backoff resets exactly when you least want it to. Files: `src/queue/backoff.py`, `tests/test_backoff_persist.py` 3. Refuse live and over-long media up front Before bytes, not after two hours of them. Files: `src/queue/guard.py` ### Done when - [ ] One host is never downloaded from concurrently by default - [ ] Backoff survives a restart - [ ] Over-long media is refused before transfer - [ ] Live streams are refused with a reason ### Watch out - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. ## M5 · Disk, retention and an honest README (production only) The library cannot fill the disk, everything exports, and the documentation says plainly what this tool will and will not do. ### Steps 1. Refuse at the ceiling, never evict The oldest file is often the one that no longer exists upstream. Files: `src/storage/ceiling.py`, `tests/test_ceiling.py` 2. Retention by explicit policy only Silence should mean keep. Files: `src/storage/retention.py` 3. Export a manifest that includes the attestations The record travels with the audio or it is not a record. Files: `src/export/manifest.py`, `tests/test_manifest.py` ### Done when - [ ] The size ceiling refuses fetches and deletes nothing - [ ] Retention touches only sources with a policy - [ ] The manifest reimports and every file resolves - [ ] The README leads with scope, not with features ### Watch out - A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all. ===== OPERATIONS.md ===== # Operations · AudioFetcher ## Backup The database and the manifest first, the audio second. Most audio can be fetched again from its publisher; the record of what you fetched, when, and on what stated basis cannot be reconstructed at all. ## Restore Restore the database, then verify every file resolves by checksum and report the missing ones rather than silently refetching them. A file missing after a restore may have been removed upstream, and quietly refetching hides that. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Not-modified rate per feed, backoff events per host, the library size against its ceiling, and the count of items whose attestation was removed. A falling not-modified rate usually means a publisher stopped sending validators, which is worth noticing before they notice you. ## Incident checklist The incident here is a fetch you cannot account for. The fetches table, the host policy rows and the attestations are what make the answer a record rather than a recollection, which is the entire reason they are separate tables. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] An unchanged feed is confirmed to transfer no body on a second poll - [ ] A matching source is confirmed stream-copied rather than re-encoded - [ ] The default build is confirmed to expose no extractor - [ ] Persisted backoff is confirmed to survive a restart - [ ] The size ceiling is confirmed to refuse fetches and delete nothing ## Launch constraint Do not market omitted AudioFetcher capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Audio files, keyed by checksum. The largest thing on the disk. LIBRARY_DIR=./library # Required. Sources, items, fetches and attestations. Back this up; the audio is refetchable and the record is not. DATABASE_PATH=./data/library.db # Required. New fetches are refused at this size. Nothing is ever evicted to make room. LIBRARY_CEILING_GB=50 # Required. Media longer than this is refused before any bytes move. MAX_DURATION_MINUTES=240 # Required. Simultaneous downloads per host. Raise it only with a reason. HOST_CONCURRENCY=1 # Required. Off unless deliberately set, and setting it needs an attestation row to exist. EXTRACTORS_ENABLED=false # Required. Identifies you honestly to the hosts you fetch from, with a way to reach you. USER_AGENT=my-audio-library/1.0 (contact: me@example.com)
README.md
# AudioFetcher · indie build A personal audio library whose hard problems are fetching politely and only once from sources that publish downloads, refusing to re-encode audio that is already in the format you asked for, and keeping an honest record of the permission claim behind anything fetched through the optional extractor. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with a small local web interface bound to the loopback address | this is one person's library and it should not be reachable from anywhere else | | Fetching | Conditional requests with stored validators, one connection per host at a time | a feed poller that ignores caching headers is a poller that gets blocked | | Encoding | ffmpeg, with stream copy preferred over transcoding wherever the format already matches | re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse | | Storage | SQLite for rows, audio files on disk keyed by checksum | the checksum is the identity of a file and the filename is only a convenience | | Extractors | An optional plugin interface, disabled unless configured, with an attestation record | the honest way to offer this is as a deliberate act with a name and a date attached | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the local interface and the fetch queue. - Get it: python.org or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **ffmpeg with ffprobe** · free - Why: Probing before encoding is what makes the stream-copy path possible, and ffprobe is how you know whether to encode at all. - Get it: Your package manager. - Verify: ffprobe -version prints a version and ffmpeg -codecs lists an MP3 encoder - [ ] **The published terms of every host you intend to fetch from, read and recorded** · free - Why: Rule zero rests on what those documents say today, and they change, so reading them is part of setup rather than part of the argument afterwards. - Get it: Read each host's terms and record the URL and the date in your configuration. - Verify: Every configured host has a terms URL and a date recorded - [ ] **Disk space with a number you have decided in advance** · free - Why: An audio library grows until something stops it, and the thing that stops it should be a ceiling you chose rather than a full filesystem. - Get it: Decide the ceiling and the path, and put both in configuration. - Verify: Filling to the ceiling refuses new fetches - [ ] **Whether any extractor will be enabled, and on what basis** · free - Why: This decision changes what the tool is, and making it deliberately at the start is the difference between a library and a liability. - Get it: Write down the answer, and if it is yes, write the attestation statement you are prepared to sign. - Verify: The README and the configuration agree, and the attestation row exists if enabled ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A hosted service other people can use. - Anything that works around a protection measure. - A shipped, enabled extractor for any platform. - Loudness normalisation and audio repair. - hosted UI for non-technical users - playlist preview with per-track checkboxes - priority queue and 320 kbps Pro path - someone else fighting YouTube bot checks, PO tokens, and geo blocks - email magic-link Pro auth and fair multi-user queueing If one of those is essential to you, that is the reason to keep paying for AudioFetcher, and the README should say so rather than pretend.
BRIEF.md
# Build brief · AudioFetcher The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal audio fetcher that downloads what publishers offer for download, encodes it once, and refuses the rest to replace AudioFetcher. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This product downloads audio from somewhere else, so the platform terms and the copyright position are the design rather than a paragraph at the end of the README. YouTube's Terms of Service, currently effective 17 March 2025, forbid accessing, reproducing, downloading, distributing or otherwise using any part of the Service or Content except as expressly permitted by the Service, with prior written permission, or as permitted by applicable law. A second restriction forbids circumventing anything that prevents or restricts copying, and a third forbids automated access except by public search engines following robots.txt. In the United States, 17 U.S.C. 1201(a)(1)(A) says that no person shall circumvent a technological measure that effectively controls access to a work. And the private copying exception many people assume exists does not exist in the United Kingdom: section 28B of the 1988 Act was inserted in 2014, the regulations that inserted it were quashed by the High Court on 17 July 2015, and the Intellectual Property Office's own notice says those acts now constitute acts of infringement. So the default build is not a video ripper. It fetches what the publisher offers for download: podcast enclosures from a feed, direct file addresses, archives that publish downloadable media, and files you already hold. A site-specific extractor is optional and off by default, and turning it on records who attested what and when. Nothing here circumvents anything, and no checkbox here is pretending to be a licence. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with a small local web interface bound to the loopback address · this is one person's library and it should not be reachable from anywhere else - Fetching: Conditional requests with stored validators, one connection per host at a time · a feed poller that ignores caching headers is a poller that gets blocked - Encoding: ffmpeg, with stream copy preferred over transcoding wherever the format already matches · re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse - Storage: SQLite for rows, audio files on disk keyed by checksum · the checksum is the identity of a file and the filename is only a convenience - Extractors: An optional plugin interface, disabled unless configured, with an attestation record · the honest way to offer this is as a deliberate act with a name and a date attached ### Data model (create this before Phase 1) - sources(id, kind, url, added_at, etag, last_modified, licence_note, attestation_id) · kind is feed, direct, archive or extractor - items(id, source_id, remote_guid, title, published_at, media_url, bytes, mime, duration_ms, checksum, state) - fetches(id, item_id, attempt, http_status, bytes, started_at, ended_at, error_excerpt) · every attempt including the refused ones - files(item_id, path, format, bitrate_kbps, channels, sample_rate, transcoded_from, checksum) · transcoded_from records the chain, or is null when the stream was copied - attestations(id, made_at, who, statement, source_url) · a claim a person made with a date, which is a different and more honest thing than a permission - host_policy(host, robots_fetched_at, crawl_delay_seconds, max_concurrency, backoff_until) · one row per host you touch - No row here says permission granted. The nearest thing is a recorded claim, and the interface says so wherever it shows one. ### Phase 1 · Sources that publish downloads Build: Add a feed by address, parse its enclosures, and record each item with the publisher's own identifier. Store the entity tag and last-modified value from every response and send them back as conditional request headers on the next poll. Read the host's robots file before crawling it and record what it said. Never fetch the same media address twice for the same item. Trap: Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. Done when: A second poll of an unchanged feed sends the stored validators and transfers no body, verified against a local server that asserts the headers arrived. An item already fetched is never fetched again even when the feed reorders or republishes it, tested with a feed fixture that does both. A Retry-After header delays the next request by at least the stated interval, tested. The robots response for each host is recorded with the time it was read, and a disallowed path is not fetched. Do not build yet: No encoding, no extractor, no library interface yet. ### Phase 2 · Encode once, or not at all Build: Probe the source stream first. If the codec and bitrate already satisfy the request, copy the stream and write files.transcoded_from as null. Otherwise transcode once, with the parameters recorded, and store the chain. Write tags from the feed metadata including the source address and the fetch date. Keep the original file until an explicit purge. Trap: Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. Done when: A source already matching the requested format produces a file whose audio stream is bit-identical to the source, tested by comparing stream checksums. An upward bitrate request is refused with an explanation rather than performed, tested. Every transcoded file records its source format and the encoder parameters in transcoded_from, and no file has a null chain unless it was a stream copy, asserted in the suite. Tags include the source address and the fetch date and survive a round trip through a tag reader. Do not build yet: No extractor, no queue, no library or export yet. ### Phase 3 · The extractor boundary Build: Define a plugin interface and ship no extractor enabled by default. Enabling one requires configuration plus an attestation row naming the person, the statement and the date. Refuse to run an extractor against a host whose published terms the operator has not recorded reading. Never implement anything that descrambles, decrypts or otherwise bypasses a protection measure, whatever a library offers. Trap: An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. Done when: With no extractor configured, the interface offers only feeds, direct addresses and local files, and the extractor code path is unreachable, tested. Enabling an extractor without an attestation row fails with a message naming what is missing, tested. Every item fetched through an extractor displays its attestation, its date and who made it, checked in the interface test. No code path in the repository performs decryption or signature evasion of any kind, verified by review and stated in the README. Do not build yet: No queue policy, no library interface, no ceilings yet. ### Phase 4 · A queue that is polite to strangers Build: Queue fetches with at most the configured concurrency per host and a delay between requests taken from the host policy. Back off on a 429 or 503 and store backoff_until so a restart does not reset it. Refuse media longer than a configured ceiling and refuse anything reporting itself as a live stream. Show the queue with the reason each item is waiting. Trap: Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. Done when: Two queued items for the same host never download concurrently at the default configuration, tested against a server that fails on overlap. A 429 response sets backoff_until and a process restart preserves it, tested by restarting mid-backoff. Media exceeding the duration ceiling is refused before any bytes are transferred, tested using the feed's declared duration and a probe. A live stream is refused with a named reason rather than downloaded indefinitely, tested. Do not build yet: No disk ceilings, no export, no retention yet. ### Phase 5 · Disk, retention and an honest README Build: Enforce a total size ceiling and refuse new fetches when it is reached rather than deleting to make room. Add retention by source with an explicit keep list. Export the whole library as a manifest of checksums, titles, sources, attestations and file paths, so the audio and the record can be moved together. Write the README so a stranger can tell in thirty seconds what is in scope and what is not. Trap: A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all. Done when: Reaching the size ceiling refuses new fetches with a named reason and deletes nothing, tested by filling to the boundary. Retention deletes only sources with an explicit policy, and an item on the keep list survives a retention run, tested. The exported manifest reimports into an empty database and every file resolves by checksum, tested. The README states the terms position, the copyright position and the extractor default in its first section, checked by review. Do not build yet: Nothing after this. ### Out of scope (and why) - A hosted service other people can paste addresses into, because you would then be operating the thing the terms above are written about, at your own risk and on your own domain. - Anything that descrambles, decrypts or works around a protection measure, which is a separate prohibition from the terms and is not negotiable here. - Shipping an enabled extractor for any platform, since the entire attestation design exists because that choice belongs to the operator and not to the author. - Loudness normalisation and audio repair, which are a genuinely interesting project and have nothing to do with the problems this build is about. ### README must contain - The platform terms position and the copyright position, stated in the first section with the sources and the dates they were read - That the private copying exception people assume exists was quashed in the United Kingdom in 2015 and that the position differs by jurisdiction - That no extractor is enabled by default, what enabling one requires, and that an attestation is a recorded claim rather than a permission - That audio already in the requested format is copied rather than re-encoded, and why an upward bitrate request is refused - Where the files live, what the size ceiling does when reached, and how to export the manifest with the audio
AGENTS.md
# Agent instructions · AudioFetcher indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with a small local web interface bound to the loopback address, Conditional requests with stored validators, one connection per host at a time, ffmpeg, with stream copy preferred over transcoding wherever the format already matches, SQLite for rows, audio files on disk keyed by checksum, An optional plugin interface, disabled unless configured, with an attestation record. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it.
BUILD_PLAN.md
# Build plan · AudioFetcher A personal audio library whose hard problems are fetching politely and only once from sources that publish downloads, refusing to re-encode audio that is already in the format you asked for, and keeping an honest record of the permission claim behind anything fetched through the optional extractor. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Sources that publish downloads Feeds and direct addresses are fetched once each, politely, with validators stored so a poll is usually free. ### Steps 1. Store validators and send them back A not-modified response should be the common case. Files: `src/fetch/conditional.py`, `tests/test_conditional.py` 2. Key items on the publisher's identifier Feeds reorder, republish and change titles. Files: `src/feeds/parse.py`, `tests/test_feed_dedupe.py` 3. Read robots and honour Retry-After Both are instructions, and ignoring them is how access ends. Files: `src/fetch/policy.py`, `tests/test_retry_after.py` ### Done when - [ ] An unchanged feed transfers no body on the second poll - [ ] A reordered or republished item is not refetched - [ ] Retry-After delays the next request - [ ] Robots responses are recorded and honoured ### Watch out - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. ## Phase 2 · Encode once, or not at all Audio already in the requested format is copied rather than re-encoded, and any transcode records what it came from. ### Steps 1. Probe before deciding to encode Most fetches do not need an encoder at all. Files: `src/encode/probe.py`, `tests/test_streamcopy.py` 2. Refuse an upward bitrate request There is nothing to put back and the file gets worse. Files: `src/encode/plan.py` 3. Record the chain and tag the source Six months later this is the only provenance you have. Files: `src/encode/run.py`, `src/encode/tags.py` ### Done when - [ ] A matching source is stream-copied bit-identically - [ ] An upward bitrate request is refused with a reason - [ ] Every transcode records its chain and parameters - [ ] Tags carry the source address and fetch date ### Watch out - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. ## Phase 3 · The extractor boundary Site-specific extraction is off unless someone deliberately turns it on and records why. ### Steps 1. Ship with no extractor enabled The default build is the one most people will run. Files: `src/extractors/__init__.py`, `tests/test_default_disabled.py` 2. Require a stored attestation, not a checkbox A claim with a name and a date is the only honest artefact here. Files: `src/extractors/attest.py`, `migrations/003_attestations.sql` 3. Refuse anything that bypasses a protection measure This is a line, not a configuration option. Files: `docs/scope.md` ### Done when - [ ] The default build exposes no extractor - [ ] Enabling without an attestation fails with a reason - [ ] Extractor items display their attestation - [ ] No circumvention code exists in the repository ### Watch out - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. ## Phase 4 · A queue that is polite to strangers Downloads run one host at a time with recorded backoff, and a long or live source is refused before it starts. ### Steps 1. Limit concurrency per host, not globally One slow publisher should not be hammered by your whole queue. Files: `src/queue/claim.py`, `tests/test_host_concurrency.py` 2. Persist backoff across restarts In-memory backoff resets exactly when you least want it to. Files: `src/queue/backoff.py`, `tests/test_backoff_persist.py` 3. Refuse live and over-long media up front Before bytes, not after two hours of them. Files: `src/queue/guard.py` ### Done when - [ ] One host is never downloaded from concurrently by default - [ ] Backoff survives a restart - [ ] Over-long media is refused before transfer - [ ] Live streams are refused with a reason ### Watch out - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. ## Not in this build - A hosted service other people can use. - Anything that works around a protection measure. - A shipped, enabled extractor for any platform. - Loudness normalisation and audio repair. ## After v1, if you want it - Chapter and transcript handling for feeds that publish them, which is metadata the publisher already offers - A local search over titles and descriptions, which costs nothing once the rows are already there - A player that reads the manifest directly, so the library works even if this application stops running
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Audio files, keyed by checksum. The largest thing on the disk. LIBRARY_DIR=./library # Required. Sources, items, fetches and attestations. Back this up; the audio is refetchable and the record is not. DATABASE_PATH=./data/library.db # Required. New fetches are refused at this size. Nothing is ever evicted to make room. LIBRARY_CEILING_GB=50 # Required. Media longer than this is refused before any bytes move. MAX_DURATION_MINUTES=240 # Required. Simultaneous downloads per host. Raise it only with a reason. HOST_CONCURRENCY=1 # Required. Off unless deliberately set, and setting it needs an attestation row to exist. EXTRACTORS_ENABLED=false # Required. Identifies you honestly to the hosts you fetch from, with a way to reach you. USER_AGENT=my-audio-library/1.0 (contact: me@example.com)
PRODUCT.md
# AudioFetcher · product brief ## Problem The core loop is genuinely one-shottable: a small local UI, yt-dlp for the fetch, ffmpeg for the encode, files land in a folder. What is not a one sitting is everything that makes a public converter feel reliable week after week · YouTube bot checks, cookies/PO tokens, geo blocks, fair queues, abuse protection, and a hosted path so non-technical people never open a terminal. Build the personal script; do not expect one prompt to replace a maintained public service. ## Product outcome A library that fetches only what publishers offer for download, transfers each thing once, copies the stream when no encoding is needed, records a dated claim behind anything fetched through the optional extractor, and stops rather than evicting when the disk ceiling is reached. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Python or Node - yt-dlp - ffmpeg - optional cookies or residential proxy if YouTube blocks datacenter IPs ## Explicit non-goals for v1 - A hosted service other people can use. - Anything that works around a protection measure. - A shipped, enabled extractor for any platform. - Loudness normalisation and audio repair. - hosted UI for non-technical users - playlist preview with per-track checkboxes - priority queue and 320 kbps Pro path - someone else fighting YouTube bot checks, PO tokens, and geo blocks - email magic-link Pro auth and fair multi-user queueing ## Success criteria - An unchanged feed is confirmed to transfer no body on a second poll - A matching source is confirmed stream-copied rather than re-encoded - The default build is confirmed to expose no extractor - Persisted backoff is confirmed to survive a restart - The size ceiling is confirmed to refuse fetches and delete nothing
BRIEF.md
# Build brief · AudioFetcher The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal audio fetcher that downloads what publishers offer for download, encodes it once, and refuses the rest to replace AudioFetcher. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This product downloads audio from somewhere else, so the platform terms and the copyright position are the design rather than a paragraph at the end of the README. YouTube's Terms of Service, currently effective 17 March 2025, forbid accessing, reproducing, downloading, distributing or otherwise using any part of the Service or Content except as expressly permitted by the Service, with prior written permission, or as permitted by applicable law. A second restriction forbids circumventing anything that prevents or restricts copying, and a third forbids automated access except by public search engines following robots.txt. In the United States, 17 U.S.C. 1201(a)(1)(A) says that no person shall circumvent a technological measure that effectively controls access to a work. And the private copying exception many people assume exists does not exist in the United Kingdom: section 28B of the 1988 Act was inserted in 2014, the regulations that inserted it were quashed by the High Court on 17 July 2015, and the Intellectual Property Office's own notice says those acts now constitute acts of infringement. So the default build is not a video ripper. It fetches what the publisher offers for download: podcast enclosures from a feed, direct file addresses, archives that publish downloadable media, and files you already hold. A site-specific extractor is optional and off by default, and turning it on records who attested what and when. Nothing here circumvents anything, and no checkbox here is pretending to be a licence. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with a small local web interface bound to the loopback address · this is one person's library and it should not be reachable from anywhere else - Fetching: Conditional requests with stored validators, one connection per host at a time · a feed poller that ignores caching headers is a poller that gets blocked - Encoding: ffmpeg, with stream copy preferred over transcoding wherever the format already matches · re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse - Storage: SQLite for rows, audio files on disk keyed by checksum · the checksum is the identity of a file and the filename is only a convenience - Extractors: An optional plugin interface, disabled unless configured, with an attestation record · the honest way to offer this is as a deliberate act with a name and a date attached ### Data model (create this before Phase 1) - sources(id, kind, url, added_at, etag, last_modified, licence_note, attestation_id) · kind is feed, direct, archive or extractor - items(id, source_id, remote_guid, title, published_at, media_url, bytes, mime, duration_ms, checksum, state) - fetches(id, item_id, attempt, http_status, bytes, started_at, ended_at, error_excerpt) · every attempt including the refused ones - files(item_id, path, format, bitrate_kbps, channels, sample_rate, transcoded_from, checksum) · transcoded_from records the chain, or is null when the stream was copied - attestations(id, made_at, who, statement, source_url) · a claim a person made with a date, which is a different and more honest thing than a permission - host_policy(host, robots_fetched_at, crawl_delay_seconds, max_concurrency, backoff_until) · one row per host you touch - No row here says permission granted. The nearest thing is a recorded claim, and the interface says so wherever it shows one. ### Phase 1 · Sources that publish downloads Build: Add a feed by address, parse its enclosures, and record each item with the publisher's own identifier. Store the entity tag and last-modified value from every response and send them back as conditional request headers on the next poll. Read the host's robots file before crawling it and record what it said. Never fetch the same media address twice for the same item. Trap: Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. Done when: A second poll of an unchanged feed sends the stored validators and transfers no body, verified against a local server that asserts the headers arrived. An item already fetched is never fetched again even when the feed reorders or republishes it, tested with a feed fixture that does both. A Retry-After header delays the next request by at least the stated interval, tested. The robots response for each host is recorded with the time it was read, and a disallowed path is not fetched. Do not build yet: No encoding, no extractor, no library interface yet. ### Phase 2 · Encode once, or not at all Build: Probe the source stream first. If the codec and bitrate already satisfy the request, copy the stream and write files.transcoded_from as null. Otherwise transcode once, with the parameters recorded, and store the chain. Write tags from the feed metadata including the source address and the fetch date. Keep the original file until an explicit purge. Trap: Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. Done when: A source already matching the requested format produces a file whose audio stream is bit-identical to the source, tested by comparing stream checksums. An upward bitrate request is refused with an explanation rather than performed, tested. Every transcoded file records its source format and the encoder parameters in transcoded_from, and no file has a null chain unless it was a stream copy, asserted in the suite. Tags include the source address and the fetch date and survive a round trip through a tag reader. Do not build yet: No extractor, no queue, no library or export yet. ### Phase 3 · The extractor boundary Build: Define a plugin interface and ship no extractor enabled by default. Enabling one requires configuration plus an attestation row naming the person, the statement and the date. Refuse to run an extractor against a host whose published terms the operator has not recorded reading. Never implement anything that descrambles, decrypts or otherwise bypasses a protection measure, whatever a library offers. Trap: An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. Done when: With no extractor configured, the interface offers only feeds, direct addresses and local files, and the extractor code path is unreachable, tested. Enabling an extractor without an attestation row fails with a message naming what is missing, tested. Every item fetched through an extractor displays its attestation, its date and who made it, checked in the interface test. No code path in the repository performs decryption or signature evasion of any kind, verified by review and stated in the README. Do not build yet: No queue policy, no library interface, no ceilings yet. ### Phase 4 · A queue that is polite to strangers Build: Queue fetches with at most the configured concurrency per host and a delay between requests taken from the host policy. Back off on a 429 or 503 and store backoff_until so a restart does not reset it. Refuse media longer than a configured ceiling and refuse anything reporting itself as a live stream. Show the queue with the reason each item is waiting. Trap: Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. Done when: Two queued items for the same host never download concurrently at the default configuration, tested against a server that fails on overlap. A 429 response sets backoff_until and a process restart preserves it, tested by restarting mid-backoff. Media exceeding the duration ceiling is refused before any bytes are transferred, tested using the feed's declared duration and a probe. A live stream is refused with a named reason rather than downloaded indefinitely, tested. Do not build yet: No disk ceilings, no export, no retention yet. ### Phase 5 · Disk, retention and an honest README Build: Enforce a total size ceiling and refuse new fetches when it is reached rather than deleting to make room. Add retention by source with an explicit keep list. Export the whole library as a manifest of checksums, titles, sources, attestations and file paths, so the audio and the record can be moved together. Write the README so a stranger can tell in thirty seconds what is in scope and what is not. Trap: A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all. Done when: Reaching the size ceiling refuses new fetches with a named reason and deletes nothing, tested by filling to the boundary. Retention deletes only sources with an explicit policy, and an item on the keep list survives a retention run, tested. The exported manifest reimports into an empty database and every file resolves by checksum, tested. The README states the terms position, the copyright position and the extractor default in its first section, checked by review. Do not build yet: Nothing after this. ### Out of scope (and why) - A hosted service other people can paste addresses into, because you would then be operating the thing the terms above are written about, at your own risk and on your own domain. - Anything that descrambles, decrypts or works around a protection measure, which is a separate prohibition from the terms and is not negotiable here. - Shipping an enabled extractor for any platform, since the entire attestation design exists because that choice belongs to the operator and not to the author. - Loudness normalisation and audio repair, which are a genuinely interesting project and have nothing to do with the problems this build is about. ### README must contain - The platform terms position and the copyright position, stated in the first section with the sources and the dates they were read - That the private copying exception people assume exists was quashed in the United Kingdom in 2015 and that the position differs by jurisdiction - That no extractor is enabled by default, what enabling one requires, and that an attestation is a recorded claim rather than a permission - That audio already in the requested format is copied rather than re-encoded, and why an upward bitrate request is refused - Where the files live, what the size ceiling does when reached, and how to export the manifest with the audio
ARCHITECTURE.md
# Architecture · AudioFetcher ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with a small local web interface bound to the loopback address | this is one person's library and it should not be reachable from anywhere else | | Fetching | Conditional requests with stored validators, one connection per host at a time | a feed poller that ignores caching headers is a poller that gets blocked | | Encoding | ffmpeg, with stream copy preferred over transcoding wherever the format already matches | re-encoding lossy audio to a higher bitrate makes a bigger file that sounds worse | | Storage | SQLite for rows, audio files on disk keyed by checksum | the checksum is the identity of a file and the filename is only a convenience | | Extractors | An optional plugin interface, disabled unless configured, with an attestation record | the honest way to offer this is as a deliberate act with a name and a date attached | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Sources | Feeds, direct addresses and validators. | Any parser, provided conditional requests and publisher identifiers are kept. | | Queue | Per-host concurrency, persisted backoff and guards. | Any runner, provided backoff survives a restart. | | Encode | Probing, stream copy and recorded transcode chains. | Any encoder, provided a matching source is copied rather than re-encoded. | | Extractors | The optional plugin boundary and attestations. | Not swappable for an always-on extractor, which changes what the tool is. | | Library | Checksums, ceiling, retention and the manifest. | Any store, provided the manifest exports with the attestations attached. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `LIBRARY_DIR` · required · Audio files, keyed by checksum. The largest thing on the disk. - `DATABASE_PATH` · required · Sources, items, fetches and attestations. Back this up; the audio is refetchable and the record is not. - `LIBRARY_CEILING_GB` · required · New fetches are refused at this size. Nothing is ever evicted to make room. - `MAX_DURATION_MINUTES` · required · Media longer than this is refused before any bytes move. - `HOST_CONCURRENCY` · required · Simultaneous downloads per host. Raise it only with a reason. - `EXTRACTORS_ENABLED` · required · Off unless deliberately set, and setting it needs an attestation row to exist. - `USER_AGENT` · required · Identifies you honestly to the hosts you fetch from, with a way to reach you. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · AudioFetcher product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with a small local web interface bound to the loopback address, Conditional requests with stored validators, one connection per host at a time, ffmpeg, with stream copy preferred over transcoding wherever the format already matches, SQLite for rows, audio files on disk keyed by checksum, An optional plugin interface, disabled unless configured, with an attestation record. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. - A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all.
MILESTONES.md
# Delivery milestones · AudioFetcher Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Sources that publish downloads Feeds and direct addresses are fetched once each, politely, with validators stored so a poll is usually free. ### Steps 1. Store validators and send them back A not-modified response should be the common case. Files: `src/fetch/conditional.py`, `tests/test_conditional.py` 2. Key items on the publisher's identifier Feeds reorder, republish and change titles. Files: `src/feeds/parse.py`, `tests/test_feed_dedupe.py` 3. Read robots and honour Retry-After Both are instructions, and ignoring them is how access ends. Files: `src/fetch/policy.py`, `tests/test_retry_after.py` ### Done when - [ ] An unchanged feed transfers no body on the second poll - [ ] A reordered or republished item is not refetched - [ ] Retry-After delays the next request - [ ] Robots responses are recorded and honoured ### Watch out - Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion. ## M2 · Encode once, or not at all Audio already in the requested format is copied rather than re-encoded, and any transcode records what it came from. ### Steps 1. Probe before deciding to encode Most fetches do not need an encoder at all. Files: `src/encode/probe.py`, `tests/test_streamcopy.py` 2. Refuse an upward bitrate request There is nothing to put back and the file gets worse. Files: `src/encode/plan.py` 3. Record the chain and tag the source Six months later this is the only provenance you have. Files: `src/encode/run.py`, `src/encode/tags.py` ### Done when - [ ] A matching source is stream-copied bit-identically - [ ] An upward bitrate request is refused with a reason - [ ] Every transcode records its chain and parameters - [ ] Tags carry the source address and fetch date ### Watch out - Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back. ## M3 · The extractor boundary Site-specific extraction is off unless someone deliberately turns it on and records why. ### Steps 1. Ship with no extractor enabled The default build is the one most people will run. Files: `src/extractors/__init__.py`, `tests/test_default_disabled.py` 2. Require a stored attestation, not a checkbox A claim with a name and a date is the only honest artefact here. Files: `src/extractors/attest.py`, `migrations/003_attestations.sql` 3. Refuse anything that bypasses a protection measure This is a line, not a configuration option. Files: `docs/scope.md` ### Done when - [ ] The default build exposes no extractor - [ ] Enabling without an attestation fails with a reason - [ ] Extractor items display their attestation - [ ] No circumvention code exists in the repository ### Watch out - An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too. ## M4 · A queue that is polite to strangers Downloads run one host at a time with recorded backoff, and a long or live source is refused before it starts. ### Steps 1. Limit concurrency per host, not globally One slow publisher should not be hammered by your whole queue. Files: `src/queue/claim.py`, `tests/test_host_concurrency.py` 2. Persist backoff across restarts In-memory backoff resets exactly when you least want it to. Files: `src/queue/backoff.py`, `tests/test_backoff_persist.py` 3. Refuse live and over-long media up front Before bytes, not after two hours of them. Files: `src/queue/guard.py` ### Done when - [ ] One host is never downloaded from concurrently by default - [ ] Backoff survives a restart - [ ] Over-long media is refused before transfer - [ ] Live streams are refused with a reason ### Watch out - Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it. ## M5 · Disk, retention and an honest README (production only) The library cannot fill the disk, everything exports, and the documentation says plainly what this tool will and will not do. ### Steps 1. Refuse at the ceiling, never evict The oldest file is often the one that no longer exists upstream. Files: `src/storage/ceiling.py`, `tests/test_ceiling.py` 2. Retention by explicit policy only Silence should mean keep. Files: `src/storage/retention.py` 3. Export a manifest that includes the attestations The record travels with the audio or it is not a record. Files: `src/export/manifest.py`, `tests/test_manifest.py` ### Done when - [ ] The size ceiling refuses fetches and deletes nothing - [ ] Retention touches only sources with a policy - [ ] The manifest reimports and every file resolves - [ ] The README leads with scope, not with features ### Watch out - A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all.
OPERATIONS.md
# Operations · AudioFetcher ## Backup The database and the manifest first, the audio second. Most audio can be fetched again from its publisher; the record of what you fetched, when, and on what stated basis cannot be reconstructed at all. ## Restore Restore the database, then verify every file resolves by checksum and report the missing ones rather than silently refetching them. A file missing after a restore may have been removed upstream, and quietly refetching hides that. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Not-modified rate per feed, backoff events per host, the library size against its ceiling, and the count of items whose attestation was removed. A falling not-modified rate usually means a publisher stopped sending validators, which is worth noticing before they notice you. ## Incident checklist The incident here is a fetch you cannot account for. The fetches table, the host policy rows and the attestations are what make the answer a record rather than a recollection, which is the entire reason they are separate tables. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] An unchanged feed is confirmed to transfer no body on a second poll - [ ] A matching source is confirmed stream-copied rather than re-encoded - [ ] The default build is confirmed to expose no extractor - [ ] Persisted backoff is confirmed to survive a restart - [ ] The size ceiling is confirmed to refuse fetches and delete nothing ## Launch constraint Do not market omitted AudioFetcher capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Audio files, keyed by checksum. The largest thing on the disk. LIBRARY_DIR=./library # Required. Sources, items, fetches and attestations. Back this up; the audio is refetchable and the record is not. DATABASE_PATH=./data/library.db # Required. New fetches are refused at this size. Nothing is ever evicted to make room. LIBRARY_CEILING_GB=50 # Required. Media longer than this is refused before any bytes move. MAX_DURATION_MINUTES=240 # Required. Simultaneous downloads per host. Raise it only with a reason. HOST_CONCURRENCY=1 # Required. Off unless deliberately set, and setting it needs an attestation row to exist. EXTRACTORS_ENABLED=false # Required. Identifies you honestly to the hosts you fetch from, with a way to reach you. USER_AGENT=my-audio-library/1.0 (contact: me@example.com)
$ choose a build depth, inspect the files, then open the complete pack in your agent
Power users already run yt-dlp. Everyone else pays to paste a link in a browser and get an MP3 without installing tools or debugging LOGIN_REQUIRED. The money is convenience plus the permanent maintenance of staying ahead of YouTube client checks, not a novel audio algorithm.
xhosted UI for non-technical users
xplaylist preview with per-track checkboxes
xpriority queue and 320 kbps Pro path
xsomeone else fighting YouTube bot checks, PO tokens, and geo blocks
xemail magic-link Pro auth and fair multi-user queueing
share on X ↗"I just replaced AudioFetcher ($6/mo) with one prompt"
Don't feel like building it? These folks already made it free.
yt-dlpUgly, terminal-shaped, and still the downloader every prettier app quietly depends on.open source↗PParabolicA polite desktop face on yt-dlp; paste a playlist, pick MP3, go make tea.open source↗TTartubeA gloriously busy desktop queue for whole channels and playlists.open source↗all 3 free alternatives to AudioFetcher →· no votes, no pay-to-list · just what's real
AudioFetcher pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| free | $0/user | $0/user | 3 conversions/day; individual videos up to 20 minutes; up to 256 kbps; 0 playlist conversions; standard queue. |
| pro | $6/user | $4.92/user | Unlimited conversions; playlists; priority queue; 320 kbps output; videos up to 12 hours. |
| supporter pass | custom | — | 30 days of unlimited conversions; playlists; priority queue; 320 kbps; videos up to 12 hours. |
free tier3 conversions/day; individual videos up to 20 minutes; up to 256 kbps; 0 playlist conversions; standard queue.
billingPro is monthly + annual; Supporter Pass is a one-time $5 purchase for 30 days.
hidden costsThe service's official terms require users to convert only content they own or have permission to use.
verified 2026-08-12 · source ↗
Vibecode AudioFetcher
Kinda. The core of AudioFetcher is buildable in a weekend with the prompt on this page, but there are real gaps: hosted UI for non-technical users, playlist preview with per-track checkboxes. Read the honest list above before committing.
How much does AudioFetcher cost?
AudioFetcher costs about $6/month (Pro, checked 2026-07-30), which is $72 per year.
What do I lose by replacing AudioFetcher?
Honestly: hosted UI for non-technical users; playlist preview with per-track checkboxes; priority queue and 320 kbps Pro path; someone else fighting YouTube bot checks, PO tokens, and geo blocks; email magic-link Pro auth and fair multi-user queueing. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to AudioFetcher?
Yes: yt-dlp (Ugly, terminal-shaped, and still the downloader every prettier app quietly depends on.) Parabolic (A polite desktop face on yt-dlp; paste a playlist, pick MP3, go make tea.) Tartube (A gloriously busy desktop queue for whole channels and playlists.) All 3 curated free alternatives are at vibecodeit.com/audiofetcher/alternatives. The prompt is for when you want it exactly your way.