Build AudioFetcher

KINDAreplaces $6/mosaves $72/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A personal audio library whose hard problems are fetching politely and only once from sources that publish downloads, refusing to re-encode audio that is already in the format you asked for, and keeping an honest record of the permission claim behind anything fetched through the optional extractor.

estimated effort weekendthe files for this build are in the project pack

RuntimePython 3.12 with a small local web interface bound to the loopback addressFetchingConditional requests with stored validators, one connection per host at a timeEncodingffmpeg, with stream copy preferred over transcoding wherever the format already matchesStorageSQLite for rows, audio files on disk keyed by checksumExtractorsAn optional plugin interface, disabled unless configured, with an attestation record

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the local interface and the fetch queue.
Get it python.org or your package manager. open ↗
Verify python3 --version prints 3.12 or higher
installfree
Why Probing before encoding is what makes the stream-copy path possible, and ffprobe is how you know whether to encode at all.
Get it Your package manager. open ↗
Verify ffprobe -version prints a version and ffmpeg -codecs lists an MP3 encoder
have readyfree
Why Rule zero rests on what those documents say today, and they change, so reading them is part of setup rather than part of the argument afterwards.
Get it Read each host's terms and record the URL and the date in your configuration.
Verify Every configured host has a terms URL and a date recorded
have readyfree
Why An audio library grows until something stops it, and the thing that stops it should be a ceiling you chose rather than a full filesystem.
Get it Decide the ceiling and the path, and put both in configuration.
Verify Filling to the ceiling refuses new fetches
decidefree
Why This decision changes what the tool is, and making it deliberately at the start is the difference between a library and a liability.
Get it Write down the answer, and if it is yes, write the attestation statement you are prepared to sign.
Verify The README and the configuration agree, and the attestation row exists if enabled

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- sources(id, kind, url, added_at, etag, last_modified, licence_note, attestation_id) · kind is feed, direct, archive or extractor
- items(id, source_id, remote_guid, title, published_at, media_url, bytes, mime, duration_ms, checksum, state)
- fetches(id, item_id, attempt, http_status, bytes, started_at, ended_at, error_excerpt) · every attempt including the refused ones
- files(item_id, path, format, bitrate_kbps, channels, sample_rate, transcoded_from, checksum) · transcoded_from records the chain, or is null when the stream was copied
- attestations(id, made_at, who, statement, source_url) · a claim a person made with a date, which is a different and more honest thing than a permission
- host_policy(host, robots_fetched_at, crawl_delay_seconds, max_concurrency, backoff_until) · one row per host you touch
- No row here says permission granted. The nearest thing is a recorded claim, and the interface says so wherever it shows one.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`LIBRARY_DIR`	required	`./library`	Audio files, keyed by checksum. The largest thing on the disk.
`DATABASE_PATH`	required	`./data/library.db`	Sources, items, fetches and attestations. Back this up; the audio is refetchable and the record is not.
`LIBRARY_CEILING_GB`	required	`50`	New fetches are refused at this size. Nothing is ever evicted to make room.
`MAX_DURATION_MINUTES`	required	`240`	Media longer than this is refused before any bytes move.
`HOST_CONCURRENCY`	required	`1`	Simultaneous downloads per host. Raise it only with a reason.
`EXTRACTORS_ENABLED`	required	`false`	Off unless deliberately set, and setting it needs an attestation row to exist.
`USER_AGENT`	required	`my-audio-library/1.0 (contact: me@example.com)`	Identifies you honestly to the hosts you fetch from, with a way to reach you.

The build, in order

1
Sources that publish downloads
Feeds and direct addresses are fetched once each, politely, with validators stored so a poll is usually free.
1. A not-modified response should be the common case.
  Files src/fetch/conditional.pytests/test_conditional.py
2. Feeds reorder, republish and change titles.
  Files src/feeds/parse.pytests/test_feed_dedupe.py
3. Both are instructions, and ignoring them is how access ends.
  Files src/fetch/policy.pytests/test_retry_after.py
done when · tick each as it passes
watch out
- Polling a feed by downloading the whole document and diffing it means you transfer the same megabyte every fifteen minutes for a feed that changes weekly, and some publishers will rate limit you for it, which then looks like the feed is broken. Send the stored validators, handle a not-modified response as the normal case, and honour a Retry-After header as an instruction rather than a suggestion.
next phase
2
Encode once, or not at all
Audio already in the requested format is copied rather than re-encoded, and any transcode records what it came from.
1. Most fetches do not need an encoder at all.
  Files src/encode/probe.pytests/test_streamcopy.py
2. There is nothing to put back and the file gets worse.
  Files src/encode/plan.py
3. Six months later this is the only provenance you have.
  Files src/encode/run.pysrc/encode/tags.py
done when · tick each as it passes
watch out
- Transcoding a 128 kbps source up to 320 kbps produces a file two and a half times the size that still contains every artefact of the first encoder plus the artefacts of the second, and a library full of these is a library that got worse while getting bigger. Probe first, copy the stream when it already satisfies the request, and refuse an upward bitrate request with a message explaining that it cannot add anything back.
next phase
3
The extractor boundary
Site-specific extraction is off unless someone deliberately turns it on and records why.
1. The default build is the one most people will run.
  Files src/extractors/__init__.pytests/test_default_disabled.py
2. A claim with a name and a date is the only honest artefact here.
  Files src/extractors/attest.pymigrations/003_attestations.sql
3. This is a line, not a configuration option.
  Files docs/scope.md
done when · tick each as it passes
watch out
- An attestation implemented as a checkbox with no stored row is worth exactly nothing when somebody asks six months later what was fetched and on what basis, and it also makes it trivially easy to click through. Store the claim with the person, the statement, the source and the date, show it in the library beside every item it covers, and make removing the attestation remove the item's downloadable status too.
next phase
4
A queue that is polite to strangers
Downloads run one host at a time with recorded backoff, and a long or live source is refused before it starts.
1. One slow publisher should not be hammered by your whole queue.
  Files src/queue/claim.pytests/test_host_concurrency.py
2. In-memory backoff resets exactly when you least want it to.
  Files src/queue/backoff.pytests/test_backoff_persist.py
3. Before bytes, not after two hours of them.
  Files src/queue/guard.py
done when · tick each as it passes
watch out
- Parallel downloads across a whole feed look fast until the publisher's host starts refusing you, and because backoff usually lives in memory a restart clears it and you walk straight back into the block. Persist backoff_until, apply it at claim time rather than at enqueue time, and keep per-host concurrency at one until you have a reason to raise it.
5
Disk, retention and an honest READMEproduct builder
The library cannot fill the disk, everything exports, and the documentation says plainly what this tool will and will not do.
1. The oldest file is often the one that no longer exists upstream.
  Files src/storage/ceiling.pytests/test_ceiling.py
2. Silence should mean keep.
  Files src/storage/retention.py
3. The record travels with the audio or it is not a record.
  Files src/export/manifest.pytests/test_manifest.py
done when · tick each as it passes
watch out
- A retention policy that deletes the oldest files to stay under a ceiling will one day delete the only copy of something the publisher has since taken down, and the queue will happily keep fetching new episodes over the top of it. Refuse new fetches at the ceiling instead of evicting, and require an explicit keep list before anything old is deleted at all.

what this build does not replace

A hosted service other people can use.
Anything that works around a protection measure.
A shipped, enabled extractor for any platform.
Loudness normalisation and audio repair.
hosted UI for non-technical users
playlist preview with per-track checkboxes
priority queue and 320 kbps Pro path
someone else fighting YouTube bot checks, PO tokens, and geo blocks
email magic-link Pro auth and fair multi-user queueing

after v1, if you want it

Chapter and transcript handling for feeds that publish them, which is metadata the publisher already offers
A local search over titles and descriptions, which costs nothing once the rows are already there
A player that reads the manifest directly, so the library works even if this application stops running

Need the files? The project pack on the verdict page hands your agent the whole brief · more audio & video.