KINDA · weekend project
price $12.99/moyou'd save $155.88/yrbuild time a weekendcategory 🎙️ meeting notesreplaced by 0 people
The pieces are all commodity now: capture audio, stream it through a local Whisper variant, keep a rolling transcript, and every few seconds ask a model for the next good question. An agent will get you that in one sitting on a laptop, and the transcript-plus-summary half will genuinely be fine. What it will not get you is a phone in your pocket that survives being backgrounded during a two hour lecture, or sub-second suggestions that arrive before the moment passes. Live assistance is a latency and UX problem more than an AI problem, and that is exactly the part that takes weeks of fiddling. Build it if you mostly sit at a desk; keep paying if you mostly do not.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Hedy. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Hedy · indie build A local study companion whose hard problems are producing an end-of-session artefact that actually aids retention rather than one that feels productive, keeping a live panel quiet enough not to cost more attention than it returns, and decoding audio without inventing sentences over silence. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and one page over a websocket, no build step | this runs on your own machine and every dependency you add is one you maintain | | Capture | sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device | the other side of a call only exists if you route it, and that is a documented setup step | | Transcription | A local speech model with overlapping chunks and voice activity detection in front | decoding silence is where invented sentences come from | | Storage | SQLite for sessions, segments, cards and attempts, plus a markdown file per session | the notes should outlive the application that made them | | Assistance | One model call per scheduled tick, rate limited above the model rather than below it | the cap has to hold even when the model answers faster than you expected | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the capture loop, the decoder and the server. - Get it: python.org or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A local speech recognition model and its runtime** · free, after the download - Why: Everything else in this build assumes the audio never leaves the machine, and a hosted transcription service would make that untrue. - Get it: Install the runtime and download a small English model on first run. - Verify: A one minute recording transcribes faster than real time on your machine - [ ] **A loopback audio device** (optional) · free - Why: Without one you capture only your own microphone, which is half a conversation and makes the transcript useless for a call. - Get it: A virtual audio device for your platform, installed and selected as an input. - Verify: It appears in the device list and a played recording produces segments - [ ] **An API key for a text model** · pay per call; the rate cap keeps it small - Why: The panel and the question generation are model calls, and they are the only networked part. - Get it: Create a key with your preferred provider and set a spend limit in their console. - Verify: Ending a test session produces questions that cite real segments - [ ] **What your institution says about recording teaching** · free - Why: Recording a lecture is not only a question between you and the lecturer, and finding out afterwards is the wrong order. - Get it: Read the recording policy where you study and write down what it permits. - Verify: A note in the README states the policy you checked and the date ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Mobile background recording. - Speaker separation. - Anything that assists during an assessed exercise. - Calendar and conferencing integrations. - Mobile apps that keep recording reliably when the screen is off, which is where most of these sessions actually happen - Tuned latency: your DIY suggestions arrive a beat late, and a beat late is useless in conversation - Speaker diarization and meeting-type presets that shape the assistant for a sales call vs a lecture vs a doctor visit - Calendar and conferencing integrations that join and label sessions for you - Hosted, searchable history across every session with no laptop babysitting If one of those is essential to you, that is the reason to keep paying for Hedy, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Hedy The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a desk-bound study companion that keeps a live transcript, offers very few prompts while you are in the room, and turns the session into questions you have to answer to replace Hedy. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This one is aimed at students, so read the learning literature before you design the output, because the obvious build optimises the condition that loses. Roediger and Karpicke, Psychological Science 2006, gave students prose passages and had them either restudy or take recall tests the same number of times. Five minutes later restudying won, 81 percent of idea units recalled against 75. Two days later it reversed: 68 percent for the tested group against 54. One week later, 56 against 42. The tested group recalled as much after a week as the restudy group did after two days. An app whose output is a beautiful summary you reread is the restudy condition, and it feels better than the thing that works. So the artefact at the end of a session is a list of questions with the answers collapsed. The summary is generated too, it is one click away, and it is not the landing page. Second, the live panel. Sana, Weston and Cepeda, Computers and Education 2013, found that students who multitasked on a laptop during a lecture scored lower than those who did not, and that students merely in direct view of a multitasking peer also scored lower. A panel that updates whenever the model returns is a multitasking machine pointed at a lecture, and the row behind you can see it. Cap the update rate in the scheduler, default the panel closed for lecture sessions, and build no sound, no animation and no notification anywhere in this application. Third, the room. Tell people you are recording, and find out what your institution says about recording a lecture before you record one. Nothing here feeds you answers during an assessed exercise, and that is a refusal rather than a missing feature. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and one page over a websocket, no build step · this runs on your own machine and every dependency you add is one you maintain - Capture: sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device · the other side of a call only exists if you route it, and that is a documented setup step - Transcription: A local speech model with overlapping chunks and voice activity detection in front · decoding silence is where invented sentences come from - Storage: SQLite for sessions, segments, cards and attempts, plus a markdown file per session · the notes should outlive the application that made them - Assistance: One model call per scheduled tick, rate limited above the model rather than below it · the cap has to hold even when the model answers faster than you expected ### Data model (create this before Phase 1) - sessions(id, kind, started_at, ended_at, brief, consent_note, device, model_id) · kind is lecture, seminar, meeting or call, and it changes the defaults rather than just a label - segments(id, session_id, t_start, t_end, text, avg_logprob, no_speech_prob) · one per decoded chunk, keeping whatever confidence the decoder reported - cards(id, session_id, at, kind, text, source_segment_ids_json, shown_at, dismissed_at) · a live prompt, with the transcript it was built from - questions(id, session_id, text, answer, source_segment_ids_json, created_at) · the artefact that matters, answers stored separately from the question text - attempts(id, question_id, at, self_rating, next_due_at) · retrieval practice needs a record of the attempt, not a record of the reading - settings(panel_updates_per_minute, lecture_panel_default_off, audio_retention_days, interval_schedule_json) - There is no streak table, no score and no notification queue. A study tool that nags is a study tool uninstalled in week three. ### Phase 1 · Capture, and a transcript that never invents Build: List input devices and let the user pick, optionally mixing two into one stream. Run voice activity detection before the decoder and skip non-speech. Decode in overlapping chunks, merge on the overlap, and push each segment to the browser with its timestamps and confidence. Show a recording indicator that cannot be hidden, and store the consent note on the session before recording starts. Trap: Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. Done when: Chunk boundaries do not lose words, tested by decoding a known recording and diffing against a single-pass transcript. A stretch of silence produces no segments at all, tested with a recorded gap. Every segment carries its start, end and the decoder's confidence values. Recording cannot start without a consent note stored on the session, and the indicator is visible whenever capture is running. Do not build yet: No assistance panel, questions, review schedule or export yet. ### Phase 2 · A panel that does as little as possible Build: Schedule at most a configured number of model calls per minute and render at most a few cards at once. Provide a manual nudge. For sessions of kind lecture, start with the panel collapsed and let the nudge be the only way it opens. No sound, no animation, no colour that moves, no notification of any kind anywhere in the application. Trap: Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. Done when: Panel updates never exceed the configured rate even with a stubbed model that returns instantly, tested. A lecture session starts with the panel collapsed and a meeting session does not, tested from the session kind. No code path plays a sound, animates an element or raises a notification, checked by review and by a grep in the suite. Every card carries the segment identifiers it was built from and clicking one seeks the transcript to that point. Do not build yet: No questions, review schedule, export or retention yet. ### Phase 3 · The session ends in questions Build: Generate short-answer questions from the transcript, each tied to the segments it came from, with the answer stored separately and collapsed by default. Land on the question list when a session ends. Generate the summary as well, and put it behind a click. Discard any question whose cited segments do not resolve rather than displaying it. Trap: Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. Done when: Ending a session lands on the question list with every answer collapsed, checked by review. Every displayed question resolves its segment identifiers to real segments, and one that does not is discarded before display, tested with a fabricated response. Revealing an answer writes an attempt row with the self rating. The summary exists and is reachable in one click and is never the landing view. Do not build yet: No review schedule, export or retention yet. ### Phase 4 · Bringing them back, without nagging Build: Compute a next due date from the attempt and its self rating using an interval schedule held in a configuration file. Derive the due list from attempts rather than storing a queue. Show the next due date on every attempt so the schedule is visible. The only way to be reminded is opening the application, and there is no path that changes that. Trap: A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. Done when: An attempt writes a next due date computed from the configured schedule, tested at the first, middle and last interval. The due list is a query over attempts and no queue table exists, checked in the schema. No notification, email or scheduled message path exists anywhere, asserted in the suite. Exported questions carry their attempt history and reimport with the same due dates. Do not build yet: No export, retention or spend ceiling yet. ### Phase 5 · Leaving, and forgetting Build: Write a markdown file per session with the transcript, the questions and the summary, plus a JSON bundle of every table. Delete audio after a configured retention while keeping segments, questions and attempts. Delete a session entirely on request. Cap monthly model spend. Provide a mode where the local decoder runs and no request leaves the machine at all. Trap: Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows. Done when: Audio older than the retention is deleted and the segments, questions and attempts survive intact, tested. Deleting a session removes its rows and its markdown file, tested. The JSON bundle reimports into an empty database with matching counts and due dates, tested. With the network disabled the application records, transcribes and shows the transcript, and the panel says the assistance is unavailable rather than failing. Do not build yet: Nothing after this. ### Out of scope (and why) - A phone that keeps recording with the screen off, which is the actual product being sold and is weeks of background audio lifecycle work per platform. - Speaker separation, which needs a different model and a tuning loop, and which is the feature people expect first and forgive last. - Anything that supplies answers during an assessed exercise, which is not a feature this build is missing but one it refuses. - Calendar and conferencing integrations that join sessions for you, which is a permanent commitment to somebody else's API. ### README must contain - Why the end-of-session artefact is a question list rather than a summary, with the retention figures and the study behind them - Why the live panel has a hard update cap and is closed by default in lectures, with the multitasking study behind it - That there are no notifications anywhere, and that this is deliberate - The loopback device setup for capturing the other side of a call, per platform - How long audio is kept, what deleting a session removes, and where the markdown files live ===== AGENTS.md ===== # Agent instructions · Hedy indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and one page over a websocket, no build step, sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device, A local speech model with overlapping chunks and voice activity detection in front, SQLite for sessions, segments, cards and attempts, plus a markdown file per session, One model call per scheduled tick, rate limited above the model rather than below it. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. ===== BUILD_PLAN.md ===== # Build plan · Hedy A local study companion whose hard problems are producing an end-of-session artefact that actually aids retention rather than one that feels productive, keeping a live panel quiet enough not to cost more attention than it returns, and decoding audio without inventing sentences over silence. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Capture, and a transcript that never invents Audio is captured from a device you chose and decoded only where somebody is actually speaking. ### Steps 1. Let the user pick and mix input devices Capturing the other side of a call is a routing problem, not a code problem. Files: `app/audio.py`, `README.md` 2. Put voice activity detection in front of the decoder Silence in, invented sentences out. Files: `app/vad.py`, `tests/test_vad.py` 3. Overlap chunks and merge on the overlap Otherwise you lose a word every boundary and never notice. Files: `app/transcribe.py`, `tests/test_overlap.py` ### Done when - [ ] Chunk boundaries do not drop words - [ ] Silence produces no segments - [ ] Segments carry timestamps and confidence - [ ] No recording without a stored consent note ### Watch out - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. ## Phase 2 · A panel that does as little as possible The live assistance costs less attention than it returns, and in a lecture it is closed. ### Steps 1. Cap the rate above the model, not below it A cap in the renderer is not a cap. Files: `app/assist.py`, `tests/test_rate.py` 2. Default the panel closed for lectures The session kind is a design decision, not a label. Files: `app/sessions.py`, `web/panel.js` 3. Make every card cite its segments A prompt you cannot trace is a prompt you cannot check. Files: `app/assist.py` ### Done when - [ ] The update rate holds with an instant model - [ ] Lecture sessions start with the panel closed - [ ] Nothing sounds, animates or notifies - [ ] Cards cite segments and seek the transcript ### Watch out - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. ## Phase 3 · The session ends in questions Finishing a session produces short-answer questions with the answers hidden, and the summary is secondary. ### Steps 1. Land on questions, not on prose The default view is the design decision here. Files: `web/session_end.js`, `app/questions.py` 2. Store answers apart from questions So the interface cannot accidentally show both. Files: `app/db.py`, `tests/test_questions.py` 3. Discard questions that cannot cite A question about something nobody said is worse than no question. Files: `app/questions.py`, `tests/test_citation.py` ### Done when - [ ] A finished session lands on collapsed questions - [ ] Every question resolves to real segments - [ ] Revealing an answer records an attempt - [ ] The summary exists but is never the landing view ### Watch out - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. ## Phase 4 · Bringing them back, without nagging Questions come due on a schedule you can read, and nothing chases you. ### Steps 1. Put the intervals in a configuration file With the reasoning next to them, not in a commit message. Files: `config/intervals.json`, `app/review.py` 2. Derive the due list, do not store it A stored queue drifts from the attempts that produced it. Files: `app/review.py`, `tests/test_review.py` 3. Leave the reminding to the person The absence of a notification path is the feature. Files: `tests/test_no_notifications.py` ### Done when - [ ] Due dates follow the configured intervals - [ ] The due list is derived from attempts - [ ] No notification path exists - [ ] Exported questions keep their attempt history ### Watch out - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. ## Not in this build - Mobile background recording. - Speaker separation. - Anything that assists during an assessed exercise. - Calendar and conferencing integrations. ## After v1, if you want it - A second question type that asks you to explain rather than recall, generated from the same segments - A weekly view of what you failed to recall, which is the only honest measure of whether this is working - Importing a slide deck as session context, so the questions use the lecturer's own terms ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. The panel and question generation. Absent means local transcription only. MODEL_API_KEY=sk-...redacted # Optional. Stored on the session, because questions differ by model. MODEL_ID=a text model identifier # Required. The local decoder. Larger is slower and the panel cap assumes it keeps up. SPEECH_MODEL=small.en # Required. Sessions, segments, questions and attempts. The attempts are the irreplaceable part. DATABASE_PATH=./data/study.db # Required. One markdown file per session. These are meant to outlive the application. SESSIONS_DIR=./sessions # Required. How long recordings of other people stay on your disk. Short is the right default. AUDIO_RETENTION_DAYS=7 # Required. Hard cap enforced in the scheduler. Lower it before you raise it. PANEL_UPDATES_PER_MINUTE=3 # Required. Assistance stops at this figure and transcription carries on. MONTHLY_COST_CEILING=3.00
You are building a lean indie version of Hedy. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Hedy · indie build A local study companion whose hard problems are producing an end-of-session artefact that actually aids retention rather than one that feels productive, keeping a live panel quiet enough not to cost more attention than it returns, and decoding audio without inventing sentences over silence. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and one page over a websocket, no build step | this runs on your own machine and every dependency you add is one you maintain | | Capture | sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device | the other side of a call only exists if you route it, and that is a documented setup step | | Transcription | A local speech model with overlapping chunks and voice activity detection in front | decoding silence is where invented sentences come from | | Storage | SQLite for sessions, segments, cards and attempts, plus a markdown file per session | the notes should outlive the application that made them | | Assistance | One model call per scheduled tick, rate limited above the model rather than below it | the cap has to hold even when the model answers faster than you expected | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the capture loop, the decoder and the server. - Get it: python.org or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A local speech recognition model and its runtime** · free, after the download - Why: Everything else in this build assumes the audio never leaves the machine, and a hosted transcription service would make that untrue. - Get it: Install the runtime and download a small English model on first run. - Verify: A one minute recording transcribes faster than real time on your machine - [ ] **A loopback audio device** (optional) · free - Why: Without one you capture only your own microphone, which is half a conversation and makes the transcript useless for a call. - Get it: A virtual audio device for your platform, installed and selected as an input. - Verify: It appears in the device list and a played recording produces segments - [ ] **An API key for a text model** · pay per call; the rate cap keeps it small - Why: The panel and the question generation are model calls, and they are the only networked part. - Get it: Create a key with your preferred provider and set a spend limit in their console. - Verify: Ending a test session produces questions that cite real segments - [ ] **What your institution says about recording teaching** · free - Why: Recording a lecture is not only a question between you and the lecturer, and finding out afterwards is the wrong order. - Get it: Read the recording policy where you study and write down what it permits. - Verify: A note in the README states the policy you checked and the date ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Mobile background recording. - Speaker separation. - Anything that assists during an assessed exercise. - Calendar and conferencing integrations. - Mobile apps that keep recording reliably when the screen is off, which is where most of these sessions actually happen - Tuned latency: your DIY suggestions arrive a beat late, and a beat late is useless in conversation - Speaker diarization and meeting-type presets that shape the assistant for a sales call vs a lecture vs a doctor visit - Calendar and conferencing integrations that join and label sessions for you - Hosted, searchable history across every session with no laptop babysitting If one of those is essential to you, that is the reason to keep paying for Hedy, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Hedy The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a desk-bound study companion that keeps a live transcript, offers very few prompts while you are in the room, and turns the session into questions you have to answer to replace Hedy. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This one is aimed at students, so read the learning literature before you design the output, because the obvious build optimises the condition that loses. Roediger and Karpicke, Psychological Science 2006, gave students prose passages and had them either restudy or take recall tests the same number of times. Five minutes later restudying won, 81 percent of idea units recalled against 75. Two days later it reversed: 68 percent for the tested group against 54. One week later, 56 against 42. The tested group recalled as much after a week as the restudy group did after two days. An app whose output is a beautiful summary you reread is the restudy condition, and it feels better than the thing that works. So the artefact at the end of a session is a list of questions with the answers collapsed. The summary is generated too, it is one click away, and it is not the landing page. Second, the live panel. Sana, Weston and Cepeda, Computers and Education 2013, found that students who multitasked on a laptop during a lecture scored lower than those who did not, and that students merely in direct view of a multitasking peer also scored lower. A panel that updates whenever the model returns is a multitasking machine pointed at a lecture, and the row behind you can see it. Cap the update rate in the scheduler, default the panel closed for lecture sessions, and build no sound, no animation and no notification anywhere in this application. Third, the room. Tell people you are recording, and find out what your institution says about recording a lecture before you record one. Nothing here feeds you answers during an assessed exercise, and that is a refusal rather than a missing feature. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and one page over a websocket, no build step · this runs on your own machine and every dependency you add is one you maintain - Capture: sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device · the other side of a call only exists if you route it, and that is a documented setup step - Transcription: A local speech model with overlapping chunks and voice activity detection in front · decoding silence is where invented sentences come from - Storage: SQLite for sessions, segments, cards and attempts, plus a markdown file per session · the notes should outlive the application that made them - Assistance: One model call per scheduled tick, rate limited above the model rather than below it · the cap has to hold even when the model answers faster than you expected ### Data model (create this before Phase 1) - sessions(id, kind, started_at, ended_at, brief, consent_note, device, model_id) · kind is lecture, seminar, meeting or call, and it changes the defaults rather than just a label - segments(id, session_id, t_start, t_end, text, avg_logprob, no_speech_prob) · one per decoded chunk, keeping whatever confidence the decoder reported - cards(id, session_id, at, kind, text, source_segment_ids_json, shown_at, dismissed_at) · a live prompt, with the transcript it was built from - questions(id, session_id, text, answer, source_segment_ids_json, created_at) · the artefact that matters, answers stored separately from the question text - attempts(id, question_id, at, self_rating, next_due_at) · retrieval practice needs a record of the attempt, not a record of the reading - settings(panel_updates_per_minute, lecture_panel_default_off, audio_retention_days, interval_schedule_json) - There is no streak table, no score and no notification queue. A study tool that nags is a study tool uninstalled in week three. ### Phase 1 · Capture, and a transcript that never invents Build: List input devices and let the user pick, optionally mixing two into one stream. Run voice activity detection before the decoder and skip non-speech. Decode in overlapping chunks, merge on the overlap, and push each segment to the browser with its timestamps and confidence. Show a recording indicator that cannot be hidden, and store the consent note on the session before recording starts. Trap: Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. Done when: Chunk boundaries do not lose words, tested by decoding a known recording and diffing against a single-pass transcript. A stretch of silence produces no segments at all, tested with a recorded gap. Every segment carries its start, end and the decoder's confidence values. Recording cannot start without a consent note stored on the session, and the indicator is visible whenever capture is running. Do not build yet: No assistance panel, questions, review schedule or export yet. ### Phase 2 · A panel that does as little as possible Build: Schedule at most a configured number of model calls per minute and render at most a few cards at once. Provide a manual nudge. For sessions of kind lecture, start with the panel collapsed and let the nudge be the only way it opens. No sound, no animation, no colour that moves, no notification of any kind anywhere in the application. Trap: Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. Done when: Panel updates never exceed the configured rate even with a stubbed model that returns instantly, tested. A lecture session starts with the panel collapsed and a meeting session does not, tested from the session kind. No code path plays a sound, animates an element or raises a notification, checked by review and by a grep in the suite. Every card carries the segment identifiers it was built from and clicking one seeks the transcript to that point. Do not build yet: No questions, review schedule, export or retention yet. ### Phase 3 · The session ends in questions Build: Generate short-answer questions from the transcript, each tied to the segments it came from, with the answer stored separately and collapsed by default. Land on the question list when a session ends. Generate the summary as well, and put it behind a click. Discard any question whose cited segments do not resolve rather than displaying it. Trap: Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. Done when: Ending a session lands on the question list with every answer collapsed, checked by review. Every displayed question resolves its segment identifiers to real segments, and one that does not is discarded before display, tested with a fabricated response. Revealing an answer writes an attempt row with the self rating. The summary exists and is reachable in one click and is never the landing view. Do not build yet: No review schedule, export or retention yet. ### Phase 4 · Bringing them back, without nagging Build: Compute a next due date from the attempt and its self rating using an interval schedule held in a configuration file. Derive the due list from attempts rather than storing a queue. Show the next due date on every attempt so the schedule is visible. The only way to be reminded is opening the application, and there is no path that changes that. Trap: A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. Done when: An attempt writes a next due date computed from the configured schedule, tested at the first, middle and last interval. The due list is a query over attempts and no queue table exists, checked in the schema. No notification, email or scheduled message path exists anywhere, asserted in the suite. Exported questions carry their attempt history and reimport with the same due dates. Do not build yet: No export, retention or spend ceiling yet. ### Phase 5 · Leaving, and forgetting Build: Write a markdown file per session with the transcript, the questions and the summary, plus a JSON bundle of every table. Delete audio after a configured retention while keeping segments, questions and attempts. Delete a session entirely on request. Cap monthly model spend. Provide a mode where the local decoder runs and no request leaves the machine at all. Trap: Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows. Done when: Audio older than the retention is deleted and the segments, questions and attempts survive intact, tested. Deleting a session removes its rows and its markdown file, tested. The JSON bundle reimports into an empty database with matching counts and due dates, tested. With the network disabled the application records, transcribes and shows the transcript, and the panel says the assistance is unavailable rather than failing. Do not build yet: Nothing after this. ### Out of scope (and why) - A phone that keeps recording with the screen off, which is the actual product being sold and is weeks of background audio lifecycle work per platform. - Speaker separation, which needs a different model and a tuning loop, and which is the feature people expect first and forgive last. - Anything that supplies answers during an assessed exercise, which is not a feature this build is missing but one it refuses. - Calendar and conferencing integrations that join sessions for you, which is a permanent commitment to somebody else's API. ### README must contain - Why the end-of-session artefact is a question list rather than a summary, with the retention figures and the study behind them - Why the live panel has a hard update cap and is closed by default in lectures, with the multitasking study behind it - That there are no notifications anywhere, and that this is deliberate - The loopback device setup for capturing the other side of a call, per platform - How long audio is kept, what deleting a session removes, and where the markdown files live ===== AGENTS.md ===== # Agent instructions · Hedy indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and one page over a websocket, no build step, sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device, A local speech model with overlapping chunks and voice activity detection in front, SQLite for sessions, segments, cards and attempts, plus a markdown file per session, One model call per scheduled tick, rate limited above the model rather than below it. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. ===== BUILD_PLAN.md ===== # Build plan · Hedy A local study companion whose hard problems are producing an end-of-session artefact that actually aids retention rather than one that feels productive, keeping a live panel quiet enough not to cost more attention than it returns, and decoding audio without inventing sentences over silence. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Capture, and a transcript that never invents Audio is captured from a device you chose and decoded only where somebody is actually speaking. ### Steps 1. Let the user pick and mix input devices Capturing the other side of a call is a routing problem, not a code problem. Files: `app/audio.py`, `README.md` 2. Put voice activity detection in front of the decoder Silence in, invented sentences out. Files: `app/vad.py`, `tests/test_vad.py` 3. Overlap chunks and merge on the overlap Otherwise you lose a word every boundary and never notice. Files: `app/transcribe.py`, `tests/test_overlap.py` ### Done when - [ ] Chunk boundaries do not drop words - [ ] Silence produces no segments - [ ] Segments carry timestamps and confidence - [ ] No recording without a stored consent note ### Watch out - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. ## Phase 2 · A panel that does as little as possible The live assistance costs less attention than it returns, and in a lecture it is closed. ### Steps 1. Cap the rate above the model, not below it A cap in the renderer is not a cap. Files: `app/assist.py`, `tests/test_rate.py` 2. Default the panel closed for lectures The session kind is a design decision, not a label. Files: `app/sessions.py`, `web/panel.js` 3. Make every card cite its segments A prompt you cannot trace is a prompt you cannot check. Files: `app/assist.py` ### Done when - [ ] The update rate holds with an instant model - [ ] Lecture sessions start with the panel closed - [ ] Nothing sounds, animates or notifies - [ ] Cards cite segments and seek the transcript ### Watch out - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. ## Phase 3 · The session ends in questions Finishing a session produces short-answer questions with the answers hidden, and the summary is secondary. ### Steps 1. Land on questions, not on prose The default view is the design decision here. Files: `web/session_end.js`, `app/questions.py` 2. Store answers apart from questions So the interface cannot accidentally show both. Files: `app/db.py`, `tests/test_questions.py` 3. Discard questions that cannot cite A question about something nobody said is worse than no question. Files: `app/questions.py`, `tests/test_citation.py` ### Done when - [ ] A finished session lands on collapsed questions - [ ] Every question resolves to real segments - [ ] Revealing an answer records an attempt - [ ] The summary exists but is never the landing view ### Watch out - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. ## Phase 4 · Bringing them back, without nagging Questions come due on a schedule you can read, and nothing chases you. ### Steps 1. Put the intervals in a configuration file With the reasoning next to them, not in a commit message. Files: `config/intervals.json`, `app/review.py` 2. Derive the due list, do not store it A stored queue drifts from the attempts that produced it. Files: `app/review.py`, `tests/test_review.py` 3. Leave the reminding to the person The absence of a notification path is the feature. Files: `tests/test_no_notifications.py` ### Done when - [ ] Due dates follow the configured intervals - [ ] The due list is derived from attempts - [ ] No notification path exists - [ ] Exported questions keep their attempt history ### Watch out - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. ## Not in this build - Mobile background recording. - Speaker separation. - Anything that assists during an assessed exercise. - Calendar and conferencing integrations. ## After v1, if you want it - A second question type that asks you to explain rather than recall, generated from the same segments - A weekly view of what you failed to recall, which is the only honest measure of whether this is working - Importing a slide deck as session context, so the questions use the lecturer's own terms ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. The panel and question generation. Absent means local transcription only. MODEL_API_KEY=sk-...redacted # Optional. Stored on the session, because questions differ by model. MODEL_ID=a text model identifier # Required. The local decoder. Larger is slower and the panel cap assumes it keeps up. SPEECH_MODEL=small.en # Required. Sessions, segments, questions and attempts. The attempts are the irreplaceable part. DATABASE_PATH=./data/study.db # Required. One markdown file per session. These are meant to outlive the application. SESSIONS_DIR=./sessions # Required. How long recordings of other people stay on your disk. Short is the right default. AUDIO_RETENTION_DAYS=7 # Required. Hard cap enforced in the scheduler. Lower it before you raise it. PANEL_UPDATES_PER_MINUTE=3 # Required. Assistance stops at this figure and transcription carries on. MONTHLY_COST_CEILING=3.00
You are building a production product version of Hedy. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Hedy · product brief ## Problem The pieces are all commodity now: capture audio, stream it through a local Whisper variant, keep a rolling transcript, and every few seconds ask a model for the next good question. An agent will get you that in one sitting on a laptop, and the transcript-plus-summary half will genuinely be fine. What it will not get you is a phone in your pocket that survives being backgrounded during a two hour lecture, or sub-second suggestions that arrive before the moment passes. Live assistance is a latency and UX problem more than an AI problem, and that is exactly the part that takes weeks of fiddling. Build it if you mostly sit at a desk; keep paying if you mostly do not. ## Product outcome A study companion that ends every session with questions rather than prose, keeps its live panel quiet enough to be worth having open, never notifies you, and forgets the audio while keeping what you learned from it. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - A laptop with a decent CPU or GPU for local Whisper, or an API key for hosted transcription - An LLM API key in .env - A virtual audio device (BlackHole on macOS, WASAPI loopback on Windows) to capture the other side of the call - Consent from whoever you are recording, which is on you ## Explicit non-goals for v1 - Mobile background recording. - Speaker separation. - Anything that assists during an assessed exercise. - Calendar and conferencing integrations. - Mobile apps that keep recording reliably when the screen is off, which is where most of these sessions actually happen - Tuned latency: your DIY suggestions arrive a beat late, and a beat late is useless in conversation - Speaker diarization and meeting-type presets that shape the assistant for a sales call vs a lecture vs a doctor visit - Calendar and conferencing integrations that join and label sessions for you - Hosted, searchable history across every session with no laptop babysitting ## Success criteria - Silence is confirmed to produce no segments - The panel rate cap is confirmed to hold with an instant model - A finished session is confirmed to land on collapsed questions - No notification path is confirmed to exist - Audio expiry is confirmed to leave questions and attempts intact ===== BRIEF.md ===== # Build brief · Hedy The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a desk-bound study companion that keeps a live transcript, offers very few prompts while you are in the room, and turns the session into questions you have to answer to replace Hedy. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This one is aimed at students, so read the learning literature before you design the output, because the obvious build optimises the condition that loses. Roediger and Karpicke, Psychological Science 2006, gave students prose passages and had them either restudy or take recall tests the same number of times. Five minutes later restudying won, 81 percent of idea units recalled against 75. Two days later it reversed: 68 percent for the tested group against 54. One week later, 56 against 42. The tested group recalled as much after a week as the restudy group did after two days. An app whose output is a beautiful summary you reread is the restudy condition, and it feels better than the thing that works. So the artefact at the end of a session is a list of questions with the answers collapsed. The summary is generated too, it is one click away, and it is not the landing page. Second, the live panel. Sana, Weston and Cepeda, Computers and Education 2013, found that students who multitasked on a laptop during a lecture scored lower than those who did not, and that students merely in direct view of a multitasking peer also scored lower. A panel that updates whenever the model returns is a multitasking machine pointed at a lecture, and the row behind you can see it. Cap the update rate in the scheduler, default the panel closed for lecture sessions, and build no sound, no animation and no notification anywhere in this application. Third, the room. Tell people you are recording, and find out what your institution says about recording a lecture before you record one. Nothing here feeds you answers during an assessed exercise, and that is a refusal rather than a missing feature. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and one page over a websocket, no build step · this runs on your own machine and every dependency you add is one you maintain - Capture: sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device · the other side of a call only exists if you route it, and that is a documented setup step - Transcription: A local speech model with overlapping chunks and voice activity detection in front · decoding silence is where invented sentences come from - Storage: SQLite for sessions, segments, cards and attempts, plus a markdown file per session · the notes should outlive the application that made them - Assistance: One model call per scheduled tick, rate limited above the model rather than below it · the cap has to hold even when the model answers faster than you expected ### Data model (create this before Phase 1) - sessions(id, kind, started_at, ended_at, brief, consent_note, device, model_id) · kind is lecture, seminar, meeting or call, and it changes the defaults rather than just a label - segments(id, session_id, t_start, t_end, text, avg_logprob, no_speech_prob) · one per decoded chunk, keeping whatever confidence the decoder reported - cards(id, session_id, at, kind, text, source_segment_ids_json, shown_at, dismissed_at) · a live prompt, with the transcript it was built from - questions(id, session_id, text, answer, source_segment_ids_json, created_at) · the artefact that matters, answers stored separately from the question text - attempts(id, question_id, at, self_rating, next_due_at) · retrieval practice needs a record of the attempt, not a record of the reading - settings(panel_updates_per_minute, lecture_panel_default_off, audio_retention_days, interval_schedule_json) - There is no streak table, no score and no notification queue. A study tool that nags is a study tool uninstalled in week three. ### Phase 1 · Capture, and a transcript that never invents Build: List input devices and let the user pick, optionally mixing two into one stream. Run voice activity detection before the decoder and skip non-speech. Decode in overlapping chunks, merge on the overlap, and push each segment to the browser with its timestamps and confidence. Show a recording indicator that cannot be hidden, and store the consent note on the session before recording starts. Trap: Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. Done when: Chunk boundaries do not lose words, tested by decoding a known recording and diffing against a single-pass transcript. A stretch of silence produces no segments at all, tested with a recorded gap. Every segment carries its start, end and the decoder's confidence values. Recording cannot start without a consent note stored on the session, and the indicator is visible whenever capture is running. Do not build yet: No assistance panel, questions, review schedule or export yet. ### Phase 2 · A panel that does as little as possible Build: Schedule at most a configured number of model calls per minute and render at most a few cards at once. Provide a manual nudge. For sessions of kind lecture, start with the panel collapsed and let the nudge be the only way it opens. No sound, no animation, no colour that moves, no notification of any kind anywhere in the application. Trap: Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. Done when: Panel updates never exceed the configured rate even with a stubbed model that returns instantly, tested. A lecture session starts with the panel collapsed and a meeting session does not, tested from the session kind. No code path plays a sound, animates an element or raises a notification, checked by review and by a grep in the suite. Every card carries the segment identifiers it was built from and clicking one seeks the transcript to that point. Do not build yet: No questions, review schedule, export or retention yet. ### Phase 3 · The session ends in questions Build: Generate short-answer questions from the transcript, each tied to the segments it came from, with the answer stored separately and collapsed by default. Land on the question list when a session ends. Generate the summary as well, and put it behind a click. Discard any question whose cited segments do not resolve rather than displaying it. Trap: Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. Done when: Ending a session lands on the question list with every answer collapsed, checked by review. Every displayed question resolves its segment identifiers to real segments, and one that does not is discarded before display, tested with a fabricated response. Revealing an answer writes an attempt row with the self rating. The summary exists and is reachable in one click and is never the landing view. Do not build yet: No review schedule, export or retention yet. ### Phase 4 · Bringing them back, without nagging Build: Compute a next due date from the attempt and its self rating using an interval schedule held in a configuration file. Derive the due list from attempts rather than storing a queue. Show the next due date on every attempt so the schedule is visible. The only way to be reminded is opening the application, and there is no path that changes that. Trap: A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. Done when: An attempt writes a next due date computed from the configured schedule, tested at the first, middle and last interval. The due list is a query over attempts and no queue table exists, checked in the schema. No notification, email or scheduled message path exists anywhere, asserted in the suite. Exported questions carry their attempt history and reimport with the same due dates. Do not build yet: No export, retention or spend ceiling yet. ### Phase 5 · Leaving, and forgetting Build: Write a markdown file per session with the transcript, the questions and the summary, plus a JSON bundle of every table. Delete audio after a configured retention while keeping segments, questions and attempts. Delete a session entirely on request. Cap monthly model spend. Provide a mode where the local decoder runs and no request leaves the machine at all. Trap: Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows. Done when: Audio older than the retention is deleted and the segments, questions and attempts survive intact, tested. Deleting a session removes its rows and its markdown file, tested. The JSON bundle reimports into an empty database with matching counts and due dates, tested. With the network disabled the application records, transcribes and shows the transcript, and the panel says the assistance is unavailable rather than failing. Do not build yet: Nothing after this. ### Out of scope (and why) - A phone that keeps recording with the screen off, which is the actual product being sold and is weeks of background audio lifecycle work per platform. - Speaker separation, which needs a different model and a tuning loop, and which is the feature people expect first and forgive last. - Anything that supplies answers during an assessed exercise, which is not a feature this build is missing but one it refuses. - Calendar and conferencing integrations that join sessions for you, which is a permanent commitment to somebody else's API. ### README must contain - Why the end-of-session artefact is a question list rather than a summary, with the retention figures and the study behind them - Why the live panel has a hard update cap and is closed by default in lectures, with the multitasking study behind it - That there are no notifications anywhere, and that this is deliberate - The loopback device setup for capturing the other side of a call, per platform - How long audio is kept, what deleting a session removes, and where the markdown files live ===== ARCHITECTURE.md ===== # Architecture · Hedy ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and one page over a websocket, no build step | this runs on your own machine and every dependency you add is one you maintain | | Capture | sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device | the other side of a call only exists if you route it, and that is a documented setup step | | Transcription | A local speech model with overlapping chunks and voice activity detection in front | decoding silence is where invented sentences come from | | Storage | SQLite for sessions, segments, cards and attempts, plus a markdown file per session | the notes should outlive the application that made them | | Assistance | One model call per scheduled tick, rate limited above the model rather than below it | the cap has to hold even when the model answers faster than you expected | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Capture | Device selection, mixing and voice activity detection. | Any capture library, provided non-speech never reaches the decoder. | | Transcription | Overlapping decode, merging and confidence. | Any local model, provided segments keep their confidence. | | Panel | Scheduling, the rate cap and card citation. | Any presentation, provided the cap lives above the model. | | Questions | Generation, citation resolution and attempts. | Not swappable for a summary, which is the point of the build. | | Review | Intervals, due dates and the derived queue. | Any schedule, provided the intervals stay in a file you can read. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `MODEL_API_KEY` · optional, secret · The panel and question generation. Absent means local transcription only. - `MODEL_ID` · optional · Stored on the session, because questions differ by model. - `SPEECH_MODEL` · required · The local decoder. Larger is slower and the panel cap assumes it keeps up. - `DATABASE_PATH` · required · Sessions, segments, questions and attempts. The attempts are the irreplaceable part. - `SESSIONS_DIR` · required · One markdown file per session. These are meant to outlive the application. - `AUDIO_RETENTION_DAYS` · required · How long recordings of other people stay on your disk. Short is the right default. - `PANEL_UPDATES_PER_MINUTE` · required · Hard cap enforced in the scheduler. Lower it before you raise it. - `MONTHLY_COST_CEILING` · required · Assistance stops at this figure and transcription carries on. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Hedy product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with FastAPI and one page over a websocket, no build step, sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device, A local speech model with overlapping chunks and voice activity detection in front, SQLite for sessions, segments, cards and attempts, plus a markdown file per session, One model call per scheduled tick, rate limited above the model rather than below it. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. - Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows. ===== MILESTONES.md ===== # Delivery milestones · Hedy Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Capture, and a transcript that never invents Audio is captured from a device you chose and decoded only where somebody is actually speaking. ### Steps 1. Let the user pick and mix input devices Capturing the other side of a call is a routing problem, not a code problem. Files: `app/audio.py`, `README.md` 2. Put voice activity detection in front of the decoder Silence in, invented sentences out. Files: `app/vad.py`, `tests/test_vad.py` 3. Overlap chunks and merge on the overlap Otherwise you lose a word every boundary and never notice. Files: `app/transcribe.py`, `tests/test_overlap.py` ### Done when - [ ] Chunk boundaries do not drop words - [ ] Silence produces no segments - [ ] Segments carry timestamps and confidence - [ ] No recording without a stored consent note ### Watch out - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. ## M2 · A panel that does as little as possible The live assistance costs less attention than it returns, and in a lecture it is closed. ### Steps 1. Cap the rate above the model, not below it A cap in the renderer is not a cap. Files: `app/assist.py`, `tests/test_rate.py` 2. Default the panel closed for lectures The session kind is a design decision, not a label. Files: `app/sessions.py`, `web/panel.js` 3. Make every card cite its segments A prompt you cannot trace is a prompt you cannot check. Files: `app/assist.py` ### Done when - [ ] The update rate holds with an instant model - [ ] Lecture sessions start with the panel closed - [ ] Nothing sounds, animates or notifies - [ ] Cards cite segments and seek the transcript ### Watch out - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. ## M3 · The session ends in questions Finishing a session produces short-answer questions with the answers hidden, and the summary is secondary. ### Steps 1. Land on questions, not on prose The default view is the design decision here. Files: `web/session_end.js`, `app/questions.py` 2. Store answers apart from questions So the interface cannot accidentally show both. Files: `app/db.py`, `tests/test_questions.py` 3. Discard questions that cannot cite A question about something nobody said is worse than no question. Files: `app/questions.py`, `tests/test_citation.py` ### Done when - [ ] A finished session lands on collapsed questions - [ ] Every question resolves to real segments - [ ] Revealing an answer records an attempt - [ ] The summary exists but is never the landing view ### Watch out - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. ## M4 · Bringing them back, without nagging Questions come due on a schedule you can read, and nothing chases you. ### Steps 1. Put the intervals in a configuration file With the reasoning next to them, not in a commit message. Files: `config/intervals.json`, `app/review.py` 2. Derive the due list, do not store it A stored queue drifts from the attempts that produced it. Files: `app/review.py`, `tests/test_review.py` 3. Leave the reminding to the person The absence of a notification path is the feature. Files: `tests/test_no_notifications.py` ### Done when - [ ] Due dates follow the configured intervals - [ ] The due list is derived from attempts - [ ] No notification path exists - [ ] Exported questions keep their attempt history ### Watch out - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. ## M5 · Leaving, and forgetting (production only) Everything exports as plain files, the audio expires, and the whole thing runs with no network. ### Steps 1. Write a markdown file per session The notes should outlive the application that made them. Files: `app/export.py` 2. Expire the audio, keep the questions Short by default, because it is a recording of other people. Files: `app/retention.py`, `tests/test_retention.py` 3. Make the offline path a tested mode Local decoding is the whole reason to run this yourself. Files: `tests/test_offline.py`, `README.md` ### Done when - [ ] Audio expires and questions survive - [ ] A session delete removes rows and the file - [ ] The bundle reimports with matching due dates - [ ] Offline mode records and transcribes ### Watch out - Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows. ===== OPERATIONS.md ===== # Operations · Hedy ## Backup The database and the sessions directory. The attempts table is the part with no other copy: transcripts can be regenerated from audio you probably deleted, and questions can be regenerated, but your record of what you could and could not recall cannot. ## Restore Restore both, confirm the question count and that due dates recompute to the same values, then check that the markdown files match the sessions in the database. A restore that loses attempts has reset your revision schedule to nothing. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Monthly model spend against the ceiling, decode speed against real time, the proportion of generated questions discarded for failing citation, and the panel update rate under a fast model. A rising discard rate means the transcript quality dropped, usually because the decoder fell behind. ## Incident checklist The failure that matters is a question with a wrong answer that you then practise. Every question carries the segments it came from, so a suspect answer can be checked against what was actually said, and a question can be deleted along with its attempts. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Silence is confirmed to produce no segments - [ ] The panel rate cap is confirmed to hold with an instant model - [ ] A finished session is confirmed to land on collapsed questions - [ ] No notification path is confirmed to exist - [ ] Audio expiry is confirmed to leave questions and attempts intact ## Launch constraint Do not market omitted Hedy capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. The panel and question generation. Absent means local transcription only. MODEL_API_KEY=sk-...redacted # Optional. Stored on the session, because questions differ by model. MODEL_ID=a text model identifier # Required. The local decoder. Larger is slower and the panel cap assumes it keeps up. SPEECH_MODEL=small.en # Required. Sessions, segments, questions and attempts. The attempts are the irreplaceable part. DATABASE_PATH=./data/study.db # Required. One markdown file per session. These are meant to outlive the application. SESSIONS_DIR=./sessions # Required. How long recordings of other people stay on your disk. Short is the right default. AUDIO_RETENTION_DAYS=7 # Required. Hard cap enforced in the scheduler. Lower it before you raise it. PANEL_UPDATES_PER_MINUTE=3 # Required. Assistance stops at this figure and transcription carries on. MONTHLY_COST_CEILING=3.00
README.md
# Hedy · indie build A local study companion whose hard problems are producing an end-of-session artefact that actually aids retention rather than one that feels productive, keeping a live panel quiet enough not to cost more attention than it returns, and decoding audio without inventing sentences over silence. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and one page over a websocket, no build step | this runs on your own machine and every dependency you add is one you maintain | | Capture | sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device | the other side of a call only exists if you route it, and that is a documented setup step | | Transcription | A local speech model with overlapping chunks and voice activity detection in front | decoding silence is where invented sentences come from | | Storage | SQLite for sessions, segments, cards and attempts, plus a markdown file per session | the notes should outlive the application that made them | | Assistance | One model call per scheduled tick, rate limited above the model rather than below it | the cap has to hold even when the model answers faster than you expected | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the capture loop, the decoder and the server. - Get it: python.org or your package manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A local speech recognition model and its runtime** · free, after the download - Why: Everything else in this build assumes the audio never leaves the machine, and a hosted transcription service would make that untrue. - Get it: Install the runtime and download a small English model on first run. - Verify: A one minute recording transcribes faster than real time on your machine - [ ] **A loopback audio device** (optional) · free - Why: Without one you capture only your own microphone, which is half a conversation and makes the transcript useless for a call. - Get it: A virtual audio device for your platform, installed and selected as an input. - Verify: It appears in the device list and a played recording produces segments - [ ] **An API key for a text model** · pay per call; the rate cap keeps it small - Why: The panel and the question generation are model calls, and they are the only networked part. - Get it: Create a key with your preferred provider and set a spend limit in their console. - Verify: Ending a test session produces questions that cite real segments - [ ] **What your institution says about recording teaching** · free - Why: Recording a lecture is not only a question between you and the lecturer, and finding out afterwards is the wrong order. - Get it: Read the recording policy where you study and write down what it permits. - Verify: A note in the README states the policy you checked and the date ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Mobile background recording. - Speaker separation. - Anything that assists during an assessed exercise. - Calendar and conferencing integrations. - Mobile apps that keep recording reliably when the screen is off, which is where most of these sessions actually happen - Tuned latency: your DIY suggestions arrive a beat late, and a beat late is useless in conversation - Speaker diarization and meeting-type presets that shape the assistant for a sales call vs a lecture vs a doctor visit - Calendar and conferencing integrations that join and label sessions for you - Hosted, searchable history across every session with no laptop babysitting If one of those is essential to you, that is the reason to keep paying for Hedy, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Hedy The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a desk-bound study companion that keeps a live transcript, offers very few prompts while you are in the room, and turns the session into questions you have to answer to replace Hedy. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This one is aimed at students, so read the learning literature before you design the output, because the obvious build optimises the condition that loses. Roediger and Karpicke, Psychological Science 2006, gave students prose passages and had them either restudy or take recall tests the same number of times. Five minutes later restudying won, 81 percent of idea units recalled against 75. Two days later it reversed: 68 percent for the tested group against 54. One week later, 56 against 42. The tested group recalled as much after a week as the restudy group did after two days. An app whose output is a beautiful summary you reread is the restudy condition, and it feels better than the thing that works. So the artefact at the end of a session is a list of questions with the answers collapsed. The summary is generated too, it is one click away, and it is not the landing page. Second, the live panel. Sana, Weston and Cepeda, Computers and Education 2013, found that students who multitasked on a laptop during a lecture scored lower than those who did not, and that students merely in direct view of a multitasking peer also scored lower. A panel that updates whenever the model returns is a multitasking machine pointed at a lecture, and the row behind you can see it. Cap the update rate in the scheduler, default the panel closed for lecture sessions, and build no sound, no animation and no notification anywhere in this application. Third, the room. Tell people you are recording, and find out what your institution says about recording a lecture before you record one. Nothing here feeds you answers during an assessed exercise, and that is a refusal rather than a missing feature. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and one page over a websocket, no build step · this runs on your own machine and every dependency you add is one you maintain - Capture: sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device · the other side of a call only exists if you route it, and that is a documented setup step - Transcription: A local speech model with overlapping chunks and voice activity detection in front · decoding silence is where invented sentences come from - Storage: SQLite for sessions, segments, cards and attempts, plus a markdown file per session · the notes should outlive the application that made them - Assistance: One model call per scheduled tick, rate limited above the model rather than below it · the cap has to hold even when the model answers faster than you expected ### Data model (create this before Phase 1) - sessions(id, kind, started_at, ended_at, brief, consent_note, device, model_id) · kind is lecture, seminar, meeting or call, and it changes the defaults rather than just a label - segments(id, session_id, t_start, t_end, text, avg_logprob, no_speech_prob) · one per decoded chunk, keeping whatever confidence the decoder reported - cards(id, session_id, at, kind, text, source_segment_ids_json, shown_at, dismissed_at) · a live prompt, with the transcript it was built from - questions(id, session_id, text, answer, source_segment_ids_json, created_at) · the artefact that matters, answers stored separately from the question text - attempts(id, question_id, at, self_rating, next_due_at) · retrieval practice needs a record of the attempt, not a record of the reading - settings(panel_updates_per_minute, lecture_panel_default_off, audio_retention_days, interval_schedule_json) - There is no streak table, no score and no notification queue. A study tool that nags is a study tool uninstalled in week three. ### Phase 1 · Capture, and a transcript that never invents Build: List input devices and let the user pick, optionally mixing two into one stream. Run voice activity detection before the decoder and skip non-speech. Decode in overlapping chunks, merge on the overlap, and push each segment to the browser with its timestamps and confidence. Show a recording indicator that cannot be hidden, and store the consent note on the session before recording starts. Trap: Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. Done when: Chunk boundaries do not lose words, tested by decoding a known recording and diffing against a single-pass transcript. A stretch of silence produces no segments at all, tested with a recorded gap. Every segment carries its start, end and the decoder's confidence values. Recording cannot start without a consent note stored on the session, and the indicator is visible whenever capture is running. Do not build yet: No assistance panel, questions, review schedule or export yet. ### Phase 2 · A panel that does as little as possible Build: Schedule at most a configured number of model calls per minute and render at most a few cards at once. Provide a manual nudge. For sessions of kind lecture, start with the panel collapsed and let the nudge be the only way it opens. No sound, no animation, no colour that moves, no notification of any kind anywhere in the application. Trap: Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. Done when: Panel updates never exceed the configured rate even with a stubbed model that returns instantly, tested. A lecture session starts with the panel collapsed and a meeting session does not, tested from the session kind. No code path plays a sound, animates an element or raises a notification, checked by review and by a grep in the suite. Every card carries the segment identifiers it was built from and clicking one seeks the transcript to that point. Do not build yet: No questions, review schedule, export or retention yet. ### Phase 3 · The session ends in questions Build: Generate short-answer questions from the transcript, each tied to the segments it came from, with the answer stored separately and collapsed by default. Land on the question list when a session ends. Generate the summary as well, and put it behind a click. Discard any question whose cited segments do not resolve rather than displaying it. Trap: Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. Done when: Ending a session lands on the question list with every answer collapsed, checked by review. Every displayed question resolves its segment identifiers to real segments, and one that does not is discarded before display, tested with a fabricated response. Revealing an answer writes an attempt row with the self rating. The summary exists and is reachable in one click and is never the landing view. Do not build yet: No review schedule, export or retention yet. ### Phase 4 · Bringing them back, without nagging Build: Compute a next due date from the attempt and its self rating using an interval schedule held in a configuration file. Derive the due list from attempts rather than storing a queue. Show the next due date on every attempt so the schedule is visible. The only way to be reminded is opening the application, and there is no path that changes that. Trap: A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. Done when: An attempt writes a next due date computed from the configured schedule, tested at the first, middle and last interval. The due list is a query over attempts and no queue table exists, checked in the schema. No notification, email or scheduled message path exists anywhere, asserted in the suite. Exported questions carry their attempt history and reimport with the same due dates. Do not build yet: No export, retention or spend ceiling yet. ### Phase 5 · Leaving, and forgetting Build: Write a markdown file per session with the transcript, the questions and the summary, plus a JSON bundle of every table. Delete audio after a configured retention while keeping segments, questions and attempts. Delete a session entirely on request. Cap monthly model spend. Provide a mode where the local decoder runs and no request leaves the machine at all. Trap: Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows. Done when: Audio older than the retention is deleted and the segments, questions and attempts survive intact, tested. Deleting a session removes its rows and its markdown file, tested. The JSON bundle reimports into an empty database with matching counts and due dates, tested. With the network disabled the application records, transcribes and shows the transcript, and the panel says the assistance is unavailable rather than failing. Do not build yet: Nothing after this. ### Out of scope (and why) - A phone that keeps recording with the screen off, which is the actual product being sold and is weeks of background audio lifecycle work per platform. - Speaker separation, which needs a different model and a tuning loop, and which is the feature people expect first and forgive last. - Anything that supplies answers during an assessed exercise, which is not a feature this build is missing but one it refuses. - Calendar and conferencing integrations that join sessions for you, which is a permanent commitment to somebody else's API. ### README must contain - Why the end-of-session artefact is a question list rather than a summary, with the retention figures and the study behind them - Why the live panel has a hard update cap and is closed by default in lectures, with the multitasking study behind it - That there are no notifications anywhere, and that this is deliberate - The loopback device setup for capturing the other side of a call, per platform - How long audio is kept, what deleting a session removes, and where the markdown files live
AGENTS.md
# Agent instructions · Hedy indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and one page over a websocket, no build step, sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device, A local speech model with overlapping chunks and voice activity detection in front, SQLite for sessions, segments, cards and attempts, plus a markdown file per session, One model call per scheduled tick, rate limited above the model rather than below it. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person.
BUILD_PLAN.md
# Build plan · Hedy A local study companion whose hard problems are producing an end-of-session artefact that actually aids retention rather than one that feels productive, keeping a live panel quiet enough not to cost more attention than it returns, and decoding audio without inventing sentences over silence. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Capture, and a transcript that never invents Audio is captured from a device you chose and decoded only where somebody is actually speaking. ### Steps 1. Let the user pick and mix input devices Capturing the other side of a call is a routing problem, not a code problem. Files: `app/audio.py`, `README.md` 2. Put voice activity detection in front of the decoder Silence in, invented sentences out. Files: `app/vad.py`, `tests/test_vad.py` 3. Overlap chunks and merge on the overlap Otherwise you lose a word every boundary and never notice. Files: `app/transcribe.py`, `tests/test_overlap.py` ### Done when - [ ] Chunk boundaries do not drop words - [ ] Silence produces no segments - [ ] Segments carry timestamps and confidence - [ ] No recording without a stored consent note ### Watch out - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. ## Phase 2 · A panel that does as little as possible The live assistance costs less attention than it returns, and in a lecture it is closed. ### Steps 1. Cap the rate above the model, not below it A cap in the renderer is not a cap. Files: `app/assist.py`, `tests/test_rate.py` 2. Default the panel closed for lectures The session kind is a design decision, not a label. Files: `app/sessions.py`, `web/panel.js` 3. Make every card cite its segments A prompt you cannot trace is a prompt you cannot check. Files: `app/assist.py` ### Done when - [ ] The update rate holds with an instant model - [ ] Lecture sessions start with the panel closed - [ ] Nothing sounds, animates or notifies - [ ] Cards cite segments and seek the transcript ### Watch out - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. ## Phase 3 · The session ends in questions Finishing a session produces short-answer questions with the answers hidden, and the summary is secondary. ### Steps 1. Land on questions, not on prose The default view is the design decision here. Files: `web/session_end.js`, `app/questions.py` 2. Store answers apart from questions So the interface cannot accidentally show both. Files: `app/db.py`, `tests/test_questions.py` 3. Discard questions that cannot cite A question about something nobody said is worse than no question. Files: `app/questions.py`, `tests/test_citation.py` ### Done when - [ ] A finished session lands on collapsed questions - [ ] Every question resolves to real segments - [ ] Revealing an answer records an attempt - [ ] The summary exists but is never the landing view ### Watch out - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. ## Phase 4 · Bringing them back, without nagging Questions come due on a schedule you can read, and nothing chases you. ### Steps 1. Put the intervals in a configuration file With the reasoning next to them, not in a commit message. Files: `config/intervals.json`, `app/review.py` 2. Derive the due list, do not store it A stored queue drifts from the attempts that produced it. Files: `app/review.py`, `tests/test_review.py` 3. Leave the reminding to the person The absence of a notification path is the feature. Files: `tests/test_no_notifications.py` ### Done when - [ ] Due dates follow the configured intervals - [ ] The due list is derived from attempts - [ ] No notification path exists - [ ] Exported questions keep their attempt history ### Watch out - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. ## Not in this build - Mobile background recording. - Speaker separation. - Anything that assists during an assessed exercise. - Calendar and conferencing integrations. ## After v1, if you want it - A second question type that asks you to explain rather than recall, generated from the same segments - A weekly view of what you failed to recall, which is the only honest measure of whether this is working - Importing a slide deck as session context, so the questions use the lecturer's own terms
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. The panel and question generation. Absent means local transcription only. MODEL_API_KEY=sk-...redacted # Optional. Stored on the session, because questions differ by model. MODEL_ID=a text model identifier # Required. The local decoder. Larger is slower and the panel cap assumes it keeps up. SPEECH_MODEL=small.en # Required. Sessions, segments, questions and attempts. The attempts are the irreplaceable part. DATABASE_PATH=./data/study.db # Required. One markdown file per session. These are meant to outlive the application. SESSIONS_DIR=./sessions # Required. How long recordings of other people stay on your disk. Short is the right default. AUDIO_RETENTION_DAYS=7 # Required. Hard cap enforced in the scheduler. Lower it before you raise it. PANEL_UPDATES_PER_MINUTE=3 # Required. Assistance stops at this figure and transcription carries on. MONTHLY_COST_CEILING=3.00
PRODUCT.md
# Hedy · product brief ## Problem The pieces are all commodity now: capture audio, stream it through a local Whisper variant, keep a rolling transcript, and every few seconds ask a model for the next good question. An agent will get you that in one sitting on a laptop, and the transcript-plus-summary half will genuinely be fine. What it will not get you is a phone in your pocket that survives being backgrounded during a two hour lecture, or sub-second suggestions that arrive before the moment passes. Live assistance is a latency and UX problem more than an AI problem, and that is exactly the part that takes weeks of fiddling. Build it if you mostly sit at a desk; keep paying if you mostly do not. ## Product outcome A study companion that ends every session with questions rather than prose, keeps its live panel quiet enough to be worth having open, never notifies you, and forgets the audio while keeping what you learned from it. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - A laptop with a decent CPU or GPU for local Whisper, or an API key for hosted transcription - An LLM API key in .env - A virtual audio device (BlackHole on macOS, WASAPI loopback on Windows) to capture the other side of the call - Consent from whoever you are recording, which is on you ## Explicit non-goals for v1 - Mobile background recording. - Speaker separation. - Anything that assists during an assessed exercise. - Calendar and conferencing integrations. - Mobile apps that keep recording reliably when the screen is off, which is where most of these sessions actually happen - Tuned latency: your DIY suggestions arrive a beat late, and a beat late is useless in conversation - Speaker diarization and meeting-type presets that shape the assistant for a sales call vs a lecture vs a doctor visit - Calendar and conferencing integrations that join and label sessions for you - Hosted, searchable history across every session with no laptop babysitting ## Success criteria - Silence is confirmed to produce no segments - The panel rate cap is confirmed to hold with an instant model - A finished session is confirmed to land on collapsed questions - No notification path is confirmed to exist - Audio expiry is confirmed to leave questions and attempts intact
BRIEF.md
# Build brief · Hedy The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a desk-bound study companion that keeps a live transcript, offers very few prompts while you are in the room, and turns the session into questions you have to answer to replace Hedy. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero This one is aimed at students, so read the learning literature before you design the output, because the obvious build optimises the condition that loses. Roediger and Karpicke, Psychological Science 2006, gave students prose passages and had them either restudy or take recall tests the same number of times. Five minutes later restudying won, 81 percent of idea units recalled against 75. Two days later it reversed: 68 percent for the tested group against 54. One week later, 56 against 42. The tested group recalled as much after a week as the restudy group did after two days. An app whose output is a beautiful summary you reread is the restudy condition, and it feels better than the thing that works. So the artefact at the end of a session is a list of questions with the answers collapsed. The summary is generated too, it is one click away, and it is not the landing page. Second, the live panel. Sana, Weston and Cepeda, Computers and Education 2013, found that students who multitasked on a laptop during a lecture scored lower than those who did not, and that students merely in direct view of a multitasking peer also scored lower. A panel that updates whenever the model returns is a multitasking machine pointed at a lecture, and the row behind you can see it. Cap the update rate in the scheduler, default the panel closed for lecture sessions, and build no sound, no animation and no notification anywhere in this application. Third, the room. Tell people you are recording, and find out what your institution says about recording a lecture before you record one. Nothing here feeds you answers during an assessed exercise, and that is a refusal rather than a missing feature. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and one page over a websocket, no build step · this runs on your own machine and every dependency you add is one you maintain - Capture: sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device · the other side of a call only exists if you route it, and that is a documented setup step - Transcription: A local speech model with overlapping chunks and voice activity detection in front · decoding silence is where invented sentences come from - Storage: SQLite for sessions, segments, cards and attempts, plus a markdown file per session · the notes should outlive the application that made them - Assistance: One model call per scheduled tick, rate limited above the model rather than below it · the cap has to hold even when the model answers faster than you expected ### Data model (create this before Phase 1) - sessions(id, kind, started_at, ended_at, brief, consent_note, device, model_id) · kind is lecture, seminar, meeting or call, and it changes the defaults rather than just a label - segments(id, session_id, t_start, t_end, text, avg_logprob, no_speech_prob) · one per decoded chunk, keeping whatever confidence the decoder reported - cards(id, session_id, at, kind, text, source_segment_ids_json, shown_at, dismissed_at) · a live prompt, with the transcript it was built from - questions(id, session_id, text, answer, source_segment_ids_json, created_at) · the artefact that matters, answers stored separately from the question text - attempts(id, question_id, at, self_rating, next_due_at) · retrieval practice needs a record of the attempt, not a record of the reading - settings(panel_updates_per_minute, lecture_panel_default_off, audio_retention_days, interval_schedule_json) - There is no streak table, no score and no notification queue. A study tool that nags is a study tool uninstalled in week three. ### Phase 1 · Capture, and a transcript that never invents Build: List input devices and let the user pick, optionally mixing two into one stream. Run voice activity detection before the decoder and skip non-speech. Decode in overlapping chunks, merge on the overlap, and push each segment to the browser with its timestamps and confidence. Show a recording indicator that cannot be hidden, and store the consent note on the session before recording starts. Trap: Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. Done when: Chunk boundaries do not lose words, tested by decoding a known recording and diffing against a single-pass transcript. A stretch of silence produces no segments at all, tested with a recorded gap. Every segment carries its start, end and the decoder's confidence values. Recording cannot start without a consent note stored on the session, and the indicator is visible whenever capture is running. Do not build yet: No assistance panel, questions, review schedule or export yet. ### Phase 2 · A panel that does as little as possible Build: Schedule at most a configured number of model calls per minute and render at most a few cards at once. Provide a manual nudge. For sessions of kind lecture, start with the panel collapsed and let the nudge be the only way it opens. No sound, no animation, no colour that moves, no notification of any kind anywhere in the application. Trap: Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. Done when: Panel updates never exceed the configured rate even with a stubbed model that returns instantly, tested. A lecture session starts with the panel collapsed and a meeting session does not, tested from the session kind. No code path plays a sound, animates an element or raises a notification, checked by review and by a grep in the suite. Every card carries the segment identifiers it was built from and clicking one seeks the transcript to that point. Do not build yet: No questions, review schedule, export or retention yet. ### Phase 3 · The session ends in questions Build: Generate short-answer questions from the transcript, each tied to the segments it came from, with the answer stored separately and collapsed by default. Land on the question list when a session ends. Generate the summary as well, and put it behind a click. Discard any question whose cited segments do not resolve rather than displaying it. Trap: Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. Done when: Ending a session lands on the question list with every answer collapsed, checked by review. Every displayed question resolves its segment identifiers to real segments, and one that does not is discarded before display, tested with a fabricated response. Revealing an answer writes an attempt row with the self rating. The summary exists and is reachable in one click and is never the landing view. Do not build yet: No review schedule, export or retention yet. ### Phase 4 · Bringing them back, without nagging Build: Compute a next due date from the attempt and its self rating using an interval schedule held in a configuration file. Derive the due list from attempts rather than storing a queue. Show the next due date on every attempt so the schedule is visible. The only way to be reminded is opening the application, and there is no path that changes that. Trap: A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. Done when: An attempt writes a next due date computed from the configured schedule, tested at the first, middle and last interval. The due list is a query over attempts and no queue table exists, checked in the schema. No notification, email or scheduled message path exists anywhere, asserted in the suite. Exported questions carry their attempt history and reimport with the same due dates. Do not build yet: No export, retention or spend ceiling yet. ### Phase 5 · Leaving, and forgetting Build: Write a markdown file per session with the transcript, the questions and the summary, plus a JSON bundle of every table. Delete audio after a configured retention while keeping segments, questions and attempts. Delete a session entirely on request. Cap monthly model spend. Provide a mode where the local decoder runs and no request leaves the machine at all. Trap: Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows. Done when: Audio older than the retention is deleted and the segments, questions and attempts survive intact, tested. Deleting a session removes its rows and its markdown file, tested. The JSON bundle reimports into an empty database with matching counts and due dates, tested. With the network disabled the application records, transcribes and shows the transcript, and the panel says the assistance is unavailable rather than failing. Do not build yet: Nothing after this. ### Out of scope (and why) - A phone that keeps recording with the screen off, which is the actual product being sold and is weeks of background audio lifecycle work per platform. - Speaker separation, which needs a different model and a tuning loop, and which is the feature people expect first and forgive last. - Anything that supplies answers during an assessed exercise, which is not a feature this build is missing but one it refuses. - Calendar and conferencing integrations that join sessions for you, which is a permanent commitment to somebody else's API. ### README must contain - Why the end-of-session artefact is a question list rather than a summary, with the retention figures and the study behind them - Why the live panel has a hard update cap and is closed by default in lectures, with the multitasking study behind it - That there are no notifications anywhere, and that this is deliberate - The loopback device setup for capturing the other side of a call, per platform - How long audio is kept, what deleting a session removes, and where the markdown files live
ARCHITECTURE.md
# Architecture · Hedy ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and one page over a websocket, no build step | this runs on your own machine and every dependency you add is one you maintain | | Capture | sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device | the other side of a call only exists if you route it, and that is a documented setup step | | Transcription | A local speech model with overlapping chunks and voice activity detection in front | decoding silence is where invented sentences come from | | Storage | SQLite for sessions, segments, cards and attempts, plus a markdown file per session | the notes should outlive the application that made them | | Assistance | One model call per scheduled tick, rate limited above the model rather than below it | the cap has to hold even when the model answers faster than you expected | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Capture | Device selection, mixing and voice activity detection. | Any capture library, provided non-speech never reaches the decoder. | | Transcription | Overlapping decode, merging and confidence. | Any local model, provided segments keep their confidence. | | Panel | Scheduling, the rate cap and card citation. | Any presentation, provided the cap lives above the model. | | Questions | Generation, citation resolution and attempts. | Not swappable for a summary, which is the point of the build. | | Review | Intervals, due dates and the derived queue. | Any schedule, provided the intervals stay in a file you can read. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `MODEL_API_KEY` · optional, secret · The panel and question generation. Absent means local transcription only. - `MODEL_ID` · optional · Stored on the session, because questions differ by model. - `SPEECH_MODEL` · required · The local decoder. Larger is slower and the panel cap assumes it keeps up. - `DATABASE_PATH` · required · Sessions, segments, questions and attempts. The attempts are the irreplaceable part. - `SESSIONS_DIR` · required · One markdown file per session. These are meant to outlive the application. - `AUDIO_RETENTION_DAYS` · required · How long recordings of other people stay on your disk. Short is the right default. - `PANEL_UPDATES_PER_MINUTE` · required · Hard cap enforced in the scheduler. Lower it before you raise it. - `MONTHLY_COST_CEILING` · required · Assistance stops at this figure and transcription carries on. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Hedy product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with FastAPI and one page over a websocket, no build step, sounddevice at 16 kHz mono, optionally mixing a microphone and a loopback device, A local speech model with overlapping chunks and voice activity detection in front, SQLite for sessions, segments, cards and attempts, plus a markdown file per session, One model call per scheduled tick, rate limited above the model rather than below it. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. - Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows.
MILESTONES.md
# Delivery milestones · Hedy Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Capture, and a transcript that never invents Audio is captured from a device you chose and decoded only where somebody is actually speaking. ### Steps 1. Let the user pick and mix input devices Capturing the other side of a call is a routing problem, not a code problem. Files: `app/audio.py`, `README.md` 2. Put voice activity detection in front of the decoder Silence in, invented sentences out. Files: `app/vad.py`, `tests/test_vad.py` 3. Overlap chunks and merge on the overlap Otherwise you lose a word every boundary and never notice. Files: `app/transcribe.py`, `tests/test_overlap.py` ### Done when - [ ] Chunk boundaries do not drop words - [ ] Silence produces no segments - [ ] Segments carry timestamps and confidence - [ ] No recording without a stored consent note ### Watch out - Decoding fixed chunks with no overlap chops a word at every boundary, so the panel reasons over a transcript that is missing a word every few seconds and you will blame the model for it. Overlap the chunks by a second and merge on the overlap, and drop non-speech before it reaches the decoder rather than after, because long silences are exactly the input that produces fluent sentences nobody said. ## M2 · A panel that does as little as possible The live assistance costs less attention than it returns, and in a lecture it is closed. ### Steps 1. Cap the rate above the model, not below it A cap in the renderer is not a cap. Files: `app/assist.py`, `tests/test_rate.py` 2. Default the panel closed for lectures The session kind is a design decision, not a label. Files: `app/sessions.py`, `web/panel.js` 3. Make every card cite its segments A prompt you cannot trace is a prompt you cannot check. Files: `app/assist.py` ### Done when - [ ] The update rate holds with an instant model - [ ] Lecture sessions start with the panel closed - [ ] Nothing sounds, animates or notifies - [ ] Cards cite segments and seek the transcript ### Watch out - Rendering a card whenever the model returns turns the session into the laptop multitasking condition that Sana and colleagues measured a cost for, both for the person with the laptop and for whoever can see the screen, and the cost lands on the lecture you are attending rather than the transcript you will read later. Cap the rate in the scheduler so the cap holds when the model is fast, default the panel closed for lectures, and time how long you actually spend looking at it before deciding the cap is too low. ## M3 · The session ends in questions Finishing a session produces short-answer questions with the answers hidden, and the summary is secondary. ### Steps 1. Land on questions, not on prose The default view is the design decision here. Files: `web/session_end.js`, `app/questions.py` 2. Store answers apart from questions So the interface cannot accidentally show both. Files: `app/db.py`, `tests/test_questions.py` 3. Discard questions that cannot cite A question about something nobody said is worse than no question. Files: `app/questions.py`, `tests/test_citation.py` ### Done when - [ ] A finished session lands on collapsed questions - [ ] Every question resolves to real segments - [ ] Revealing an answer records an attempt - [ ] The summary exists but is never the landing view ### Watch out - Generating a tidy summary and calling it revision optimises the condition that lost by fourteen points at one week in Roediger and Karpicke, and it will feel like the better product because rereading feels fluent and recall does not. Make the question list the page you land on, keep answers collapsed until an attempt has been recorded, and let the summary exist as the second tab rather than the first. ## M4 · Bringing them back, without nagging Questions come due on a schedule you can read, and nothing chases you. ### Steps 1. Put the intervals in a configuration file With the reasoning next to them, not in a commit message. Files: `config/intervals.json`, `app/review.py` 2. Derive the due list, do not store it A stored queue drifts from the attempts that produced it. Files: `app/review.py`, `tests/test_review.py` 3. Leave the reminding to the person The absence of a notification path is the feature. Files: `tests/test_no_notifications.py` ### Done when - [ ] Due dates follow the configured intervals - [ ] The due list is derived from attempts - [ ] No notification path exists - [ ] Exported questions keep their attempt history ### Watch out - A review queue that sends a notification is the feature that gets a study app deleted in the second week, and an interval schedule hidden in code is one you cannot argue with when it feels wrong. Keep the intervals in a configuration file with the reasoning written beside them, show the next due date after every attempt, and leave the reminding to the person. ## M5 · Leaving, and forgetting (production only) Everything exports as plain files, the audio expires, and the whole thing runs with no network. ### Steps 1. Write a markdown file per session The notes should outlive the application that made them. Files: `app/export.py` 2. Expire the audio, keep the questions Short by default, because it is a recording of other people. Files: `app/retention.py`, `tests/test_retention.py` 3. Make the offline path a tested mode Local decoding is the whole reason to run this yourself. Files: `tests/test_offline.py`, `README.md` ### Done when - [ ] Audio expires and questions survive - [ ] A session delete removes rows and the file - [ ] The bundle reimports with matching due dates - [ ] Offline mode records and transcribes ### Watch out - Keeping the audio indefinitely because disk is cheap leaves a year of other people's voices on a laptop that travels, and the audio stops being useful long before the questions do. Set the retention short by default, make audio deletion keep the segments and questions intact, and make a full session delete remove the markdown file as well as the rows.
OPERATIONS.md
# Operations · Hedy ## Backup The database and the sessions directory. The attempts table is the part with no other copy: transcripts can be regenerated from audio you probably deleted, and questions can be regenerated, but your record of what you could and could not recall cannot. ## Restore Restore both, confirm the question count and that due dates recompute to the same values, then check that the markdown files match the sessions in the database. A restore that loses attempts has reset your revision schedule to nothing. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Monthly model spend against the ceiling, decode speed against real time, the proportion of generated questions discarded for failing citation, and the panel update rate under a fast model. A rising discard rate means the transcript quality dropped, usually because the decoder fell behind. ## Incident checklist The failure that matters is a question with a wrong answer that you then practise. Every question carries the segments it came from, so a suspect answer can be checked against what was actually said, and a question can be deleted along with its attempts. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Silence is confirmed to produce no segments - [ ] The panel rate cap is confirmed to hold with an instant model - [ ] A finished session is confirmed to land on collapsed questions - [ ] No notification path is confirmed to exist - [ ] Audio expiry is confirmed to leave questions and attempts intact ## Launch constraint Do not market omitted Hedy capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Optional · secret. The panel and question generation. Absent means local transcription only. MODEL_API_KEY=sk-...redacted # Optional. Stored on the session, because questions differ by model. MODEL_ID=a text model identifier # Required. The local decoder. Larger is slower and the panel cap assumes it keeps up. SPEECH_MODEL=small.en # Required. Sessions, segments, questions and attempts. The attempts are the irreplaceable part. DATABASE_PATH=./data/study.db # Required. One markdown file per session. These are meant to outlive the application. SESSIONS_DIR=./sessions # Required. How long recordings of other people stay on your disk. Short is the right default. AUDIO_RETENTION_DAYS=7 # Required. Hard cap enforced in the scheduler. Lower it before you raise it. PANEL_UPDATES_PER_MINUTE=3 # Required. Assistance stops at this figure and transcription carries on. MONTHLY_COST_CEILING=3.00
$ choose a build depth, inspect the files, then open the complete pack in your agent
Because real-time is unforgiving. A post-hoc summarizer can be sloppy and still useful, but a live copilot that lags four seconds or drops audio when you switch apps is worse than nothing, and people pay to not think about that. The mobile side compounds it: background audio on iOS is a permissions and lifecycle minefield that nobody wants to solve for themselves. Add the meeting-type presets and calendar hookups and the subscription is mostly buying tuning you would otherwise do by hand for a month.
xMobile apps that keep recording reliably when the screen is off, which is where most of these sessions actually happen
xTuned latency: your DIY suggestions arrive a beat late, and a beat late is useless in conversation
xSpeaker diarization and meeting-type presets that shape the assistant for a sales call vs a lecture vs a doctor visit
xCalendar and conferencing integrations that join and label sessions for you
xHosted, searchable history across every session with no laptop babysitting
Nothing worth pointing at. That's why the prompt exists.
share on X ↗"I just replaced Hedy ($12.99/mo) with one prompt"
Vibecode Hedy
Kinda. The core of Hedy is buildable in a weekend with the prompt on this page, but there are real gaps: Mobile apps that keep recording reliably when the screen is off, which is where most of these sessions actually happen, Tuned latency: your DIY suggestions arrive a beat late, and a beat late is useless in conversation. Read the honest list above before committing.
How much does Hedy cost?
Hedy costs about $12.99/month (Pro, checked 2026-08-18), which is $155.88 per year.
What do I lose by replacing Hedy?
Honestly: Mobile apps that keep recording reliably when the screen is off, which is where most of these sessions actually happen; Tuned latency: your DIY suggestions arrive a beat late, and a beat late is useless in conversation; Speaker diarization and meeting-type presets that shape the assistant for a sales call vs a lecture vs a doctor visit; Calendar and conferencing integrations that join and label sessions for you; Hosted, searchable history across every session with no laptop babysitting. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Hedy?
No mature open-source alternative worth pointing at, which is exactly why the one-shot prompt on this page exists.