Vibecode Sembly AI
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Sembly AI, transcribe a call and produce structured minutes, tasks, and searchable topics. The hard boundary is meeting bots, team collaboration, automations, and broad integrations, plus capture reliability, integrations, and collaboration.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Sembly AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Sembly AI · indie build A local meeting notebook whose hard problems are capturing the far side of a call at all and proving you did, keeping two independently clocked audio tracks aligned over an hour, and producing tasks that never invent an owner or a deadline nobody said. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records | audio capture is an operating system conversation and a hosted app cannot have it | | Capture | Two independent tracks written to separate files as they arrive, never premixed | a mix throws away the only free speaker label you will ever get | | Transcription | A local speech model per track, segments stored with track, start and end | the recordings are of other people and should not need to leave the machine | | Alignment | A shared wall-clock timeline with per-track drift correction | two capture devices have two clocks and an hour is long enough for that to matter | | Storage | SQLite beside the audio files, with search over segments | the notebook is worth more than any single meeting and has to survive a reinstall | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the capture service and the interface. - Get it: python.org or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A working system audio capture path on your platform** · free, or the price of a virtual audio device - Why: This is the whole product and it differs by operating system and version, so confirm it with a ten second test recording before writing anything else. - Get it: Read your platform's current documentation for capturing what the speakers are playing, grant the permission, and record ten seconds of music. - Verify: A test recording of the remote track contains audible sound, not silence - [ ] **A local speech model and its weights** · free, plus disk for the weights - Why: These are recordings of other people talking, and sending them to a service is a decision that should be deliberate rather than a default. - Get it: Download a local speech recognition model suitable for your machine. - Verify: A one minute recording transcribes locally in an acceptable time - [ ] **ffmpeg** · free - Why: Imports the recordings other tools produce and converts anything you were sent. - Get it: Your package manager, or ffmpeg.org. - Verify: ffmpeg -version prints a version - [ ] **Disk you have measured, with a retention figure chosen** · free - Why: Hours of two-track audio adds up faster than people expect and the notebook becomes a liability before it becomes a space problem. - Get it: Decide the retention period in days and check you have room for that many hours. - Verify: The configured retention and the available disk are both written in the README ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A meeting bot that joins calls for you. - Calendar auto-join. - Team workspaces and shared retention. - Mobile capture. - meeting bots, team collaboration, automations, and broad integrations - calendar auto-join - reliable speaker diarization - mobile capture - team search and sharing If one of those is essential to you, that is the reason to keep paying for Sembly AI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Sembly AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal meeting notebook that records your microphone and the call's audio as two separate tracks and turns them into minutes and tasks to replace Sembly AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The hard part of this product is not the notes. It is getting the other person's voice into the file at all, and the web platform will let you believe you have it when you have not. A browser can ask for display capture with audio, but the specification and the documentation are explicit that support for audio tracks varies by browser and by the surface the user picks, and that the returned stream may contain no audio track even when audio was requested and system audio was asked for. On the desktop side the operating system gates it separately again: capturing what the speakers are playing is a different permission from the microphone, and a denial can present itself as a successful capture of silence rather than as an error. Check what your target platform and version require before you design the capture, because it decides whether this is a web app with a helper or a desktop app. So: record two tracks, never one mix. The microphone is one file, the call audio is another, and every segment knows which track it came from. Before a recording is accepted, measure both tracks and refuse a recording whose remote track is silent, with a message naming the permission to check. A notebook full of transcripts containing only your half of every conversation is the failure mode here, and it is silent, cumulative and discovered weeks later. One more rule: record only conversations you are in, and say out loud that you are recording before you start. There is no bot here that joins a call for you and no capture without a visible indicator. Store who was told with the recording. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records · audio capture is an operating system conversation and a hosted app cannot have it - Capture: Two independent tracks written to separate files as they arrive, never premixed · a mix throws away the only free speaker label you will ever get - Transcription: A local speech model per track, segments stored with track, start and end · the recordings are of other people and should not need to leave the machine - Alignment: A shared wall-clock timeline with per-track drift correction · two capture devices have two clocks and an hour is long enough for that to matter - Storage: SQLite beside the audio files, with search over segments · the notebook is worth more than any single meeting and has to survive a reinstall ### Data model (create this before Phase 1) - meetings(id, title, started_at, ended_at, consent_note, state) · the consent note is required before recording starts - tracks(id, meeting_id, kind, file_path, started_at_wall, sample_rate, frames_written, rms_summary) · kind is microphone or remote - segments(id, meeting_id, track_id, start_ms, end_ms, text, model_id, confidence) · times are on the shared timeline, not the track's own - speakers(id, meeting_id, label, display_name, source) · source is track for your own voice, cluster for a guess, manual for a correction - segment_speakers(segment_id, speaker_id, assigned_by) · assigned_by is diarisation or human - items(id, meeting_id, kind, text, owner_speaker_id, due_text, segment_ids_json, state) · kind is decision, action or question, and owner may be null forever - topics(id, meeting_id, label, segment_ids_json) · derived, rebuildable, never the only copy of anything - Every generated sentence in this schema carries the segment identifiers it came from. There is no summary row without them. ### Phase 1 · Two tracks, or an honest refusal Build: Enumerate capture devices and require the user to pick one for the microphone and one for the call audio, naming what each is. Write both tracks to disk as frames arrive, flushing often. Record each track's wall-clock start. Compute a rolling level for both tracks and, if the remote track stays below a floor for a stated interval, stop and say which permission to check. Trap: Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. Done when: A recording produces two files with independent frame counts and recorded wall-clock start times, checked on disk. A remote track that stays silent past the configured interval stops the recording with a message naming the permission, tested by muting the source. Killing the process mid-recording leaves both files playable up to the last flush, tested. Recording cannot start without a consent note stored on the meeting. Do not build yet: No transcription, speakers, notes or search yet. ### Phase 2 · One timeline, two clocks Build: Convert each track's frame index to a shared timeline using its recorded wall-clock start and its measured effective sample rate rather than its nominal one. Estimate drift by comparing elapsed wall time against frames written, and store the correction per track. Render the transcript as a single interleaved stream of segments ordered on the shared timeline. Trap: Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. Done when: A recording of an hour has both tracks aligned within a stated tolerance at the final minute, tested against a synchronised marker played into both. Segment times are stored on the shared timeline and the per-track drift correction is stored with the track. The interleaved transcript orders segments correctly across tracks, tested with overlapping speech. Playing a segment seeks to the right place in the right file. Do not build yet: No speaker names, notes, tasks or search yet. ### Phase 3 · Your voice is known, everyone else is a guess Build: Label every microphone-track segment with you, with certainty, because the track says so. Cluster only the remote track, present those labels as provisional, and let the user name them once per meeting with the name propagating to every segment in that cluster. Record whether a label came from clustering or from a human, and show the difference in the interface. Trap: Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. Done when: Every segment from the microphone track is attributed to the local speaker without clustering, verified in the database. Remote speaker labels are shown as provisional until named, and naming one propagates to that cluster's segments in one action. The source of every attribution is stored and shown. Re-running transcription does not overwrite a human attribution, tested. Do not build yet: No minutes, tasks or search yet. ### Phase 4 · Tasks that nobody invented Build: Generate items only from approved transcript text, requiring each item to cite the segment identifiers it came from. Drop any item whose citations do not resolve. Leave the owner null unless a speaker was named in the cited segments, and leave the due date as the words that were actually said rather than a parsed date. Every item plays its source audio in one click. Trap: Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. Done when: Every stored item has at least one segment identifier that resolves to a real segment in that meeting, enforced at write time and tested with a fabricated citation. An item whose cited segments name no speaker is stored with a null owner and displays as unassigned. Due dates are stored as the spoken words, and no item carries a calendar date that was not said. One click on any item plays the cited audio. Do not build yet: No cross-meeting search, retention or export yet. ### Phase 5 · A notebook rather than a pile Build: Index segments for full text search with the meeting, speaker and timestamp attached, so a result is a place in an audio file rather than a paragraph. Export a meeting as Markdown plus a transcript file plus its audio. Apply an audio retention period that deletes recordings while keeping transcripts and items, and show how much disk the notebook is using. Trap: Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread. Done when: A search result names the meeting, the speaker and the timestamp and plays from that point, tested. Audio older than the retention period is deleted and its transcript, items and citations survive with a marker that the audio is gone, tested with a clock override. Exporting a meeting produces Markdown, a timed transcript and the audio if present, and reimports into an empty instance. Total disk use by meeting is shown in the interface. Do not build yet: Nothing after this. ### Out of scope (and why) - A bot that joins meetings on your behalf, which records people who were told a person was attending and is exactly the capability this build refuses. - Calendar auto-join, which is the same refusal wearing a schedule. - Team workspaces and shared retention policies, which turn a personal notebook into a system holding other people's conversations under your name. - Mobile capture, which is a second set of audio permissions and a second alignment problem for very little gain. ### README must contain - That capturing call audio is an operating system permission separate from the microphone, and exactly which one to grant on your platform - That a refused system audio permission can look like a successful recording of silence, and how this build detects it - That two tracks are recorded and why a single mixed file would be worse - That speaker labels on the remote track are clustered guesses and your own track is not - Where recordings live, how long audio is kept, and what survives when it is deleted ===== AGENTS.md ===== # Agent instructions · Sembly AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records, Two independent tracks written to separate files as they arrive, never premixed, A local speech model per track, segments stored with track, start and end, A shared wall-clock timeline with per-track drift correction, SQLite beside the audio files, with search over segments. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. ===== BUILD_PLAN.md ===== # Build plan · Sembly AI A local meeting notebook whose hard problems are capturing the far side of a call at all and proving you did, keeping two independently clocked audio tracks aligned over an hour, and producing tasks that never invent an owner or a deadline nobody said. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Two tracks, or an honest refusal A recording captures microphone and call audio separately, and a silent far side stops the recording rather than being discovered later. ### Steps 1. Make the user choose two devices by name Guessing which device is the call is how you get two microphones. Files: `capture.py`, `devices.py` 2. Flush both tracks continuously and record wall-clock starts The start times are what lets you align the tracks later. Files: `capture.py` 3. Fail loudly on a silent remote track This is the failure that ruins a notebook quietly. Files: `capture.py`, `tests/test_silence.py` ### Done when - [ ] Two files with separate frame counts and start times - [ ] A silent remote track stops the recording with a named cause - [ ] A kill mid-recording leaves both files playable - [ ] No recording starts without a consent note ### Watch out - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. ## Phase 2 · One timeline, two clocks Segments from both tracks line up at the end of an hour, not just at the start. ### Steps 1. Measure the effective sample rate, do not trust the nominal one The difference is invisible for a minute and fatal for an hour. Files: `timeline.py`, `tests/test_drift.py` 2. Store times on the shared timeline only Two sets of times in one table is how alignment bugs hide. Files: `transcribe.py` 3. Interleave for display, keep tracks separate in storage The display is a view; the tracks are the record. Files: `views/transcript.html` ### Done when - [ ] Tracks align within tolerance at the end of an hour - [ ] Drift corrections are stored per track - [ ] Interleaving is correct across overlapping speech - [ ] Clicking a segment seeks to the right file and offset ### Watch out - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. ## Phase 3 · Your voice is known, everyone else is a guess Speaker labels distinguish what is certain from what was clustered, and corrections stick. ### Steps 1. Attribute the microphone track by construction It is the only speaker label in this build that is not a guess. Files: `speakers.py` 2. Cluster the remote track only, mark it provisional Show the user which labels are evidence and which are inference. Files: `diarise.py`, `views/transcript.html` 3. Never let a re-run overwrite a human correction The corrections are the most expensive data in the notebook. Files: `speakers.py`, `tests/test_corrections.py` ### Done when - [ ] Microphone segments are attributed without clustering - [ ] Remote labels are provisional until named - [ ] Naming a cluster propagates in one action - [ ] Re-running never overwrites a human attribution ### Watch out - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. ## Phase 4 · Tasks that nobody invented Decisions, actions and questions each point at the seconds of audio they came from, and an unassigned action stays unassigned. ### Steps 1. Resolve every citation before storing an item An unresolvable citation means the item was invented. Files: `items.py`, `tests/test_citations.py` 2. Make unassigned a state, not a blank Filling the blank is the moment this product starts lying. Files: `items.py`, `views/items.html` 3. Keep the due date as spoken words 'Early next week' is information; a parsed Tuesday is a fabrication. Files: `items.py` ### Done when - [ ] Every item's citations resolve or the item is dropped - [ ] Unowned actions display as unassigned - [ ] No item carries an unspoken calendar date - [ ] One click plays an item's source audio ### Watch out - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. ## Not in this build - A meeting bot that joins calls for you. - Calendar auto-join. - Team workspaces and shared retention. - Mobile capture. ## After v1, if you want it - Importing a recording the meeting platform made itself, which sidesteps the capture problem entirely when the platform offers one - A weekly review that lists every unassigned action older than a fortnight, which is the honest output of refusing to invent owners - Per-speaker voice profiles kept locally, so a regular participant is named across meetings rather than per meeting ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, transcripts and the database. Everything the notebook is. DATA_DIR=./data # Required. Chosen by name, because device indexes move between reboots. MIC_DEVICE=Built-in Microphone # Required. The call audio device. Getting this wrong records you twice. REMOTE_DEVICE=System Audio Capture # Required. How long a silent remote track is tolerated before the recording stops. SILENCE_ABORT_SECONDS=45 # Required. The drift the alignment test is allowed at the end of an hour. ALIGNMENT_TOLERANCE_MS=250 # Required. Recordings are deleted after this; transcripts and items are kept. AUDIO_RETENTION_DAYS=60 # Required. Stored with every segment, because two models give two transcripts. SPEECH_MODEL_PATH=./models/speech
You are building a lean indie version of Sembly AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Sembly AI · indie build A local meeting notebook whose hard problems are capturing the far side of a call at all and proving you did, keeping two independently clocked audio tracks aligned over an hour, and producing tasks that never invent an owner or a deadline nobody said. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records | audio capture is an operating system conversation and a hosted app cannot have it | | Capture | Two independent tracks written to separate files as they arrive, never premixed | a mix throws away the only free speaker label you will ever get | | Transcription | A local speech model per track, segments stored with track, start and end | the recordings are of other people and should not need to leave the machine | | Alignment | A shared wall-clock timeline with per-track drift correction | two capture devices have two clocks and an hour is long enough for that to matter | | Storage | SQLite beside the audio files, with search over segments | the notebook is worth more than any single meeting and has to survive a reinstall | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the capture service and the interface. - Get it: python.org or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A working system audio capture path on your platform** · free, or the price of a virtual audio device - Why: This is the whole product and it differs by operating system and version, so confirm it with a ten second test recording before writing anything else. - Get it: Read your platform's current documentation for capturing what the speakers are playing, grant the permission, and record ten seconds of music. - Verify: A test recording of the remote track contains audible sound, not silence - [ ] **A local speech model and its weights** · free, plus disk for the weights - Why: These are recordings of other people talking, and sending them to a service is a decision that should be deliberate rather than a default. - Get it: Download a local speech recognition model suitable for your machine. - Verify: A one minute recording transcribes locally in an acceptable time - [ ] **ffmpeg** · free - Why: Imports the recordings other tools produce and converts anything you were sent. - Get it: Your package manager, or ffmpeg.org. - Verify: ffmpeg -version prints a version - [ ] **Disk you have measured, with a retention figure chosen** · free - Why: Hours of two-track audio adds up faster than people expect and the notebook becomes a liability before it becomes a space problem. - Get it: Decide the retention period in days and check you have room for that many hours. - Verify: The configured retention and the available disk are both written in the README ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A meeting bot that joins calls for you. - Calendar auto-join. - Team workspaces and shared retention. - Mobile capture. - meeting bots, team collaboration, automations, and broad integrations - calendar auto-join - reliable speaker diarization - mobile capture - team search and sharing If one of those is essential to you, that is the reason to keep paying for Sembly AI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Sembly AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal meeting notebook that records your microphone and the call's audio as two separate tracks and turns them into minutes and tasks to replace Sembly AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The hard part of this product is not the notes. It is getting the other person's voice into the file at all, and the web platform will let you believe you have it when you have not. A browser can ask for display capture with audio, but the specification and the documentation are explicit that support for audio tracks varies by browser and by the surface the user picks, and that the returned stream may contain no audio track even when audio was requested and system audio was asked for. On the desktop side the operating system gates it separately again: capturing what the speakers are playing is a different permission from the microphone, and a denial can present itself as a successful capture of silence rather than as an error. Check what your target platform and version require before you design the capture, because it decides whether this is a web app with a helper or a desktop app. So: record two tracks, never one mix. The microphone is one file, the call audio is another, and every segment knows which track it came from. Before a recording is accepted, measure both tracks and refuse a recording whose remote track is silent, with a message naming the permission to check. A notebook full of transcripts containing only your half of every conversation is the failure mode here, and it is silent, cumulative and discovered weeks later. One more rule: record only conversations you are in, and say out loud that you are recording before you start. There is no bot here that joins a call for you and no capture without a visible indicator. Store who was told with the recording. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records · audio capture is an operating system conversation and a hosted app cannot have it - Capture: Two independent tracks written to separate files as they arrive, never premixed · a mix throws away the only free speaker label you will ever get - Transcription: A local speech model per track, segments stored with track, start and end · the recordings are of other people and should not need to leave the machine - Alignment: A shared wall-clock timeline with per-track drift correction · two capture devices have two clocks and an hour is long enough for that to matter - Storage: SQLite beside the audio files, with search over segments · the notebook is worth more than any single meeting and has to survive a reinstall ### Data model (create this before Phase 1) - meetings(id, title, started_at, ended_at, consent_note, state) · the consent note is required before recording starts - tracks(id, meeting_id, kind, file_path, started_at_wall, sample_rate, frames_written, rms_summary) · kind is microphone or remote - segments(id, meeting_id, track_id, start_ms, end_ms, text, model_id, confidence) · times are on the shared timeline, not the track's own - speakers(id, meeting_id, label, display_name, source) · source is track for your own voice, cluster for a guess, manual for a correction - segment_speakers(segment_id, speaker_id, assigned_by) · assigned_by is diarisation or human - items(id, meeting_id, kind, text, owner_speaker_id, due_text, segment_ids_json, state) · kind is decision, action or question, and owner may be null forever - topics(id, meeting_id, label, segment_ids_json) · derived, rebuildable, never the only copy of anything - Every generated sentence in this schema carries the segment identifiers it came from. There is no summary row without them. ### Phase 1 · Two tracks, or an honest refusal Build: Enumerate capture devices and require the user to pick one for the microphone and one for the call audio, naming what each is. Write both tracks to disk as frames arrive, flushing often. Record each track's wall-clock start. Compute a rolling level for both tracks and, if the remote track stays below a floor for a stated interval, stop and say which permission to check. Trap: Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. Done when: A recording produces two files with independent frame counts and recorded wall-clock start times, checked on disk. A remote track that stays silent past the configured interval stops the recording with a message naming the permission, tested by muting the source. Killing the process mid-recording leaves both files playable up to the last flush, tested. Recording cannot start without a consent note stored on the meeting. Do not build yet: No transcription, speakers, notes or search yet. ### Phase 2 · One timeline, two clocks Build: Convert each track's frame index to a shared timeline using its recorded wall-clock start and its measured effective sample rate rather than its nominal one. Estimate drift by comparing elapsed wall time against frames written, and store the correction per track. Render the transcript as a single interleaved stream of segments ordered on the shared timeline. Trap: Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. Done when: A recording of an hour has both tracks aligned within a stated tolerance at the final minute, tested against a synchronised marker played into both. Segment times are stored on the shared timeline and the per-track drift correction is stored with the track. The interleaved transcript orders segments correctly across tracks, tested with overlapping speech. Playing a segment seeks to the right place in the right file. Do not build yet: No speaker names, notes, tasks or search yet. ### Phase 3 · Your voice is known, everyone else is a guess Build: Label every microphone-track segment with you, with certainty, because the track says so. Cluster only the remote track, present those labels as provisional, and let the user name them once per meeting with the name propagating to every segment in that cluster. Record whether a label came from clustering or from a human, and show the difference in the interface. Trap: Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. Done when: Every segment from the microphone track is attributed to the local speaker without clustering, verified in the database. Remote speaker labels are shown as provisional until named, and naming one propagates to that cluster's segments in one action. The source of every attribution is stored and shown. Re-running transcription does not overwrite a human attribution, tested. Do not build yet: No minutes, tasks or search yet. ### Phase 4 · Tasks that nobody invented Build: Generate items only from approved transcript text, requiring each item to cite the segment identifiers it came from. Drop any item whose citations do not resolve. Leave the owner null unless a speaker was named in the cited segments, and leave the due date as the words that were actually said rather than a parsed date. Every item plays its source audio in one click. Trap: Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. Done when: Every stored item has at least one segment identifier that resolves to a real segment in that meeting, enforced at write time and tested with a fabricated citation. An item whose cited segments name no speaker is stored with a null owner and displays as unassigned. Due dates are stored as the spoken words, and no item carries a calendar date that was not said. One click on any item plays the cited audio. Do not build yet: No cross-meeting search, retention or export yet. ### Phase 5 · A notebook rather than a pile Build: Index segments for full text search with the meeting, speaker and timestamp attached, so a result is a place in an audio file rather than a paragraph. Export a meeting as Markdown plus a transcript file plus its audio. Apply an audio retention period that deletes recordings while keeping transcripts and items, and show how much disk the notebook is using. Trap: Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread. Done when: A search result names the meeting, the speaker and the timestamp and plays from that point, tested. Audio older than the retention period is deleted and its transcript, items and citations survive with a marker that the audio is gone, tested with a clock override. Exporting a meeting produces Markdown, a timed transcript and the audio if present, and reimports into an empty instance. Total disk use by meeting is shown in the interface. Do not build yet: Nothing after this. ### Out of scope (and why) - A bot that joins meetings on your behalf, which records people who were told a person was attending and is exactly the capability this build refuses. - Calendar auto-join, which is the same refusal wearing a schedule. - Team workspaces and shared retention policies, which turn a personal notebook into a system holding other people's conversations under your name. - Mobile capture, which is a second set of audio permissions and a second alignment problem for very little gain. ### README must contain - That capturing call audio is an operating system permission separate from the microphone, and exactly which one to grant on your platform - That a refused system audio permission can look like a successful recording of silence, and how this build detects it - That two tracks are recorded and why a single mixed file would be worse - That speaker labels on the remote track are clustered guesses and your own track is not - Where recordings live, how long audio is kept, and what survives when it is deleted ===== AGENTS.md ===== # Agent instructions · Sembly AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records, Two independent tracks written to separate files as they arrive, never premixed, A local speech model per track, segments stored with track, start and end, A shared wall-clock timeline with per-track drift correction, SQLite beside the audio files, with search over segments. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. ===== BUILD_PLAN.md ===== # Build plan · Sembly AI A local meeting notebook whose hard problems are capturing the far side of a call at all and proving you did, keeping two independently clocked audio tracks aligned over an hour, and producing tasks that never invent an owner or a deadline nobody said. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Two tracks, or an honest refusal A recording captures microphone and call audio separately, and a silent far side stops the recording rather than being discovered later. ### Steps 1. Make the user choose two devices by name Guessing which device is the call is how you get two microphones. Files: `capture.py`, `devices.py` 2. Flush both tracks continuously and record wall-clock starts The start times are what lets you align the tracks later. Files: `capture.py` 3. Fail loudly on a silent remote track This is the failure that ruins a notebook quietly. Files: `capture.py`, `tests/test_silence.py` ### Done when - [ ] Two files with separate frame counts and start times - [ ] A silent remote track stops the recording with a named cause - [ ] A kill mid-recording leaves both files playable - [ ] No recording starts without a consent note ### Watch out - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. ## Phase 2 · One timeline, two clocks Segments from both tracks line up at the end of an hour, not just at the start. ### Steps 1. Measure the effective sample rate, do not trust the nominal one The difference is invisible for a minute and fatal for an hour. Files: `timeline.py`, `tests/test_drift.py` 2. Store times on the shared timeline only Two sets of times in one table is how alignment bugs hide. Files: `transcribe.py` 3. Interleave for display, keep tracks separate in storage The display is a view; the tracks are the record. Files: `views/transcript.html` ### Done when - [ ] Tracks align within tolerance at the end of an hour - [ ] Drift corrections are stored per track - [ ] Interleaving is correct across overlapping speech - [ ] Clicking a segment seeks to the right file and offset ### Watch out - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. ## Phase 3 · Your voice is known, everyone else is a guess Speaker labels distinguish what is certain from what was clustered, and corrections stick. ### Steps 1. Attribute the microphone track by construction It is the only speaker label in this build that is not a guess. Files: `speakers.py` 2. Cluster the remote track only, mark it provisional Show the user which labels are evidence and which are inference. Files: `diarise.py`, `views/transcript.html` 3. Never let a re-run overwrite a human correction The corrections are the most expensive data in the notebook. Files: `speakers.py`, `tests/test_corrections.py` ### Done when - [ ] Microphone segments are attributed without clustering - [ ] Remote labels are provisional until named - [ ] Naming a cluster propagates in one action - [ ] Re-running never overwrites a human attribution ### Watch out - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. ## Phase 4 · Tasks that nobody invented Decisions, actions and questions each point at the seconds of audio they came from, and an unassigned action stays unassigned. ### Steps 1. Resolve every citation before storing an item An unresolvable citation means the item was invented. Files: `items.py`, `tests/test_citations.py` 2. Make unassigned a state, not a blank Filling the blank is the moment this product starts lying. Files: `items.py`, `views/items.html` 3. Keep the due date as spoken words 'Early next week' is information; a parsed Tuesday is a fabrication. Files: `items.py` ### Done when - [ ] Every item's citations resolve or the item is dropped - [ ] Unowned actions display as unassigned - [ ] No item carries an unspoken calendar date - [ ] One click plays an item's source audio ### Watch out - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. ## Not in this build - A meeting bot that joins calls for you. - Calendar auto-join. - Team workspaces and shared retention. - Mobile capture. ## After v1, if you want it - Importing a recording the meeting platform made itself, which sidesteps the capture problem entirely when the platform offers one - A weekly review that lists every unassigned action older than a fortnight, which is the honest output of refusing to invent owners - Per-speaker voice profiles kept locally, so a regular participant is named across meetings rather than per meeting ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, transcripts and the database. Everything the notebook is. DATA_DIR=./data # Required. Chosen by name, because device indexes move between reboots. MIC_DEVICE=Built-in Microphone # Required. The call audio device. Getting this wrong records you twice. REMOTE_DEVICE=System Audio Capture # Required. How long a silent remote track is tolerated before the recording stops. SILENCE_ABORT_SECONDS=45 # Required. The drift the alignment test is allowed at the end of an hour. ALIGNMENT_TOLERANCE_MS=250 # Required. Recordings are deleted after this; transcripts and items are kept. AUDIO_RETENTION_DAYS=60 # Required. Stored with every segment, because two models give two transcripts. SPEECH_MODEL_PATH=./models/speech
You are building a production product version of Sembly AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Sembly AI · product brief ## Problem The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Sembly AI, transcribe a call and produce structured minutes, tasks, and searchable topics. The hard boundary is meeting bots, team collaboration, automations, and broad integrations, plus capture reliability, integrations, and collaboration. ## Product outcome A notebook that either captures both sides of a call or tells you within a minute that it did not, keeps an hour of two-track audio in alignment, attributes your own voice by construction rather than by guess, and produces tasks that point back at the seconds they came from. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - desktop microphone access - whisper.cpp model files - optional OpenAI or Anthropic API key - local storage ## Explicit non-goals for v1 - A meeting bot that joins calls for you. - Calendar auto-join. - Team workspaces and shared retention. - Mobile capture. - meeting bots, team collaboration, automations, and broad integrations - calendar auto-join - reliable speaker diarization - mobile capture - team search and sharing ## Success criteria - A muted remote source is confirmed to abort the recording with a named cause - Alignment is confirmed within tolerance at the end of a long recording - Microphone segments are confirmed attributed without clustering - A fabricated citation is confirmed to drop its item - Expired audio is confirmed to leave transcripts and items intact ===== BRIEF.md ===== # Build brief · Sembly AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal meeting notebook that records your microphone and the call's audio as two separate tracks and turns them into minutes and tasks to replace Sembly AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The hard part of this product is not the notes. It is getting the other person's voice into the file at all, and the web platform will let you believe you have it when you have not. A browser can ask for display capture with audio, but the specification and the documentation are explicit that support for audio tracks varies by browser and by the surface the user picks, and that the returned stream may contain no audio track even when audio was requested and system audio was asked for. On the desktop side the operating system gates it separately again: capturing what the speakers are playing is a different permission from the microphone, and a denial can present itself as a successful capture of silence rather than as an error. Check what your target platform and version require before you design the capture, because it decides whether this is a web app with a helper or a desktop app. So: record two tracks, never one mix. The microphone is one file, the call audio is another, and every segment knows which track it came from. Before a recording is accepted, measure both tracks and refuse a recording whose remote track is silent, with a message naming the permission to check. A notebook full of transcripts containing only your half of every conversation is the failure mode here, and it is silent, cumulative and discovered weeks later. One more rule: record only conversations you are in, and say out loud that you are recording before you start. There is no bot here that joins a call for you and no capture without a visible indicator. Store who was told with the recording. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records · audio capture is an operating system conversation and a hosted app cannot have it - Capture: Two independent tracks written to separate files as they arrive, never premixed · a mix throws away the only free speaker label you will ever get - Transcription: A local speech model per track, segments stored with track, start and end · the recordings are of other people and should not need to leave the machine - Alignment: A shared wall-clock timeline with per-track drift correction · two capture devices have two clocks and an hour is long enough for that to matter - Storage: SQLite beside the audio files, with search over segments · the notebook is worth more than any single meeting and has to survive a reinstall ### Data model (create this before Phase 1) - meetings(id, title, started_at, ended_at, consent_note, state) · the consent note is required before recording starts - tracks(id, meeting_id, kind, file_path, started_at_wall, sample_rate, frames_written, rms_summary) · kind is microphone or remote - segments(id, meeting_id, track_id, start_ms, end_ms, text, model_id, confidence) · times are on the shared timeline, not the track's own - speakers(id, meeting_id, label, display_name, source) · source is track for your own voice, cluster for a guess, manual for a correction - segment_speakers(segment_id, speaker_id, assigned_by) · assigned_by is diarisation or human - items(id, meeting_id, kind, text, owner_speaker_id, due_text, segment_ids_json, state) · kind is decision, action or question, and owner may be null forever - topics(id, meeting_id, label, segment_ids_json) · derived, rebuildable, never the only copy of anything - Every generated sentence in this schema carries the segment identifiers it came from. There is no summary row without them. ### Phase 1 · Two tracks, or an honest refusal Build: Enumerate capture devices and require the user to pick one for the microphone and one for the call audio, naming what each is. Write both tracks to disk as frames arrive, flushing often. Record each track's wall-clock start. Compute a rolling level for both tracks and, if the remote track stays below a floor for a stated interval, stop and say which permission to check. Trap: Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. Done when: A recording produces two files with independent frame counts and recorded wall-clock start times, checked on disk. A remote track that stays silent past the configured interval stops the recording with a message naming the permission, tested by muting the source. Killing the process mid-recording leaves both files playable up to the last flush, tested. Recording cannot start without a consent note stored on the meeting. Do not build yet: No transcription, speakers, notes or search yet. ### Phase 2 · One timeline, two clocks Build: Convert each track's frame index to a shared timeline using its recorded wall-clock start and its measured effective sample rate rather than its nominal one. Estimate drift by comparing elapsed wall time against frames written, and store the correction per track. Render the transcript as a single interleaved stream of segments ordered on the shared timeline. Trap: Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. Done when: A recording of an hour has both tracks aligned within a stated tolerance at the final minute, tested against a synchronised marker played into both. Segment times are stored on the shared timeline and the per-track drift correction is stored with the track. The interleaved transcript orders segments correctly across tracks, tested with overlapping speech. Playing a segment seeks to the right place in the right file. Do not build yet: No speaker names, notes, tasks or search yet. ### Phase 3 · Your voice is known, everyone else is a guess Build: Label every microphone-track segment with you, with certainty, because the track says so. Cluster only the remote track, present those labels as provisional, and let the user name them once per meeting with the name propagating to every segment in that cluster. Record whether a label came from clustering or from a human, and show the difference in the interface. Trap: Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. Done when: Every segment from the microphone track is attributed to the local speaker without clustering, verified in the database. Remote speaker labels are shown as provisional until named, and naming one propagates to that cluster's segments in one action. The source of every attribution is stored and shown. Re-running transcription does not overwrite a human attribution, tested. Do not build yet: No minutes, tasks or search yet. ### Phase 4 · Tasks that nobody invented Build: Generate items only from approved transcript text, requiring each item to cite the segment identifiers it came from. Drop any item whose citations do not resolve. Leave the owner null unless a speaker was named in the cited segments, and leave the due date as the words that were actually said rather than a parsed date. Every item plays its source audio in one click. Trap: Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. Done when: Every stored item has at least one segment identifier that resolves to a real segment in that meeting, enforced at write time and tested with a fabricated citation. An item whose cited segments name no speaker is stored with a null owner and displays as unassigned. Due dates are stored as the spoken words, and no item carries a calendar date that was not said. One click on any item plays the cited audio. Do not build yet: No cross-meeting search, retention or export yet. ### Phase 5 · A notebook rather than a pile Build: Index segments for full text search with the meeting, speaker and timestamp attached, so a result is a place in an audio file rather than a paragraph. Export a meeting as Markdown plus a transcript file plus its audio. Apply an audio retention period that deletes recordings while keeping transcripts and items, and show how much disk the notebook is using. Trap: Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread. Done when: A search result names the meeting, the speaker and the timestamp and plays from that point, tested. Audio older than the retention period is deleted and its transcript, items and citations survive with a marker that the audio is gone, tested with a clock override. Exporting a meeting produces Markdown, a timed transcript and the audio if present, and reimports into an empty instance. Total disk use by meeting is shown in the interface. Do not build yet: Nothing after this. ### Out of scope (and why) - A bot that joins meetings on your behalf, which records people who were told a person was attending and is exactly the capability this build refuses. - Calendar auto-join, which is the same refusal wearing a schedule. - Team workspaces and shared retention policies, which turn a personal notebook into a system holding other people's conversations under your name. - Mobile capture, which is a second set of audio permissions and a second alignment problem for very little gain. ### README must contain - That capturing call audio is an operating system permission separate from the microphone, and exactly which one to grant on your platform - That a refused system audio permission can look like a successful recording of silence, and how this build detects it - That two tracks are recorded and why a single mixed file would be worse - That speaker labels on the remote track are clustered guesses and your own track is not - Where recordings live, how long audio is kept, and what survives when it is deleted ===== ARCHITECTURE.md ===== # Architecture · Sembly AI ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records | audio capture is an operating system conversation and a hosted app cannot have it | | Capture | Two independent tracks written to separate files as they arrive, never premixed | a mix throws away the only free speaker label you will ever get | | Transcription | A local speech model per track, segments stored with track, start and end | the recordings are of other people and should not need to leave the machine | | Alignment | A shared wall-clock timeline with per-track drift correction | two capture devices have two clocks and an hour is long enough for that to matter | | Storage | SQLite beside the audio files, with search over segments | the notebook is worth more than any single meeting and has to survive a reinstall | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Capture | Device selection, two-track writing, silence detection. | Any capture backend, provided it stays two tracks and detects silence live. | | Timeline | Wall-clock starts, drift measurement, shared times. | Any method, provided alignment is tested at the end of a long recording. | | Transcription | Per-track segments and model identity. | Any speech model, provided segments keep their track. | | Speakers | Local certainty, remote clustering, human corrections. | Any clustering, provided corrections outrank it permanently. | | Items | Decisions, actions, questions and their citations. | Any model, provided unresolvable citations drop the item. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATA_DIR` · required · Recordings, transcripts and the database. Everything the notebook is. - `MIC_DEVICE` · required · Chosen by name, because device indexes move between reboots. - `REMOTE_DEVICE` · required · The call audio device. Getting this wrong records you twice. - `SILENCE_ABORT_SECONDS` · required · How long a silent remote track is tolerated before the recording stops. - `ALIGNMENT_TOLERANCE_MS` · required · The drift the alignment test is allowed at the end of an hour. - `AUDIO_RETENTION_DAYS` · required · Recordings are deleted after this; transcripts and items are kept. - `SPEECH_MODEL_PATH` · required · Stored with every segment, because two models give two transcripts. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Sembly AI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records, Two independent tracks written to separate files as they arrive, never premixed, A local speech model per track, segments stored with track, start and end, A shared wall-clock timeline with per-track drift correction, SQLite beside the audio files, with search over segments. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. - Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread. ===== MILESTONES.md ===== # Delivery milestones · Sembly AI Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Two tracks, or an honest refusal A recording captures microphone and call audio separately, and a silent far side stops the recording rather than being discovered later. ### Steps 1. Make the user choose two devices by name Guessing which device is the call is how you get two microphones. Files: `capture.py`, `devices.py` 2. Flush both tracks continuously and record wall-clock starts The start times are what lets you align the tracks later. Files: `capture.py` 3. Fail loudly on a silent remote track This is the failure that ruins a notebook quietly. Files: `capture.py`, `tests/test_silence.py` ### Done when - [ ] Two files with separate frame counts and start times - [ ] A silent remote track stops the recording with a named cause - [ ] A kill mid-recording leaves both files playable - [ ] No recording starts without a consent note ### Watch out - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. ## M2 · One timeline, two clocks Segments from both tracks line up at the end of an hour, not just at the start. ### Steps 1. Measure the effective sample rate, do not trust the nominal one The difference is invisible for a minute and fatal for an hour. Files: `timeline.py`, `tests/test_drift.py` 2. Store times on the shared timeline only Two sets of times in one table is how alignment bugs hide. Files: `transcribe.py` 3. Interleave for display, keep tracks separate in storage The display is a view; the tracks are the record. Files: `views/transcript.html` ### Done when - [ ] Tracks align within tolerance at the end of an hour - [ ] Drift corrections are stored per track - [ ] Interleaving is correct across overlapping speech - [ ] Clicking a segment seeks to the right file and offset ### Watch out - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. ## M3 · Your voice is known, everyone else is a guess Speaker labels distinguish what is certain from what was clustered, and corrections stick. ### Steps 1. Attribute the microphone track by construction It is the only speaker label in this build that is not a guess. Files: `speakers.py` 2. Cluster the remote track only, mark it provisional Show the user which labels are evidence and which are inference. Files: `diarise.py`, `views/transcript.html` 3. Never let a re-run overwrite a human correction The corrections are the most expensive data in the notebook. Files: `speakers.py`, `tests/test_corrections.py` ### Done when - [ ] Microphone segments are attributed without clustering - [ ] Remote labels are provisional until named - [ ] Naming a cluster propagates in one action - [ ] Re-running never overwrites a human attribution ### Watch out - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. ## M4 · Tasks that nobody invented Decisions, actions and questions each point at the seconds of audio they came from, and an unassigned action stays unassigned. ### Steps 1. Resolve every citation before storing an item An unresolvable citation means the item was invented. Files: `items.py`, `tests/test_citations.py` 2. Make unassigned a state, not a blank Filling the blank is the moment this product starts lying. Files: `items.py`, `views/items.html` 3. Keep the due date as spoken words 'Early next week' is information; a parsed Tuesday is a fabrication. Files: `items.py` ### Done when - [ ] Every item's citations resolve or the item is dropped - [ ] Unowned actions display as unassigned - [ ] No item carries an unspoken calendar date - [ ] One click plays an item's source audio ### Watch out - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. ## M5 · A notebook rather than a pile (production only) Search works across meetings, storage stays bounded, and everything can be taken elsewhere. ### Steps 1. Make a search result a position in a recording A paragraph with no timestamp is not checkable. Files: `search.py`, `views/search.html` 2. Expire audio, keep transcripts and items The recording is the liability; the notes are the asset. Files: `retention.py`, `tests/test_retention.py` 3. Export a meeting as files anyone can read Markdown and a timed transcript outlive this application. Files: `export.py` ### Done when - [ ] Search results play from the matched moment - [ ] Expired audio leaves transcripts and citations intact - [ ] Export round-trips into an empty instance - [ ] Disk use per meeting is visible ### Watch out - Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread. ===== OPERATIONS.md ===== # Operations · Sembly AI ## Backup The database and the transcripts first, the audio second. Corrections and items cannot be regenerated; the audio can at least be re-transcribed while it exists, and after retention it is gone by design anyway. ## Restore Restore the database and transcripts, then confirm that item citations still resolve and that search returns results. If audio was restored too, check a segment plays at the right offset, because a partial audio restore breaks alignment silently. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Recordings aborted for a silent remote track, measured drift per recording against the tolerance, transcription failures, and disk used against the retention period. A drift figure creeping up usually means a device changed. ## Incident checklist The incident here is a fortnight of meetings where only your voice was recorded, or a set of minutes assigning work to someone who never agreed. Both are caught by gates rather than by noticing: the silence abort and the citation resolution. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A muted remote source is confirmed to abort the recording with a named cause - [ ] Alignment is confirmed within tolerance at the end of a long recording - [ ] Microphone segments are confirmed attributed without clustering - [ ] A fabricated citation is confirmed to drop its item - [ ] Expired audio is confirmed to leave transcripts and items intact ## Launch constraint Do not market omitted Sembly AI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, transcripts and the database. Everything the notebook is. DATA_DIR=./data # Required. Chosen by name, because device indexes move between reboots. MIC_DEVICE=Built-in Microphone # Required. The call audio device. Getting this wrong records you twice. REMOTE_DEVICE=System Audio Capture # Required. How long a silent remote track is tolerated before the recording stops. SILENCE_ABORT_SECONDS=45 # Required. The drift the alignment test is allowed at the end of an hour. ALIGNMENT_TOLERANCE_MS=250 # Required. Recordings are deleted after this; transcripts and items are kept. AUDIO_RETENTION_DAYS=60 # Required. Stored with every segment, because two models give two transcripts. SPEECH_MODEL_PATH=./models/speech
README.md
# Sembly AI · indie build A local meeting notebook whose hard problems are capturing the far side of a call at all and proving you did, keeping two independently clocked audio tracks aligned over an hour, and producing tasks that never invent an owner or a deadline nobody said. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records | audio capture is an operating system conversation and a hosted app cannot have it | | Capture | Two independent tracks written to separate files as they arrive, never premixed | a mix throws away the only free speaker label you will ever get | | Transcription | A local speech model per track, segments stored with track, start and end | the recordings are of other people and should not need to leave the machine | | Alignment | A shared wall-clock timeline with per-track drift correction | two capture devices have two clocks and an hour is long enough for that to matter | | Storage | SQLite beside the audio files, with search over segments | the notebook is worth more than any single meeting and has to survive a reinstall | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the capture service and the interface. - Get it: python.org or your version manager. - Verify: python3 --version prints 3.12 or higher - [ ] **A working system audio capture path on your platform** · free, or the price of a virtual audio device - Why: This is the whole product and it differs by operating system and version, so confirm it with a ten second test recording before writing anything else. - Get it: Read your platform's current documentation for capturing what the speakers are playing, grant the permission, and record ten seconds of music. - Verify: A test recording of the remote track contains audible sound, not silence - [ ] **A local speech model and its weights** · free, plus disk for the weights - Why: These are recordings of other people talking, and sending them to a service is a decision that should be deliberate rather than a default. - Get it: Download a local speech recognition model suitable for your machine. - Verify: A one minute recording transcribes locally in an acceptable time - [ ] **ffmpeg** · free - Why: Imports the recordings other tools produce and converts anything you were sent. - Get it: Your package manager, or ffmpeg.org. - Verify: ffmpeg -version prints a version - [ ] **Disk you have measured, with a retention figure chosen** · free - Why: Hours of two-track audio adds up faster than people expect and the notebook becomes a liability before it becomes a space problem. - Get it: Decide the retention period in days and check you have room for that many hours. - Verify: The configured retention and the available disk are both written in the README ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A meeting bot that joins calls for you. - Calendar auto-join. - Team workspaces and shared retention. - Mobile capture. - meeting bots, team collaboration, automations, and broad integrations - calendar auto-join - reliable speaker diarization - mobile capture - team search and sharing If one of those is essential to you, that is the reason to keep paying for Sembly AI, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Sembly AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal meeting notebook that records your microphone and the call's audio as two separate tracks and turns them into minutes and tasks to replace Sembly AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The hard part of this product is not the notes. It is getting the other person's voice into the file at all, and the web platform will let you believe you have it when you have not. A browser can ask for display capture with audio, but the specification and the documentation are explicit that support for audio tracks varies by browser and by the surface the user picks, and that the returned stream may contain no audio track even when audio was requested and system audio was asked for. On the desktop side the operating system gates it separately again: capturing what the speakers are playing is a different permission from the microphone, and a denial can present itself as a successful capture of silence rather than as an error. Check what your target platform and version require before you design the capture, because it decides whether this is a web app with a helper or a desktop app. So: record two tracks, never one mix. The microphone is one file, the call audio is another, and every segment knows which track it came from. Before a recording is accepted, measure both tracks and refuse a recording whose remote track is silent, with a message naming the permission to check. A notebook full of transcripts containing only your half of every conversation is the failure mode here, and it is silent, cumulative and discovered weeks later. One more rule: record only conversations you are in, and say out loud that you are recording before you start. There is no bot here that joins a call for you and no capture without a visible indicator. Store who was told with the recording. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records · audio capture is an operating system conversation and a hosted app cannot have it - Capture: Two independent tracks written to separate files as they arrive, never premixed · a mix throws away the only free speaker label you will ever get - Transcription: A local speech model per track, segments stored with track, start and end · the recordings are of other people and should not need to leave the machine - Alignment: A shared wall-clock timeline with per-track drift correction · two capture devices have two clocks and an hour is long enough for that to matter - Storage: SQLite beside the audio files, with search over segments · the notebook is worth more than any single meeting and has to survive a reinstall ### Data model (create this before Phase 1) - meetings(id, title, started_at, ended_at, consent_note, state) · the consent note is required before recording starts - tracks(id, meeting_id, kind, file_path, started_at_wall, sample_rate, frames_written, rms_summary) · kind is microphone or remote - segments(id, meeting_id, track_id, start_ms, end_ms, text, model_id, confidence) · times are on the shared timeline, not the track's own - speakers(id, meeting_id, label, display_name, source) · source is track for your own voice, cluster for a guess, manual for a correction - segment_speakers(segment_id, speaker_id, assigned_by) · assigned_by is diarisation or human - items(id, meeting_id, kind, text, owner_speaker_id, due_text, segment_ids_json, state) · kind is decision, action or question, and owner may be null forever - topics(id, meeting_id, label, segment_ids_json) · derived, rebuildable, never the only copy of anything - Every generated sentence in this schema carries the segment identifiers it came from. There is no summary row without them. ### Phase 1 · Two tracks, or an honest refusal Build: Enumerate capture devices and require the user to pick one for the microphone and one for the call audio, naming what each is. Write both tracks to disk as frames arrive, flushing often. Record each track's wall-clock start. Compute a rolling level for both tracks and, if the remote track stays below a floor for a stated interval, stop and say which permission to check. Trap: Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. Done when: A recording produces two files with independent frame counts and recorded wall-clock start times, checked on disk. A remote track that stays silent past the configured interval stops the recording with a message naming the permission, tested by muting the source. Killing the process mid-recording leaves both files playable up to the last flush, tested. Recording cannot start without a consent note stored on the meeting. Do not build yet: No transcription, speakers, notes or search yet. ### Phase 2 · One timeline, two clocks Build: Convert each track's frame index to a shared timeline using its recorded wall-clock start and its measured effective sample rate rather than its nominal one. Estimate drift by comparing elapsed wall time against frames written, and store the correction per track. Render the transcript as a single interleaved stream of segments ordered on the shared timeline. Trap: Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. Done when: A recording of an hour has both tracks aligned within a stated tolerance at the final minute, tested against a synchronised marker played into both. Segment times are stored on the shared timeline and the per-track drift correction is stored with the track. The interleaved transcript orders segments correctly across tracks, tested with overlapping speech. Playing a segment seeks to the right place in the right file. Do not build yet: No speaker names, notes, tasks or search yet. ### Phase 3 · Your voice is known, everyone else is a guess Build: Label every microphone-track segment with you, with certainty, because the track says so. Cluster only the remote track, present those labels as provisional, and let the user name them once per meeting with the name propagating to every segment in that cluster. Record whether a label came from clustering or from a human, and show the difference in the interface. Trap: Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. Done when: Every segment from the microphone track is attributed to the local speaker without clustering, verified in the database. Remote speaker labels are shown as provisional until named, and naming one propagates to that cluster's segments in one action. The source of every attribution is stored and shown. Re-running transcription does not overwrite a human attribution, tested. Do not build yet: No minutes, tasks or search yet. ### Phase 4 · Tasks that nobody invented Build: Generate items only from approved transcript text, requiring each item to cite the segment identifiers it came from. Drop any item whose citations do not resolve. Leave the owner null unless a speaker was named in the cited segments, and leave the due date as the words that were actually said rather than a parsed date. Every item plays its source audio in one click. Trap: Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. Done when: Every stored item has at least one segment identifier that resolves to a real segment in that meeting, enforced at write time and tested with a fabricated citation. An item whose cited segments name no speaker is stored with a null owner and displays as unassigned. Due dates are stored as the spoken words, and no item carries a calendar date that was not said. One click on any item plays the cited audio. Do not build yet: No cross-meeting search, retention or export yet. ### Phase 5 · A notebook rather than a pile Build: Index segments for full text search with the meeting, speaker and timestamp attached, so a result is a place in an audio file rather than a paragraph. Export a meeting as Markdown plus a transcript file plus its audio. Apply an audio retention period that deletes recordings while keeping transcripts and items, and show how much disk the notebook is using. Trap: Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread. Done when: A search result names the meeting, the speaker and the timestamp and plays from that point, tested. Audio older than the retention period is deleted and its transcript, items and citations survive with a marker that the audio is gone, tested with a clock override. Exporting a meeting produces Markdown, a timed transcript and the audio if present, and reimports into an empty instance. Total disk use by meeting is shown in the interface. Do not build yet: Nothing after this. ### Out of scope (and why) - A bot that joins meetings on your behalf, which records people who were told a person was attending and is exactly the capability this build refuses. - Calendar auto-join, which is the same refusal wearing a schedule. - Team workspaces and shared retention policies, which turn a personal notebook into a system holding other people's conversations under your name. - Mobile capture, which is a second set of audio permissions and a second alignment problem for very little gain. ### README must contain - That capturing call audio is an operating system permission separate from the microphone, and exactly which one to grant on your platform - That a refused system audio permission can look like a successful recording of silence, and how this build detects it - That two tracks are recorded and why a single mixed file would be worse - That speaker labels on the remote track are clustered guesses and your own track is not - Where recordings live, how long audio is kept, and what survives when it is deleted
AGENTS.md
# Agent instructions · Sembly AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records, Two independent tracks written to separate files as they arrive, never premixed, A local speech model per track, segments stored with track, start and end, A shared wall-clock timeline with per-track drift correction, SQLite beside the audio files, with search over segments. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled.
BUILD_PLAN.md
# Build plan · Sembly AI A local meeting notebook whose hard problems are capturing the far side of a call at all and proving you did, keeping two independently clocked audio tracks aligned over an hour, and producing tasks that never invent an owner or a deadline nobody said. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Two tracks, or an honest refusal A recording captures microphone and call audio separately, and a silent far side stops the recording rather than being discovered later. ### Steps 1. Make the user choose two devices by name Guessing which device is the call is how you get two microphones. Files: `capture.py`, `devices.py` 2. Flush both tracks continuously and record wall-clock starts The start times are what lets you align the tracks later. Files: `capture.py` 3. Fail loudly on a silent remote track This is the failure that ruins a notebook quietly. Files: `capture.py`, `tests/test_silence.py` ### Done when - [ ] Two files with separate frame counts and start times - [ ] A silent remote track stops the recording with a named cause - [ ] A kill mid-recording leaves both files playable - [ ] No recording starts without a consent note ### Watch out - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. ## Phase 2 · One timeline, two clocks Segments from both tracks line up at the end of an hour, not just at the start. ### Steps 1. Measure the effective sample rate, do not trust the nominal one The difference is invisible for a minute and fatal for an hour. Files: `timeline.py`, `tests/test_drift.py` 2. Store times on the shared timeline only Two sets of times in one table is how alignment bugs hide. Files: `transcribe.py` 3. Interleave for display, keep tracks separate in storage The display is a view; the tracks are the record. Files: `views/transcript.html` ### Done when - [ ] Tracks align within tolerance at the end of an hour - [ ] Drift corrections are stored per track - [ ] Interleaving is correct across overlapping speech - [ ] Clicking a segment seeks to the right file and offset ### Watch out - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. ## Phase 3 · Your voice is known, everyone else is a guess Speaker labels distinguish what is certain from what was clustered, and corrections stick. ### Steps 1. Attribute the microphone track by construction It is the only speaker label in this build that is not a guess. Files: `speakers.py` 2. Cluster the remote track only, mark it provisional Show the user which labels are evidence and which are inference. Files: `diarise.py`, `views/transcript.html` 3. Never let a re-run overwrite a human correction The corrections are the most expensive data in the notebook. Files: `speakers.py`, `tests/test_corrections.py` ### Done when - [ ] Microphone segments are attributed without clustering - [ ] Remote labels are provisional until named - [ ] Naming a cluster propagates in one action - [ ] Re-running never overwrites a human attribution ### Watch out - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. ## Phase 4 · Tasks that nobody invented Decisions, actions and questions each point at the seconds of audio they came from, and an unassigned action stays unassigned. ### Steps 1. Resolve every citation before storing an item An unresolvable citation means the item was invented. Files: `items.py`, `tests/test_citations.py` 2. Make unassigned a state, not a blank Filling the blank is the moment this product starts lying. Files: `items.py`, `views/items.html` 3. Keep the due date as spoken words 'Early next week' is information; a parsed Tuesday is a fabrication. Files: `items.py` ### Done when - [ ] Every item's citations resolve or the item is dropped - [ ] Unowned actions display as unassigned - [ ] No item carries an unspoken calendar date - [ ] One click plays an item's source audio ### Watch out - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. ## Not in this build - A meeting bot that joins calls for you. - Calendar auto-join. - Team workspaces and shared retention. - Mobile capture. ## After v1, if you want it - Importing a recording the meeting platform made itself, which sidesteps the capture problem entirely when the platform offers one - A weekly review that lists every unassigned action older than a fortnight, which is the honest output of refusing to invent owners - Per-speaker voice profiles kept locally, so a regular participant is named across meetings rather than per meeting
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, transcripts and the database. Everything the notebook is. DATA_DIR=./data # Required. Chosen by name, because device indexes move between reboots. MIC_DEVICE=Built-in Microphone # Required. The call audio device. Getting this wrong records you twice. REMOTE_DEVICE=System Audio Capture # Required. How long a silent remote track is tolerated before the recording stops. SILENCE_ABORT_SECONDS=45 # Required. The drift the alignment test is allowed at the end of an hour. ALIGNMENT_TOLERANCE_MS=250 # Required. Recordings are deleted after this; transcripts and items are kept. AUDIO_RETENTION_DAYS=60 # Required. Stored with every segment, because two models give two transcripts. SPEECH_MODEL_PATH=./models/speech
PRODUCT.md
# Sembly AI · product brief ## Problem The core loop is buildable, but a dependable replacement becomes a real weekend or multi-day project. For Sembly AI, transcribe a call and produce structured minutes, tasks, and searchable topics. The hard boundary is meeting bots, team collaboration, automations, and broad integrations, plus capture reliability, integrations, and collaboration. ## Product outcome A notebook that either captures both sides of a call or tells you within a minute that it did not, keeps an hour of two-track audio in alignment, attributes your own voice by construction rather than by guess, and produces tasks that point back at the seconds they came from. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - desktop microphone access - whisper.cpp model files - optional OpenAI or Anthropic API key - local storage ## Explicit non-goals for v1 - A meeting bot that joins calls for you. - Calendar auto-join. - Team workspaces and shared retention. - Mobile capture. - meeting bots, team collaboration, automations, and broad integrations - calendar auto-join - reliable speaker diarization - mobile capture - team search and sharing ## Success criteria - A muted remote source is confirmed to abort the recording with a named cause - Alignment is confirmed within tolerance at the end of a long recording - Microphone segments are confirmed attributed without clustering - A fabricated citation is confirmed to drop its item - Expired audio is confirmed to leave transcripts and items intact
BRIEF.md
# Build brief · Sembly AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a personal meeting notebook that records your microphone and the call's audio as two separate tracks and turns them into minutes and tasks to replace Sembly AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero The hard part of this product is not the notes. It is getting the other person's voice into the file at all, and the web platform will let you believe you have it when you have not. A browser can ask for display capture with audio, but the specification and the documentation are explicit that support for audio tracks varies by browser and by the surface the user picks, and that the returned stream may contain no audio track even when audio was requested and system audio was asked for. On the desktop side the operating system gates it separately again: capturing what the speakers are playing is a different permission from the microphone, and a denial can present itself as a successful capture of silence rather than as an error. Check what your target platform and version require before you design the capture, because it decides whether this is a web app with a helper or a desktop app. So: record two tracks, never one mix. The microphone is one file, the call audio is another, and every segment knows which track it came from. Before a recording is accepted, measure both tracks and refuse a recording whose remote track is silent, with a message naming the permission to check. A notebook full of transcripts containing only your half of every conversation is the failure mode here, and it is silent, cumulative and discovered weeks later. One more rule: record only conversations you are in, and say out loud that you are recording before you start. There is no bot here that joins a call for you and no capture without a visible indicator. Store who was told with the recording. ### Stack (fixed, do not substitute) - Runtime: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records · audio capture is an operating system conversation and a hosted app cannot have it - Capture: Two independent tracks written to separate files as they arrive, never premixed · a mix throws away the only free speaker label you will ever get - Transcription: A local speech model per track, segments stored with track, start and end · the recordings are of other people and should not need to leave the machine - Alignment: A shared wall-clock timeline with per-track drift correction · two capture devices have two clocks and an hour is long enough for that to matter - Storage: SQLite beside the audio files, with search over segments · the notebook is worth more than any single meeting and has to survive a reinstall ### Data model (create this before Phase 1) - meetings(id, title, started_at, ended_at, consent_note, state) · the consent note is required before recording starts - tracks(id, meeting_id, kind, file_path, started_at_wall, sample_rate, frames_written, rms_summary) · kind is microphone or remote - segments(id, meeting_id, track_id, start_ms, end_ms, text, model_id, confidence) · times are on the shared timeline, not the track's own - speakers(id, meeting_id, label, display_name, source) · source is track for your own voice, cluster for a guess, manual for a correction - segment_speakers(segment_id, speaker_id, assigned_by) · assigned_by is diarisation or human - items(id, meeting_id, kind, text, owner_speaker_id, due_text, segment_ids_json, state) · kind is decision, action or question, and owner may be null forever - topics(id, meeting_id, label, segment_ids_json) · derived, rebuildable, never the only copy of anything - Every generated sentence in this schema carries the segment identifiers it came from. There is no summary row without them. ### Phase 1 · Two tracks, or an honest refusal Build: Enumerate capture devices and require the user to pick one for the microphone and one for the call audio, naming what each is. Write both tracks to disk as frames arrive, flushing often. Record each track's wall-clock start. Compute a rolling level for both tracks and, if the remote track stays below a floor for a stated interval, stop and say which permission to check. Trap: Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. Done when: A recording produces two files with independent frame counts and recorded wall-clock start times, checked on disk. A remote track that stays silent past the configured interval stops the recording with a message naming the permission, tested by muting the source. Killing the process mid-recording leaves both files playable up to the last flush, tested. Recording cannot start without a consent note stored on the meeting. Do not build yet: No transcription, speakers, notes or search yet. ### Phase 2 · One timeline, two clocks Build: Convert each track's frame index to a shared timeline using its recorded wall-clock start and its measured effective sample rate rather than its nominal one. Estimate drift by comparing elapsed wall time against frames written, and store the correction per track. Render the transcript as a single interleaved stream of segments ordered on the shared timeline. Trap: Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. Done when: A recording of an hour has both tracks aligned within a stated tolerance at the final minute, tested against a synchronised marker played into both. Segment times are stored on the shared timeline and the per-track drift correction is stored with the track. The interleaved transcript orders segments correctly across tracks, tested with overlapping speech. Playing a segment seeks to the right place in the right file. Do not build yet: No speaker names, notes, tasks or search yet. ### Phase 3 · Your voice is known, everyone else is a guess Build: Label every microphone-track segment with you, with certainty, because the track says so. Cluster only the remote track, present those labels as provisional, and let the user name them once per meeting with the name propagating to every segment in that cluster. Record whether a label came from clustering or from a human, and show the difference in the interface. Trap: Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. Done when: Every segment from the microphone track is attributed to the local speaker without clustering, verified in the database. Remote speaker labels are shown as provisional until named, and naming one propagates to that cluster's segments in one action. The source of every attribution is stored and shown. Re-running transcription does not overwrite a human attribution, tested. Do not build yet: No minutes, tasks or search yet. ### Phase 4 · Tasks that nobody invented Build: Generate items only from approved transcript text, requiring each item to cite the segment identifiers it came from. Drop any item whose citations do not resolve. Leave the owner null unless a speaker was named in the cited segments, and leave the due date as the words that were actually said rather than a parsed date. Every item plays its source audio in one click. Trap: Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. Done when: Every stored item has at least one segment identifier that resolves to a real segment in that meeting, enforced at write time and tested with a fabricated citation. An item whose cited segments name no speaker is stored with a null owner and displays as unassigned. Due dates are stored as the spoken words, and no item carries a calendar date that was not said. One click on any item plays the cited audio. Do not build yet: No cross-meeting search, retention or export yet. ### Phase 5 · A notebook rather than a pile Build: Index segments for full text search with the meeting, speaker and timestamp attached, so a result is a place in an audio file rather than a paragraph. Export a meeting as Markdown plus a transcript file plus its audio. Apply an audio retention period that deletes recordings while keeping transcripts and items, and show how much disk the notebook is using. Trap: Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread. Done when: A search result names the meeting, the speaker and the timestamp and plays from that point, tested. Audio older than the retention period is deleted and its transcript, items and citations survive with a marker that the audio is gone, tested with a clock override. Exporting a meeting produces Markdown, a timed transcript and the audio if present, and reimports into an empty instance. Total disk use by meeting is shown in the interface. Do not build yet: Nothing after this. ### Out of scope (and why) - A bot that joins meetings on your behalf, which records people who were told a person was attending and is exactly the capability this build refuses. - Calendar auto-join, which is the same refusal wearing a schedule. - Team workspaces and shared retention policies, which turn a personal notebook into a system holding other people's conversations under your name. - Mobile capture, which is a second set of audio permissions and a second alignment problem for very little gain. ### README must contain - That capturing call audio is an operating system permission separate from the microphone, and exactly which one to grant on your platform - That a refused system audio permission can look like a successful recording of silence, and how this build detects it - That two tracks are recorded and why a single mixed file would be worse - That speaker labels on the remote track are clustered guesses and your own track is not - Where recordings live, how long audio is kept, and what survives when it is deleted
ARCHITECTURE.md
# Architecture · Sembly AI ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records | audio capture is an operating system conversation and a hosted app cannot have it | | Capture | Two independent tracks written to separate files as they arrive, never premixed | a mix throws away the only free speaker label you will ever get | | Transcription | A local speech model per track, segments stored with track, start and end | the recordings are of other people and should not need to leave the machine | | Alignment | A shared wall-clock timeline with per-track drift correction | two capture devices have two clocks and an hour is long enough for that to matter | | Storage | SQLite beside the audio files, with search over segments | the notebook is worth more than any single meeting and has to survive a reinstall | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Capture | Device selection, two-track writing, silence detection. | Any capture backend, provided it stays two tracks and detects silence live. | | Timeline | Wall-clock starts, drift measurement, shared times. | Any method, provided alignment is tested at the end of a long recording. | | Transcription | Per-track segments and model identity. | Any speech model, provided segments keep their track. | | Speakers | Local certainty, remote clustering, human corrections. | Any clustering, provided corrections outrank it permanently. | | Items | Decisions, actions, questions and their citations. | Any model, provided unresolvable citations drop the item. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATA_DIR` · required · Recordings, transcripts and the database. Everything the notebook is. - `MIC_DEVICE` · required · Chosen by name, because device indexes move between reboots. - `REMOTE_DEVICE` · required · The call audio device. Getting this wrong records you twice. - `SILENCE_ABORT_SECONDS` · required · How long a silent remote track is tolerated before the recording stops. - `ALIGNMENT_TOLERANCE_MS` · required · The drift the alignment test is allowed at the end of an hour. - `AUDIO_RETENTION_DAYS` · required · Recordings are deleted after this; transcripts and items are kept. - `SPEECH_MODEL_PATH` · required · Stored with every segment, because two models give two transcripts. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Sembly AI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Python 3.12 with FastAPI and a plain HTML interface, running on the machine that records, Two independent tracks written to separate files as they arrive, never premixed, A local speech model per track, segments stored with track, start and end, A shared wall-clock timeline with per-track drift correction, SQLite beside the audio files, with search over segments. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. - Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread.
MILESTONES.md
# Delivery milestones · Sembly AI Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Two tracks, or an honest refusal A recording captures microphone and call audio separately, and a silent far side stops the recording rather than being discovered later. ### Steps 1. Make the user choose two devices by name Guessing which device is the call is how you get two microphones. Files: `capture.py`, `devices.py` 2. Flush both tracks continuously and record wall-clock starts The start times are what lets you align the tracks later. Files: `capture.py` 3. Fail loudly on a silent remote track This is the failure that ruins a notebook quietly. Files: `capture.py`, `tests/test_silence.py` ### Done when - [ ] Two files with separate frame counts and start times - [ ] A silent remote track stops the recording with a named cause - [ ] A kill mid-recording leaves both files playable - [ ] No recording starts without a consent note ### Watch out - Requesting system audio and getting a stream with no audio track, or a track of digital silence because a permission was refused without an error, produces a recording that looks complete and contains only you. Measure both tracks during capture, not afterwards, and make a silent remote track an error the user sees within the first minute rather than a discovery in the transcript. ## M2 · One timeline, two clocks Segments from both tracks line up at the end of an hour, not just at the start. ### Steps 1. Measure the effective sample rate, do not trust the nominal one The difference is invisible for a minute and fatal for an hour. Files: `timeline.py`, `tests/test_drift.py` 2. Store times on the shared timeline only Two sets of times in one table is how alignment bugs hide. Files: `transcribe.py` 3. Interleave for display, keep tracks separate in storage The display is a view; the tracks are the record. Files: `views/transcript.html` ### Done when - [ ] Tracks align within tolerance at the end of an hour - [ ] Drift corrections are stored per track - [ ] Interleaving is correct across overlapping speech - [ ] Clicking a segment seeks to the right file and offset ### Watch out - Assuming both devices produce exactly their nominal sample rate means a small constant error accumulates, and by the end of a sixty minute recording your reply sits several seconds before the question it answered, which makes the interleaved transcript read as nonsense and quietly ruins every note built on it. Measure frames against wall time per track, store the correction, and test alignment at the end of a long recording rather than the start. ## M3 · Your voice is known, everyone else is a guess Speaker labels distinguish what is certain from what was clustered, and corrections stick. ### Steps 1. Attribute the microphone track by construction It is the only speaker label in this build that is not a guess. Files: `speakers.py` 2. Cluster the remote track only, mark it provisional Show the user which labels are evidence and which are inference. Files: `diarise.py`, `views/transcript.html` 3. Never let a re-run overwrite a human correction The corrections are the most expensive data in the notebook. Files: `speakers.py`, `tests/test_corrections.py` ### Done when - [ ] Microphone segments are attributed without clustering - [ ] Remote labels are provisional until named - [ ] Naming a cluster propagates in one action - [ ] Re-running never overwrites a human attribution ### Watch out - Running diarisation across a premixed file, or across both tracks together, throws away the one piece of ground truth you actually have and then asks you to trust a guess about it, so your own sentences start being attributed to the client. Cluster the remote track only, and mark your own track by construction. ## M4 · Tasks that nobody invented Decisions, actions and questions each point at the seconds of audio they came from, and an unassigned action stays unassigned. ### Steps 1. Resolve every citation before storing an item An unresolvable citation means the item was invented. Files: `items.py`, `tests/test_citations.py` 2. Make unassigned a state, not a blank Filling the blank is the moment this product starts lying. Files: `items.py`, `views/items.html` 3. Keep the due date as spoken words 'Early next week' is information; a parsed Tuesday is a fabrication. Files: `items.py` ### Done when - [ ] Every item's citations resolve or the item is dropped - [ ] Unowned actions display as unassigned - [ ] No item carries an unspoken calendar date - [ ] One click plays an item's source audio ### Watch out - Asking a model for action items with owners produces a tidy table in which the quietest person in the meeting has been assigned three tasks they never accepted, and because the table looks authoritative nobody checks it against the recording. Require citations, resolve them, and make an unowned action a first class state rather than a gap to be filled. ## M5 · A notebook rather than a pile (production only) Search works across meetings, storage stays bounded, and everything can be taken elsewhere. ### Steps 1. Make a search result a position in a recording A paragraph with no timestamp is not checkable. Files: `search.py`, `views/search.html` 2. Expire audio, keep transcripts and items The recording is the liability; the notes are the asset. Files: `retention.py`, `tests/test_retention.py` 3. Export a meeting as files anyone can read Markdown and a timed transcript outlive this application. Files: `export.py` ### Done when - [ ] Search results play from the matched moment - [ ] Expired audio leaves transcripts and citations intact - [ ] Export round-trips into an empty instance - [ ] Disk use per meeting is visible ### Watch out - Keeping every recording forever turns a notebook into tens of gigabytes of other people's voices on a laptop that travels, and the day it is lost the problem is not your notes. Set a retention period on audio from the first release, delete on schedule, and keep the transcripts and items which are what you actually reread.
OPERATIONS.md
# Operations · Sembly AI ## Backup The database and the transcripts first, the audio second. Corrections and items cannot be regenerated; the audio can at least be re-transcribed while it exists, and after retention it is gone by design anyway. ## Restore Restore the database and transcripts, then confirm that item citations still resolve and that search returns results. If audio was restored too, check a segment plays at the right offset, because a partial audio restore breaks alignment silently. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Recordings aborted for a silent remote track, measured drift per recording against the tolerance, transcription failures, and disk used against the retention period. A drift figure creeping up usually means a device changed. ## Incident checklist The incident here is a fortnight of meetings where only your voice was recorded, or a set of minutes assigning work to someone who never agreed. Both are caught by gates rather than by noticing: the silence abort and the citation resolution. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A muted remote source is confirmed to abort the recording with a named cause - [ ] Alignment is confirmed within tolerance at the end of a long recording - [ ] Microphone segments are confirmed attributed without clustering - [ ] A fabricated citation is confirmed to drop its item - [ ] Expired audio is confirmed to leave transcripts and items intact ## Launch constraint Do not market omitted Sembly AI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Recordings, transcripts and the database. Everything the notebook is. DATA_DIR=./data # Required. Chosen by name, because device indexes move between reboots. MIC_DEVICE=Built-in Microphone # Required. The call audio device. Getting this wrong records you twice. REMOTE_DEVICE=System Audio Capture # Required. How long a silent remote track is tolerated before the recording stops. SILENCE_ABORT_SECONDS=45 # Required. The drift the alignment test is allowed at the end of an hour. ALIGNMENT_TOLERANCE_MS=250 # Required. Recordings are deleted after this; transcripts and items are kept. AUDIO_RETENTION_DAYS=60 # Required. Stored with every segment, because two models give two transcripts. SPEECH_MODEL_PATH=./models/speech
$ choose a build depth, inspect the files, then open the complete pack in your agent
People still pay for Sembly AI because a meeting tool must capture every call without surprising anyone, then make the result searchable and shareable across a team. The recurring cost buys audio permissions, model updates, calendar APIs, storage, speaker correction, and sync, not just the visible interface.
xmeeting bots, team collaboration, automations, and broad integrations
xcalendar auto-join
xreliable speaker diarization
xmobile capture
xteam search and sharing
share on X ↗"I just replaced Sembly AI ($17/mo) with one prompt"
Don't feel like building it? These folks already made it free.
OpenWhisprRecords or imports audio, labels speakers, generates notes and keeps the result searchable.open source↗SSpeakrSelf-hosted transcription, templates, topics, tasks, search and chat in one polished web app.open source↗
MuesliLocal transcripts plus configurable notes, decisions and action items; the searchable notebook stays on your Mac.open source↗all 4 free alternatives to Sembly AI →· no votes, no pay-to-list · just what's real
Sembly AI pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| basic | $17/user | $10/user | Unlimited meeting hours; 5 upload hours per paid user; 1-year history; 3 native automations. |
| pro | $29/user | $20/user | Up to 40 users/workspace; 10 upload hours, 5 AI documents, 5 AI insights, 5 native automations, and 5 webhooks per paid user; 2-year history. |
| max | $39/user | $30/user | Minimum 3 users, maximum 500; 15 upload hours, 40 AI documents, 40 AI insights, 20 automations, and 20 webhooks per paid user; unlimited history. |
| enterprise | custom | — | Custom invoicing, security reviews, support, and success package; no public numeric price. |
free tierno free tier
billingmonthly + annual (annual advertised 30% lower); free trials offered with no duration published
hidden costsMAX requires at least 3 paid users. Upload hours, AI documents, AI insights, automations, and webhooks remain per-user quotas despite unlimited meeting hours. The former $15 'Professional' plan on file is not in the current live lineup.
verified 2026-08-14 · source ↗
Vibecode Sembly AI
Kinda. The core of Sembly AI is buildable in a weekend with the prompt on this page, but there are real gaps: meeting bots, team collaboration, automations, and broad integrations, calendar auto-join. Read the honest list above before committing.
How much does Sembly AI cost?
Sembly AI costs about $17/month (Basic, checked 2026-08-14), which is $204 per year.
What do I lose by replacing Sembly AI?
Honestly: meeting bots, team collaboration, automations, and broad integrations; calendar auto-join; reliable speaker diarization; mobile capture; team search and sharing. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Sembly AI?
Yes: OpenWhispr (Records or imports audio, labels speakers, generates notes and keeps the result searchable.) Speakr (Self-hosted transcription, templates, topics, tasks, search and chat in one polished web app.) Muesli (Local transcripts plus configurable notes, decisions and action items; the searchable notebook stays on your Mac.) All 4 curated free alternatives are at vibecodeit.com/sembly-ai/alternatives. The prompt is for when you want it exactly your way.