KINDA · weekend project
The pipeline is no longer exotic: capture mic audio, segment it with a voice activity detector, transcribe with Whisper, translate, speak it back with a local TTS voice. An agent can wire that into a working local app in a weekend and it will genuinely translate a conversation. What it will not do out of the gate is stay graceful for an hour: chunk boundaries clip words, speaker turns bleed together, latency creeps as the buffer grows, and the sentence by sentence pacing that makes these apps usable in real conversation is a tuning problem, not a coding problem. You also get no phone app, which is where voice translation actually happens. Fine for a desk setup and travel prep, unconvincing when you are holding it out to a stranger in a market.
the project pack5 files · written for this build, step by step
You are building a lean indie version of ekto. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # ekto · indie build A speech translation pipeline where every design decision is a latency trade: recognition wants complete sentences, translation wants complete clauses, speech synthesis wants complete phrases, and a conversation wants none of them to take longer than a natural pause. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Recognition | Streaming-shaped recognition with voice activity segmentation | the segment boundary decides both latency and translation quality | | Translation | A model run per stable segment, never per partial word | translating a half-sentence produces confident nonsense | | Synthesis | Speech generated per translated segment with a small queue | the output must not fall behind the conversation | | Latency | A stated budget per stage, measured continuously | the pipeline is only usable if the total stays under a conversational pause | | Runtime | Local models where they fit, with the trade stated | conversations are private and the alternative is sending both sides to a service | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Orchestrates recognition, translation and synthesis. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **Speech recognition with word timings for both languages** · free locally, needs machine time - Why: Segmentation and latency measurement both depend on timings, and quality differs sharply between languages. - Get it: A local engine with models for both languages, checked for whether it emits word timings. - Verify: Both languages recognise with word timings - [ ] **A translation model for your language pair** · free locally; per-character if hosted - Why: Quality varies enormously by pair, and a pair that translates poorly makes every other decision in the build irrelevant. - Get it: A local or hosted model for your specific pair, evaluated on real conversational speech rather than written text. - Verify: Twenty real conversational sentences translate acceptably in both directions - [ ] **Speech synthesis for both languages** · free for open models - Why: Output speech is half the experience and synthesis latency is a large part of the total budget. - Get it: A local engine with voices for both languages, measured for time to first audio. - Verify: A sentence synthesises in both languages within your latency budget - [ ] **A recorded real bilingual conversation** · free, and requires asking people - Why: Every parameter here is tuned against real speech with interruptions, overlaps and people who do not pause, and none of that appears in read-aloud test sentences. - Get it: Record a genuine conversation with both speakers' consent, including a stretch where they talk over each other. - Verify: A recording exists with natural pauses, overlap and at least one long unbroken utterance ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Commercial translation quality across many language pairs. - Preserving the speaker's voice in translation. - Simultaneous interpretation quality. - Use where a mistranslation carries real consequence. - Long session reliability: memory growth, drifting segmentation and dropped turns after the first 20 minutes - Clean sentence by sentence pacing and turn detection, which is most of the perceived quality - A mobile app, so no translating anything while standing up - Offline or low-bandwidth behavior tuned for actual travel - Latency budgets someone else already fought for: streaming partial results instead of waiting for a full segment If one of those is essential to you, that is the reason to keep paying for ekto, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · ekto The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me near real-time speech translation for a two-way conversation to replace Ekto. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Recognition: Streaming-shaped recognition with voice activity segmentation · the segment boundary decides both latency and translation quality - Translation: A model run per stable segment, never per partial word · translating a half-sentence produces confident nonsense - Synthesis: Speech generated per translated segment with a small queue · the output must not fall behind the conversation - Latency: A stated budget per stage, measured continuously · the pipeline is only usable if the total stays under a conversational pause - Runtime: Local models where they fit, with the trade stated · conversations are private and the alternative is sending both sides to a service ### Data model (create this before Phase 1) - conversations(id, language_a, language_b, started_at, ended_at, disclosed_at) - segments(id, conversation_id, speaker_side, start_ms, end_ms, source_text, stability, superseded_by) - translations(segment_id, target_language, text, model, confidence, at) - utterances(translation_id, audio_path, played_at, duration_ms) - latencies(segment_id, stage, ms) · stage is recognition, translation, synthesis or total - corrections(segment_id, before, after, at) · when recognition revised after translation had already spoken - A segment records whether it was superseded, because a translation of a revised segment is a correction the listener already heard the wrong version of. ### Phase 1 · Deciding where a sentence ends Build: Segment incoming speech on voice activity and pause length, tuned so segments are complete clauses rather than fragments. Measure the delay each segmentation choice adds. Handle a speaker who does not pause. Trap: Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. Done when: Segments are complete clauses in normal speech, checked against a recording of real conversation. A speaker who does not pause is segmented on a maximum length rather than never. The added delay per segmentation setting is measured and recorded. Segmentation works for both languages, which may pause differently. Do not build yet: No translation, synthesis or playback. ### Phase 2 · Translating complete thoughts Build: Translate per stable segment with a confidence. Never translate a partial. Detect and flag low confidence rather than speaking it as though certain. Handle names, numbers and terms that must not be translated. Trap: Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. Done when: Only stable segments are translated, verified by checking no partial reaches the translator. Confidence is reported per translation and low confidence is signalled to the listener rather than spoken plainly. Names and numbers survive translation intact, tested with a list of both. A term list can force specific translations. Do not build yet: No synthesis or playback yet. ### Phase 3 · Speaking without falling behind Build: Synthesise per translated segment into a small queue. Detect the queue growing, which means the pipeline is slower than the conversation, and respond by shortening rather than by accumulating. Measure total latency continuously. Trap: A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. Done when: Total latency from speech to translated audio is measured and stated. A fast exchange does not accumulate an unbounded queue, tested with rapid alternating speech. Exceeding the queue bound produces a stated behaviour rather than growing lag. Playback does not overlap the original speaker's next utterance destructively. Do not build yet: No two-way handling or corrections yet. ### Phase 4 · Two people, two directions Build: Handle both directions concurrently, with separate segmentation per side. Avoid translating your own synthesised output back. Handle both people speaking at once, which is normal in conversation. Trap: Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. Done when: Both directions translate concurrently without either blocking the other. Synthesised output is never recognised and re-translated, tested by deliberately playing output into the microphone. Overlapping speech from both parties is handled rather than producing interleaved nonsense. Each side's segmentation is tuned independently. Do not build yet: No corrections or transcripts yet. ### Phase 5 · Corrections, transcripts and consent Build: Handle recognition revising a segment already translated and spoken, by issuing a correction rather than silently moving on. Offer a transcript, requiring disclosure to record. Delete conversations on a short retention by default. Trap: Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript. Done when: A revised segment produces an audible or visible correction rather than a silent transcript edit. Recording a transcript requires a disclosure record and is refused without one. Conversations delete on a short retention by default. The latency and confidence record is available afterwards for a conversation that went wrong. Do not build yet: Nothing after this. ### Out of scope (and why) - Commercial translation quality across many language pairs, which is where the hosted services have invested most and where the gap is widest for low-resource languages. - Preserving the speaker's voice in the translation, which raises consent questions about voice cloning. - Simultaneous interpretation quality, which is a professional skill involving anticipation that no pipeline of this shape reproduces. - Use in any setting where a mistranslation carries real consequence, such as medical or legal, which needs a professional interpreter. ### README must contain - That segment boundaries are the central trade between latency and translation quality - That partial segments are never translated because they produce confident nonsense - That low confidence is signalled rather than spoken with equal fluency - That the synthesis queue is bounded, because being a minute behind is worse than skipping - That a corrected segment produces a spoken or visible correction rather than a silent fix ===== AGENTS.md ===== # Agent instructions · ekto indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Streaming-shaped recognition with voice activity segmentation, A model run per stable segment, never per partial word, Speech generated per translated segment with a small queue, A stated budget per stage, measured continuously, Local models where they fit, with the trade stated. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. ===== BUILD_PLAN.md ===== # Build plan · ekto A speech translation pipeline where every design decision is a latency trade: recognition wants complete sentences, translation wants complete clauses, speech synthesis wants complete phrases, and a conversation wants none of them to take longer than a natural pause. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Deciding where a sentence ends Speech is cut into segments that are complete enough to translate. ### Steps 1. Segment on voice activity and pause length Both. Activity alone cuts mid-clause. Files: `src/segment.py` 2. Cap segment length A speaker who does not pause must still be translated. Files: `src/segment.py` 3. Measure the delay each setting adds The trade is the project. Measure it rather than guessing. Files: `src/latency.py` ### Done when - [ ] Segments are complete clauses in real speech - [ ] A non-pausing speaker is still segmented - [ ] Delay per setting is measured - [ ] Both languages segment acceptably ### Watch out - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. ## Phase 2 · Translating complete thoughts Each segment translates correctly and the pipeline knows when it is unsure. ### Steps 1. Translate only stable segments A partial clause translates into confident nonsense. Files: `src/translate.py` 2. Signal low confidence to the listener Equal fluency for a guess and a certainty is the dangerous default. Files: `src/confidence.py` 3. Protect names, numbers and terms A mistranslated number in a conversation is a real problem. Files: `src/protect.py`, `tests/test_protect.py` ### Done when - [ ] No partial segment reaches the translator - [ ] Low confidence is signalled rather than spoken plainly - [ ] Names and numbers survive intact - [ ] A term list forces specific translations ### Watch out - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. ## Phase 3 · Speaking without falling behind Translated speech plays promptly and the queue never grows unbounded. ### Steps 1. Synthesise into a bounded queue Unbounded means a minute behind during a fast exchange. Files: `src/synthesise.py` 2. Respond to queue growth by shortening Not by accumulating. Being far behind is worse than skipping. Files: `src/queue.py` 3. Measure end-to-end latency continuously It is the number the whole build is judged on. Files: `src/latency.py` ### Done when - [ ] Total latency is measured and stated - [ ] A fast exchange does not accumulate lag - [ ] Exceeding the queue bound has a stated behaviour - [ ] Playback does not destructively overlap speech ### Watch out - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. ## Phase 4 · Two people, two directions Both sides are translated without the tool talking over itself. ### Steps 1. Run both directions concurrently Independently segmented. Languages pause differently. Files: `src/duplex.py` 2. Prevent the feedback loop Gate input during playback or identify your own output. Test it deliberately. Files: `src/feedback.py`, `tests/test_loop.py` 3. Handle simultaneous speech People talk over each other. It is not an error case. Files: `src/duplex.py` ### Done when - [ ] Both directions translate concurrently - [ ] Synthesised output is never re-translated - [ ] Overlapping speech is handled - [ ] Each side segments independently ### Watch out - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. ## Not in this build - Commercial translation quality across many language pairs. - Preserving the speaker's voice in translation. - Simultaneous interpretation quality. - Use where a mistranslation carries real consequence. ## After v1, if you want it - A shared screen view showing both transcripts, which helps far more than it sounds - Per-domain term lists, since the vocabulary that matters is usually specific - Latency tuning presets, so a slow conversation and a fast one use different settings ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. One side of the conversation. Segmentation is tuned per language. LANGUAGE_A=en # Required. The other side. Pause patterns differ, so tuning is per side. LANGUAGE_B=pt # Required. A speaker who does not pause must still be translated. MAX_SEGMENT_MS=6000 # Required. The central trade. Shorter is faster and produces fragments. PAUSE_THRESHOLD_MS=500 # Required. Bounded. Growing lag is worse than dropping a segment. SYNTHESIS_QUEUE_MAX=2 # Required. Total speech to translated audio. Beyond this a conversation stops working. LATENCY_BUDGET_MS=2500
You are building a lean indie version of ekto. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # ekto · indie build A speech translation pipeline where every design decision is a latency trade: recognition wants complete sentences, translation wants complete clauses, speech synthesis wants complete phrases, and a conversation wants none of them to take longer than a natural pause. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Recognition | Streaming-shaped recognition with voice activity segmentation | the segment boundary decides both latency and translation quality | | Translation | A model run per stable segment, never per partial word | translating a half-sentence produces confident nonsense | | Synthesis | Speech generated per translated segment with a small queue | the output must not fall behind the conversation | | Latency | A stated budget per stage, measured continuously | the pipeline is only usable if the total stays under a conversational pause | | Runtime | Local models where they fit, with the trade stated | conversations are private and the alternative is sending both sides to a service | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Orchestrates recognition, translation and synthesis. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **Speech recognition with word timings for both languages** · free locally, needs machine time - Why: Segmentation and latency measurement both depend on timings, and quality differs sharply between languages. - Get it: A local engine with models for both languages, checked for whether it emits word timings. - Verify: Both languages recognise with word timings - [ ] **A translation model for your language pair** · free locally; per-character if hosted - Why: Quality varies enormously by pair, and a pair that translates poorly makes every other decision in the build irrelevant. - Get it: A local or hosted model for your specific pair, evaluated on real conversational speech rather than written text. - Verify: Twenty real conversational sentences translate acceptably in both directions - [ ] **Speech synthesis for both languages** · free for open models - Why: Output speech is half the experience and synthesis latency is a large part of the total budget. - Get it: A local engine with voices for both languages, measured for time to first audio. - Verify: A sentence synthesises in both languages within your latency budget - [ ] **A recorded real bilingual conversation** · free, and requires asking people - Why: Every parameter here is tuned against real speech with interruptions, overlaps and people who do not pause, and none of that appears in read-aloud test sentences. - Get it: Record a genuine conversation with both speakers' consent, including a stretch where they talk over each other. - Verify: A recording exists with natural pauses, overlap and at least one long unbroken utterance ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Commercial translation quality across many language pairs. - Preserving the speaker's voice in translation. - Simultaneous interpretation quality. - Use where a mistranslation carries real consequence. - Long session reliability: memory growth, drifting segmentation and dropped turns after the first 20 minutes - Clean sentence by sentence pacing and turn detection, which is most of the perceived quality - A mobile app, so no translating anything while standing up - Offline or low-bandwidth behavior tuned for actual travel - Latency budgets someone else already fought for: streaming partial results instead of waiting for a full segment If one of those is essential to you, that is the reason to keep paying for ekto, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · ekto The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me near real-time speech translation for a two-way conversation to replace Ekto. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Recognition: Streaming-shaped recognition with voice activity segmentation · the segment boundary decides both latency and translation quality - Translation: A model run per stable segment, never per partial word · translating a half-sentence produces confident nonsense - Synthesis: Speech generated per translated segment with a small queue · the output must not fall behind the conversation - Latency: A stated budget per stage, measured continuously · the pipeline is only usable if the total stays under a conversational pause - Runtime: Local models where they fit, with the trade stated · conversations are private and the alternative is sending both sides to a service ### Data model (create this before Phase 1) - conversations(id, language_a, language_b, started_at, ended_at, disclosed_at) - segments(id, conversation_id, speaker_side, start_ms, end_ms, source_text, stability, superseded_by) - translations(segment_id, target_language, text, model, confidence, at) - utterances(translation_id, audio_path, played_at, duration_ms) - latencies(segment_id, stage, ms) · stage is recognition, translation, synthesis or total - corrections(segment_id, before, after, at) · when recognition revised after translation had already spoken - A segment records whether it was superseded, because a translation of a revised segment is a correction the listener already heard the wrong version of. ### Phase 1 · Deciding where a sentence ends Build: Segment incoming speech on voice activity and pause length, tuned so segments are complete clauses rather than fragments. Measure the delay each segmentation choice adds. Handle a speaker who does not pause. Trap: Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. Done when: Segments are complete clauses in normal speech, checked against a recording of real conversation. A speaker who does not pause is segmented on a maximum length rather than never. The added delay per segmentation setting is measured and recorded. Segmentation works for both languages, which may pause differently. Do not build yet: No translation, synthesis or playback. ### Phase 2 · Translating complete thoughts Build: Translate per stable segment with a confidence. Never translate a partial. Detect and flag low confidence rather than speaking it as though certain. Handle names, numbers and terms that must not be translated. Trap: Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. Done when: Only stable segments are translated, verified by checking no partial reaches the translator. Confidence is reported per translation and low confidence is signalled to the listener rather than spoken plainly. Names and numbers survive translation intact, tested with a list of both. A term list can force specific translations. Do not build yet: No synthesis or playback yet. ### Phase 3 · Speaking without falling behind Build: Synthesise per translated segment into a small queue. Detect the queue growing, which means the pipeline is slower than the conversation, and respond by shortening rather than by accumulating. Measure total latency continuously. Trap: A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. Done when: Total latency from speech to translated audio is measured and stated. A fast exchange does not accumulate an unbounded queue, tested with rapid alternating speech. Exceeding the queue bound produces a stated behaviour rather than growing lag. Playback does not overlap the original speaker's next utterance destructively. Do not build yet: No two-way handling or corrections yet. ### Phase 4 · Two people, two directions Build: Handle both directions concurrently, with separate segmentation per side. Avoid translating your own synthesised output back. Handle both people speaking at once, which is normal in conversation. Trap: Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. Done when: Both directions translate concurrently without either blocking the other. Synthesised output is never recognised and re-translated, tested by deliberately playing output into the microphone. Overlapping speech from both parties is handled rather than producing interleaved nonsense. Each side's segmentation is tuned independently. Do not build yet: No corrections or transcripts yet. ### Phase 5 · Corrections, transcripts and consent Build: Handle recognition revising a segment already translated and spoken, by issuing a correction rather than silently moving on. Offer a transcript, requiring disclosure to record. Delete conversations on a short retention by default. Trap: Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript. Done when: A revised segment produces an audible or visible correction rather than a silent transcript edit. Recording a transcript requires a disclosure record and is refused without one. Conversations delete on a short retention by default. The latency and confidence record is available afterwards for a conversation that went wrong. Do not build yet: Nothing after this. ### Out of scope (and why) - Commercial translation quality across many language pairs, which is where the hosted services have invested most and where the gap is widest for low-resource languages. - Preserving the speaker's voice in the translation, which raises consent questions about voice cloning. - Simultaneous interpretation quality, which is a professional skill involving anticipation that no pipeline of this shape reproduces. - Use in any setting where a mistranslation carries real consequence, such as medical or legal, which needs a professional interpreter. ### README must contain - That segment boundaries are the central trade between latency and translation quality - That partial segments are never translated because they produce confident nonsense - That low confidence is signalled rather than spoken with equal fluency - That the synthesis queue is bounded, because being a minute behind is worse than skipping - That a corrected segment produces a spoken or visible correction rather than a silent fix ===== AGENTS.md ===== # Agent instructions · ekto indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Streaming-shaped recognition with voice activity segmentation, A model run per stable segment, never per partial word, Speech generated per translated segment with a small queue, A stated budget per stage, measured continuously, Local models where they fit, with the trade stated. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. ===== BUILD_PLAN.md ===== # Build plan · ekto A speech translation pipeline where every design decision is a latency trade: recognition wants complete sentences, translation wants complete clauses, speech synthesis wants complete phrases, and a conversation wants none of them to take longer than a natural pause. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Deciding where a sentence ends Speech is cut into segments that are complete enough to translate. ### Steps 1. Segment on voice activity and pause length Both. Activity alone cuts mid-clause. Files: `src/segment.py` 2. Cap segment length A speaker who does not pause must still be translated. Files: `src/segment.py` 3. Measure the delay each setting adds The trade is the project. Measure it rather than guessing. Files: `src/latency.py` ### Done when - [ ] Segments are complete clauses in real speech - [ ] A non-pausing speaker is still segmented - [ ] Delay per setting is measured - [ ] Both languages segment acceptably ### Watch out - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. ## Phase 2 · Translating complete thoughts Each segment translates correctly and the pipeline knows when it is unsure. ### Steps 1. Translate only stable segments A partial clause translates into confident nonsense. Files: `src/translate.py` 2. Signal low confidence to the listener Equal fluency for a guess and a certainty is the dangerous default. Files: `src/confidence.py` 3. Protect names, numbers and terms A mistranslated number in a conversation is a real problem. Files: `src/protect.py`, `tests/test_protect.py` ### Done when - [ ] No partial segment reaches the translator - [ ] Low confidence is signalled rather than spoken plainly - [ ] Names and numbers survive intact - [ ] A term list forces specific translations ### Watch out - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. ## Phase 3 · Speaking without falling behind Translated speech plays promptly and the queue never grows unbounded. ### Steps 1. Synthesise into a bounded queue Unbounded means a minute behind during a fast exchange. Files: `src/synthesise.py` 2. Respond to queue growth by shortening Not by accumulating. Being far behind is worse than skipping. Files: `src/queue.py` 3. Measure end-to-end latency continuously It is the number the whole build is judged on. Files: `src/latency.py` ### Done when - [ ] Total latency is measured and stated - [ ] A fast exchange does not accumulate lag - [ ] Exceeding the queue bound has a stated behaviour - [ ] Playback does not destructively overlap speech ### Watch out - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. ## Phase 4 · Two people, two directions Both sides are translated without the tool talking over itself. ### Steps 1. Run both directions concurrently Independently segmented. Languages pause differently. Files: `src/duplex.py` 2. Prevent the feedback loop Gate input during playback or identify your own output. Test it deliberately. Files: `src/feedback.py`, `tests/test_loop.py` 3. Handle simultaneous speech People talk over each other. It is not an error case. Files: `src/duplex.py` ### Done when - [ ] Both directions translate concurrently - [ ] Synthesised output is never re-translated - [ ] Overlapping speech is handled - [ ] Each side segments independently ### Watch out - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. ## Not in this build - Commercial translation quality across many language pairs. - Preserving the speaker's voice in translation. - Simultaneous interpretation quality. - Use where a mistranslation carries real consequence. ## After v1, if you want it - A shared screen view showing both transcripts, which helps far more than it sounds - Per-domain term lists, since the vocabulary that matters is usually specific - Latency tuning presets, so a slow conversation and a fast one use different settings ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. One side of the conversation. Segmentation is tuned per language. LANGUAGE_A=en # Required. The other side. Pause patterns differ, so tuning is per side. LANGUAGE_B=pt # Required. A speaker who does not pause must still be translated. MAX_SEGMENT_MS=6000 # Required. The central trade. Shorter is faster and produces fragments. PAUSE_THRESHOLD_MS=500 # Required. Bounded. Growing lag is worse than dropping a segment. SYNTHESIS_QUEUE_MAX=2 # Required. Total speech to translated audio. Beyond this a conversation stops working. LATENCY_BUDGET_MS=2500
You are building a production product version of ekto. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # ekto · product brief ## Problem The pipeline is no longer exotic: capture mic audio, segment it with a voice activity detector, transcribe with Whisper, translate, speak it back with a local TTS voice. An agent can wire that into a working local app in a weekend and it will genuinely translate a conversation. What it will not do out of the gate is stay graceful for an hour: chunk boundaries clip words, speaker turns bleed together, latency creeps as the buffer grows, and the sentence by sentence pacing that makes these apps usable in real conversation is a tuning problem, not a coding problem. You also get no phone app, which is where voice translation actually happens. Fine for a desk setup and travel prep, unconvincing when you are holding it out to a stranger in a market. ## Product outcome A translation pipeline that stays inside a conversational pause, never speaks a guess with the confidence of a certainty, and does not translate its own output back into a loop. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Python 3.11 and a machine with at least 8GB RAM, GPU strongly preferred - Local model downloads: faster-whisper and a Piper voice per target language - A browser with mic permission, or an API key if you swap in a hosted translation model - Headphones, otherwise the TTS output feeds back into the mic ## Explicit non-goals for v1 - Commercial translation quality across many language pairs. - Preserving the speaker's voice in translation. - Simultaneous interpretation quality. - Use where a mistranslation carries real consequence. - Long session reliability: memory growth, drifting segmentation and dropped turns after the first 20 minutes - Clean sentence by sentence pacing and turn detection, which is most of the perceived quality - A mobile app, so no translating anything while standing up - Offline or low-bandwidth behavior tuned for actual travel - Latency budgets someone else already fought for: streaming partial results instead of waiting for a full segment ## Success criteria - No partial segment is confirmed to reach the translator - Total latency is confirmed measured and within budget - A fast exchange is confirmed not to accumulate lag - Synthesised output is confirmed never re-translated - A revised segment is confirmed to produce a correction ===== BRIEF.md ===== # Build brief · ekto The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me near real-time speech translation for a two-way conversation to replace Ekto. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Recognition: Streaming-shaped recognition with voice activity segmentation · the segment boundary decides both latency and translation quality - Translation: A model run per stable segment, never per partial word · translating a half-sentence produces confident nonsense - Synthesis: Speech generated per translated segment with a small queue · the output must not fall behind the conversation - Latency: A stated budget per stage, measured continuously · the pipeline is only usable if the total stays under a conversational pause - Runtime: Local models where they fit, with the trade stated · conversations are private and the alternative is sending both sides to a service ### Data model (create this before Phase 1) - conversations(id, language_a, language_b, started_at, ended_at, disclosed_at) - segments(id, conversation_id, speaker_side, start_ms, end_ms, source_text, stability, superseded_by) - translations(segment_id, target_language, text, model, confidence, at) - utterances(translation_id, audio_path, played_at, duration_ms) - latencies(segment_id, stage, ms) · stage is recognition, translation, synthesis or total - corrections(segment_id, before, after, at) · when recognition revised after translation had already spoken - A segment records whether it was superseded, because a translation of a revised segment is a correction the listener already heard the wrong version of. ### Phase 1 · Deciding where a sentence ends Build: Segment incoming speech on voice activity and pause length, tuned so segments are complete clauses rather than fragments. Measure the delay each segmentation choice adds. Handle a speaker who does not pause. Trap: Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. Done when: Segments are complete clauses in normal speech, checked against a recording of real conversation. A speaker who does not pause is segmented on a maximum length rather than never. The added delay per segmentation setting is measured and recorded. Segmentation works for both languages, which may pause differently. Do not build yet: No translation, synthesis or playback. ### Phase 2 · Translating complete thoughts Build: Translate per stable segment with a confidence. Never translate a partial. Detect and flag low confidence rather than speaking it as though certain. Handle names, numbers and terms that must not be translated. Trap: Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. Done when: Only stable segments are translated, verified by checking no partial reaches the translator. Confidence is reported per translation and low confidence is signalled to the listener rather than spoken plainly. Names and numbers survive translation intact, tested with a list of both. A term list can force specific translations. Do not build yet: No synthesis or playback yet. ### Phase 3 · Speaking without falling behind Build: Synthesise per translated segment into a small queue. Detect the queue growing, which means the pipeline is slower than the conversation, and respond by shortening rather than by accumulating. Measure total latency continuously. Trap: A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. Done when: Total latency from speech to translated audio is measured and stated. A fast exchange does not accumulate an unbounded queue, tested with rapid alternating speech. Exceeding the queue bound produces a stated behaviour rather than growing lag. Playback does not overlap the original speaker's next utterance destructively. Do not build yet: No two-way handling or corrections yet. ### Phase 4 · Two people, two directions Build: Handle both directions concurrently, with separate segmentation per side. Avoid translating your own synthesised output back. Handle both people speaking at once, which is normal in conversation. Trap: Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. Done when: Both directions translate concurrently without either blocking the other. Synthesised output is never recognised and re-translated, tested by deliberately playing output into the microphone. Overlapping speech from both parties is handled rather than producing interleaved nonsense. Each side's segmentation is tuned independently. Do not build yet: No corrections or transcripts yet. ### Phase 5 · Corrections, transcripts and consent Build: Handle recognition revising a segment already translated and spoken, by issuing a correction rather than silently moving on. Offer a transcript, requiring disclosure to record. Delete conversations on a short retention by default. Trap: Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript. Done when: A revised segment produces an audible or visible correction rather than a silent transcript edit. Recording a transcript requires a disclosure record and is refused without one. Conversations delete on a short retention by default. The latency and confidence record is available afterwards for a conversation that went wrong. Do not build yet: Nothing after this. ### Out of scope (and why) - Commercial translation quality across many language pairs, which is where the hosted services have invested most and where the gap is widest for low-resource languages. - Preserving the speaker's voice in the translation, which raises consent questions about voice cloning. - Simultaneous interpretation quality, which is a professional skill involving anticipation that no pipeline of this shape reproduces. - Use in any setting where a mistranslation carries real consequence, such as medical or legal, which needs a professional interpreter. ### README must contain - That segment boundaries are the central trade between latency and translation quality - That partial segments are never translated because they produce confident nonsense - That low confidence is signalled rather than spoken with equal fluency - That the synthesis queue is bounded, because being a minute behind is worse than skipping - That a corrected segment produces a spoken or visible correction rather than a silent fix ===== ARCHITECTURE.md ===== # Architecture · ekto ## Stack | Part | Choice | Why | | --- | --- | --- | | Recognition | Streaming-shaped recognition with voice activity segmentation | the segment boundary decides both latency and translation quality | | Translation | A model run per stable segment, never per partial word | translating a half-sentence produces confident nonsense | | Synthesis | Speech generated per translated segment with a small queue | the output must not fall behind the conversation | | Latency | A stated budget per stage, measured continuously | the pipeline is only usable if the total stays under a conversational pause | | Runtime | Local models where they fit, with the trade stated | conversations are private and the alternative is sending both sides to a service | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Segmentation | Cutting speech into translatable units. | Any approach, provided the latency it adds is measured rather than assumed. | | Translation | Stable segments and confidence. | Any model, provided partials are never sent and uncertainty is signalled. | | Synthesis | Bounded output queue. | Any engine, provided the queue cannot grow unbounded. | | Duplex | Both directions and feedback prevention. | Any implementation, provided the loop is tested deliberately. | | Corrections | Handling revised segments audibly. | Any presentation, provided a silent transcript fix is not the whole response. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `LANGUAGE_A` · required · One side of the conversation. Segmentation is tuned per language. - `LANGUAGE_B` · required · The other side. Pause patterns differ, so tuning is per side. - `MAX_SEGMENT_MS` · required · A speaker who does not pause must still be translated. - `PAUSE_THRESHOLD_MS` · required · The central trade. Shorter is faster and produces fragments. - `SYNTHESIS_QUEUE_MAX` · required · Bounded. Growing lag is worse than dropping a segment. - `LATENCY_BUDGET_MS` · required · Total speech to translated audio. Beyond this a conversation stops working. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · ekto product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Streaming-shaped recognition with voice activity segmentation, A model run per stable segment, never per partial word, Speech generated per translated segment with a small queue, A stated budget per stage, measured continuously, Local models where they fit, with the trade stated. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. - Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript. ===== MILESTONES.md ===== # Delivery milestones · ekto Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Deciding where a sentence ends Speech is cut into segments that are complete enough to translate. ### Steps 1. Segment on voice activity and pause length Both. Activity alone cuts mid-clause. Files: `src/segment.py` 2. Cap segment length A speaker who does not pause must still be translated. Files: `src/segment.py` 3. Measure the delay each setting adds The trade is the project. Measure it rather than guessing. Files: `src/latency.py` ### Done when - [ ] Segments are complete clauses in real speech - [ ] A non-pausing speaker is still segmented - [ ] Delay per setting is measured - [ ] Both languages segment acceptably ### Watch out - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. ## M2 · Translating complete thoughts Each segment translates correctly and the pipeline knows when it is unsure. ### Steps 1. Translate only stable segments A partial clause translates into confident nonsense. Files: `src/translate.py` 2. Signal low confidence to the listener Equal fluency for a guess and a certainty is the dangerous default. Files: `src/confidence.py` 3. Protect names, numbers and terms A mistranslated number in a conversation is a real problem. Files: `src/protect.py`, `tests/test_protect.py` ### Done when - [ ] No partial segment reaches the translator - [ ] Low confidence is signalled rather than spoken plainly - [ ] Names and numbers survive intact - [ ] A term list forces specific translations ### Watch out - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. ## M3 · Speaking without falling behind Translated speech plays promptly and the queue never grows unbounded. ### Steps 1. Synthesise into a bounded queue Unbounded means a minute behind during a fast exchange. Files: `src/synthesise.py` 2. Respond to queue growth by shortening Not by accumulating. Being far behind is worse than skipping. Files: `src/queue.py` 3. Measure end-to-end latency continuously It is the number the whole build is judged on. Files: `src/latency.py` ### Done when - [ ] Total latency is measured and stated - [ ] A fast exchange does not accumulate lag - [ ] Exceeding the queue bound has a stated behaviour - [ ] Playback does not destructively overlap speech ### Watch out - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. ## M4 · Two people, two directions Both sides are translated without the tool talking over itself. ### Steps 1. Run both directions concurrently Independently segmented. Languages pause differently. Files: `src/duplex.py` 2. Prevent the feedback loop Gate input during playback or identify your own output. Test it deliberately. Files: `src/feedback.py`, `tests/test_loop.py` 3. Handle simultaneous speech People talk over each other. It is not an error case. Files: `src/duplex.py` ### Done when - [ ] Both directions translate concurrently - [ ] Synthesised output is never re-translated - [ ] Overlapping speech is handled - [ ] Each side segments independently ### Watch out - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. ## M5 · Corrections, transcripts and consent (production only) Mistakes are correctable, the record exists if wanted, and the other party knows. ### Steps 1. Issue corrections audibly The listener was told something wrong. Fixing the log does not fix that. Files: `src/corrections.py` 2. Require disclosure to record A transcript is a recording of another person's speech. Files: `src/transcript.py` 3. Delete on a short retention A conversation is not an archive unless someone chose that. Files: `src/retention.py` ### Done when - [ ] A revised segment produces a visible correction - [ ] Recording requires a disclosure record - [ ] Conversations delete on a short retention - [ ] Latency and confidence are reviewable afterwards ### Watch out - Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript. ===== OPERATIONS.md ===== # Operations · ekto ## Backup Conversations delete on a short retention by default, so there is usually nothing to back up. Term lists and tuning parameters are small and worth keeping, since retuning against a real recording takes time. ## Restore Restore configuration and confirm retention resumes, since a restore that reinstates deleted conversations recreates recordings of other people's speech. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring End-to-end latency and its distribution is the health metric, along with the proportion of segments translated at low confidence. A rising low-confidence rate usually means the audio has degraded rather than the model. ## Incident checklist A feedback loop filling the channel is the most visible failure and is reproducible by playing output into the microphone. A translation that was wrong and never corrected is the most consequential, and the correction path is what prevents it. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] No partial segment is confirmed to reach the translator - [ ] Total latency is confirmed measured and within budget - [ ] A fast exchange is confirmed not to accumulate lag - [ ] Synthesised output is confirmed never re-translated - [ ] A revised segment is confirmed to produce a correction ## Launch constraint Do not market omitted ekto capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. One side of the conversation. Segmentation is tuned per language. LANGUAGE_A=en # Required. The other side. Pause patterns differ, so tuning is per side. LANGUAGE_B=pt # Required. A speaker who does not pause must still be translated. MAX_SEGMENT_MS=6000 # Required. The central trade. Shorter is faster and produces fragments. PAUSE_THRESHOLD_MS=500 # Required. Bounded. Growing lag is worse than dropping a segment. SYNTHESIS_QUEUE_MAX=2 # Required. Total speech to translated audio. Beyond this a conversation stops working. LATENCY_BUDGET_MS=2500
README.md
# ekto · indie build A speech translation pipeline where every design decision is a latency trade: recognition wants complete sentences, translation wants complete clauses, speech synthesis wants complete phrases, and a conversation wants none of them to take longer than a natural pause. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Recognition | Streaming-shaped recognition with voice activity segmentation | the segment boundary decides both latency and translation quality | | Translation | A model run per stable segment, never per partial word | translating a half-sentence produces confident nonsense | | Synthesis | Speech generated per translated segment with a small queue | the output must not fall behind the conversation | | Latency | A stated budget per stage, measured continuously | the pipeline is only usable if the total stays under a conversational pause | | Runtime | Local models where they fit, with the trade stated | conversations are private and the alternative is sending both sides to a service | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Orchestrates recognition, translation and synthesis. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **Speech recognition with word timings for both languages** · free locally, needs machine time - Why: Segmentation and latency measurement both depend on timings, and quality differs sharply between languages. - Get it: A local engine with models for both languages, checked for whether it emits word timings. - Verify: Both languages recognise with word timings - [ ] **A translation model for your language pair** · free locally; per-character if hosted - Why: Quality varies enormously by pair, and a pair that translates poorly makes every other decision in the build irrelevant. - Get it: A local or hosted model for your specific pair, evaluated on real conversational speech rather than written text. - Verify: Twenty real conversational sentences translate acceptably in both directions - [ ] **Speech synthesis for both languages** · free for open models - Why: Output speech is half the experience and synthesis latency is a large part of the total budget. - Get it: A local engine with voices for both languages, measured for time to first audio. - Verify: A sentence synthesises in both languages within your latency budget - [ ] **A recorded real bilingual conversation** · free, and requires asking people - Why: Every parameter here is tuned against real speech with interruptions, overlaps and people who do not pause, and none of that appears in read-aloud test sentences. - Get it: Record a genuine conversation with both speakers' consent, including a stretch where they talk over each other. - Verify: A recording exists with natural pauses, overlap and at least one long unbroken utterance ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Commercial translation quality across many language pairs. - Preserving the speaker's voice in translation. - Simultaneous interpretation quality. - Use where a mistranslation carries real consequence. - Long session reliability: memory growth, drifting segmentation and dropped turns after the first 20 minutes - Clean sentence by sentence pacing and turn detection, which is most of the perceived quality - A mobile app, so no translating anything while standing up - Offline or low-bandwidth behavior tuned for actual travel - Latency budgets someone else already fought for: streaming partial results instead of waiting for a full segment If one of those is essential to you, that is the reason to keep paying for ekto, and the README should say so rather than pretend.
BRIEF.md
# Build brief · ekto The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me near real-time speech translation for a two-way conversation to replace Ekto. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Recognition: Streaming-shaped recognition with voice activity segmentation · the segment boundary decides both latency and translation quality - Translation: A model run per stable segment, never per partial word · translating a half-sentence produces confident nonsense - Synthesis: Speech generated per translated segment with a small queue · the output must not fall behind the conversation - Latency: A stated budget per stage, measured continuously · the pipeline is only usable if the total stays under a conversational pause - Runtime: Local models where they fit, with the trade stated · conversations are private and the alternative is sending both sides to a service ### Data model (create this before Phase 1) - conversations(id, language_a, language_b, started_at, ended_at, disclosed_at) - segments(id, conversation_id, speaker_side, start_ms, end_ms, source_text, stability, superseded_by) - translations(segment_id, target_language, text, model, confidence, at) - utterances(translation_id, audio_path, played_at, duration_ms) - latencies(segment_id, stage, ms) · stage is recognition, translation, synthesis or total - corrections(segment_id, before, after, at) · when recognition revised after translation had already spoken - A segment records whether it was superseded, because a translation of a revised segment is a correction the listener already heard the wrong version of. ### Phase 1 · Deciding where a sentence ends Build: Segment incoming speech on voice activity and pause length, tuned so segments are complete clauses rather than fragments. Measure the delay each segmentation choice adds. Handle a speaker who does not pause. Trap: Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. Done when: Segments are complete clauses in normal speech, checked against a recording of real conversation. A speaker who does not pause is segmented on a maximum length rather than never. The added delay per segmentation setting is measured and recorded. Segmentation works for both languages, which may pause differently. Do not build yet: No translation, synthesis or playback. ### Phase 2 · Translating complete thoughts Build: Translate per stable segment with a confidence. Never translate a partial. Detect and flag low confidence rather than speaking it as though certain. Handle names, numbers and terms that must not be translated. Trap: Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. Done when: Only stable segments are translated, verified by checking no partial reaches the translator. Confidence is reported per translation and low confidence is signalled to the listener rather than spoken plainly. Names and numbers survive translation intact, tested with a list of both. A term list can force specific translations. Do not build yet: No synthesis or playback yet. ### Phase 3 · Speaking without falling behind Build: Synthesise per translated segment into a small queue. Detect the queue growing, which means the pipeline is slower than the conversation, and respond by shortening rather than by accumulating. Measure total latency continuously. Trap: A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. Done when: Total latency from speech to translated audio is measured and stated. A fast exchange does not accumulate an unbounded queue, tested with rapid alternating speech. Exceeding the queue bound produces a stated behaviour rather than growing lag. Playback does not overlap the original speaker's next utterance destructively. Do not build yet: No two-way handling or corrections yet. ### Phase 4 · Two people, two directions Build: Handle both directions concurrently, with separate segmentation per side. Avoid translating your own synthesised output back. Handle both people speaking at once, which is normal in conversation. Trap: Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. Done when: Both directions translate concurrently without either blocking the other. Synthesised output is never recognised and re-translated, tested by deliberately playing output into the microphone. Overlapping speech from both parties is handled rather than producing interleaved nonsense. Each side's segmentation is tuned independently. Do not build yet: No corrections or transcripts yet. ### Phase 5 · Corrections, transcripts and consent Build: Handle recognition revising a segment already translated and spoken, by issuing a correction rather than silently moving on. Offer a transcript, requiring disclosure to record. Delete conversations on a short retention by default. Trap: Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript. Done when: A revised segment produces an audible or visible correction rather than a silent transcript edit. Recording a transcript requires a disclosure record and is refused without one. Conversations delete on a short retention by default. The latency and confidence record is available afterwards for a conversation that went wrong. Do not build yet: Nothing after this. ### Out of scope (and why) - Commercial translation quality across many language pairs, which is where the hosted services have invested most and where the gap is widest for low-resource languages. - Preserving the speaker's voice in the translation, which raises consent questions about voice cloning. - Simultaneous interpretation quality, which is a professional skill involving anticipation that no pipeline of this shape reproduces. - Use in any setting where a mistranslation carries real consequence, such as medical or legal, which needs a professional interpreter. ### README must contain - That segment boundaries are the central trade between latency and translation quality - That partial segments are never translated because they produce confident nonsense - That low confidence is signalled rather than spoken with equal fluency - That the synthesis queue is bounded, because being a minute behind is worse than skipping - That a corrected segment produces a spoken or visible correction rather than a silent fix
AGENTS.md
# Agent instructions · ekto indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Streaming-shaped recognition with voice activity segmentation, A model run per stable segment, never per partial word, Speech generated per translated segment with a small queue, A stated budget per stage, measured continuously, Local models where they fit, with the trade stated. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation.
BUILD_PLAN.md
# Build plan · ekto A speech translation pipeline where every design decision is a latency trade: recognition wants complete sentences, translation wants complete clauses, speech synthesis wants complete phrases, and a conversation wants none of them to take longer than a natural pause. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Deciding where a sentence ends Speech is cut into segments that are complete enough to translate. ### Steps 1. Segment on voice activity and pause length Both. Activity alone cuts mid-clause. Files: `src/segment.py` 2. Cap segment length A speaker who does not pause must still be translated. Files: `src/segment.py` 3. Measure the delay each setting adds The trade is the project. Measure it rather than guessing. Files: `src/latency.py` ### Done when - [ ] Segments are complete clauses in real speech - [ ] A non-pausing speaker is still segmented - [ ] Delay per setting is measured - [ ] Both languages segment acceptably ### Watch out - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. ## Phase 2 · Translating complete thoughts Each segment translates correctly and the pipeline knows when it is unsure. ### Steps 1. Translate only stable segments A partial clause translates into confident nonsense. Files: `src/translate.py` 2. Signal low confidence to the listener Equal fluency for a guess and a certainty is the dangerous default. Files: `src/confidence.py` 3. Protect names, numbers and terms A mistranslated number in a conversation is a real problem. Files: `src/protect.py`, `tests/test_protect.py` ### Done when - [ ] No partial segment reaches the translator - [ ] Low confidence is signalled rather than spoken plainly - [ ] Names and numbers survive intact - [ ] A term list forces specific translations ### Watch out - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. ## Phase 3 · Speaking without falling behind Translated speech plays promptly and the queue never grows unbounded. ### Steps 1. Synthesise into a bounded queue Unbounded means a minute behind during a fast exchange. Files: `src/synthesise.py` 2. Respond to queue growth by shortening Not by accumulating. Being far behind is worse than skipping. Files: `src/queue.py` 3. Measure end-to-end latency continuously It is the number the whole build is judged on. Files: `src/latency.py` ### Done when - [ ] Total latency is measured and stated - [ ] A fast exchange does not accumulate lag - [ ] Exceeding the queue bound has a stated behaviour - [ ] Playback does not destructively overlap speech ### Watch out - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. ## Phase 4 · Two people, two directions Both sides are translated without the tool talking over itself. ### Steps 1. Run both directions concurrently Independently segmented. Languages pause differently. Files: `src/duplex.py` 2. Prevent the feedback loop Gate input during playback or identify your own output. Test it deliberately. Files: `src/feedback.py`, `tests/test_loop.py` 3. Handle simultaneous speech People talk over each other. It is not an error case. Files: `src/duplex.py` ### Done when - [ ] Both directions translate concurrently - [ ] Synthesised output is never re-translated - [ ] Overlapping speech is handled - [ ] Each side segments independently ### Watch out - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. ## Not in this build - Commercial translation quality across many language pairs. - Preserving the speaker's voice in translation. - Simultaneous interpretation quality. - Use where a mistranslation carries real consequence. ## After v1, if you want it - A shared screen view showing both transcripts, which helps far more than it sounds - Per-domain term lists, since the vocabulary that matters is usually specific - Latency tuning presets, so a slow conversation and a fast one use different settings
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. One side of the conversation. Segmentation is tuned per language. LANGUAGE_A=en # Required. The other side. Pause patterns differ, so tuning is per side. LANGUAGE_B=pt # Required. A speaker who does not pause must still be translated. MAX_SEGMENT_MS=6000 # Required. The central trade. Shorter is faster and produces fragments. PAUSE_THRESHOLD_MS=500 # Required. Bounded. Growing lag is worse than dropping a segment. SYNTHESIS_QUEUE_MAX=2 # Required. Total speech to translated audio. Beyond this a conversation stops working. LATENCY_BUDGET_MS=2500
PRODUCT.md
# ekto · product brief ## Problem The pipeline is no longer exotic: capture mic audio, segment it with a voice activity detector, transcribe with Whisper, translate, speak it back with a local TTS voice. An agent can wire that into a working local app in a weekend and it will genuinely translate a conversation. What it will not do out of the gate is stay graceful for an hour: chunk boundaries clip words, speaker turns bleed together, latency creeps as the buffer grows, and the sentence by sentence pacing that makes these apps usable in real conversation is a tuning problem, not a coding problem. You also get no phone app, which is where voice translation actually happens. Fine for a desk setup and travel prep, unconvincing when you are holding it out to a stranger in a market. ## Product outcome A translation pipeline that stays inside a conversational pause, never speaks a guess with the confidence of a certainty, and does not translate its own output back into a loop. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Python 3.11 and a machine with at least 8GB RAM, GPU strongly preferred - Local model downloads: faster-whisper and a Piper voice per target language - A browser with mic permission, or an API key if you swap in a hosted translation model - Headphones, otherwise the TTS output feeds back into the mic ## Explicit non-goals for v1 - Commercial translation quality across many language pairs. - Preserving the speaker's voice in translation. - Simultaneous interpretation quality. - Use where a mistranslation carries real consequence. - Long session reliability: memory growth, drifting segmentation and dropped turns after the first 20 minutes - Clean sentence by sentence pacing and turn detection, which is most of the perceived quality - A mobile app, so no translating anything while standing up - Offline or low-bandwidth behavior tuned for actual travel - Latency budgets someone else already fought for: streaming partial results instead of waiting for a full segment ## Success criteria - No partial segment is confirmed to reach the translator - Total latency is confirmed measured and within budget - A fast exchange is confirmed not to accumulate lag - Synthesised output is confirmed never re-translated - A revised segment is confirmed to produce a correction
BRIEF.md
# Build brief · ekto The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me near real-time speech translation for a two-way conversation to replace Ekto. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Recognition: Streaming-shaped recognition with voice activity segmentation · the segment boundary decides both latency and translation quality - Translation: A model run per stable segment, never per partial word · translating a half-sentence produces confident nonsense - Synthesis: Speech generated per translated segment with a small queue · the output must not fall behind the conversation - Latency: A stated budget per stage, measured continuously · the pipeline is only usable if the total stays under a conversational pause - Runtime: Local models where they fit, with the trade stated · conversations are private and the alternative is sending both sides to a service ### Data model (create this before Phase 1) - conversations(id, language_a, language_b, started_at, ended_at, disclosed_at) - segments(id, conversation_id, speaker_side, start_ms, end_ms, source_text, stability, superseded_by) - translations(segment_id, target_language, text, model, confidence, at) - utterances(translation_id, audio_path, played_at, duration_ms) - latencies(segment_id, stage, ms) · stage is recognition, translation, synthesis or total - corrections(segment_id, before, after, at) · when recognition revised after translation had already spoken - A segment records whether it was superseded, because a translation of a revised segment is a correction the listener already heard the wrong version of. ### Phase 1 · Deciding where a sentence ends Build: Segment incoming speech on voice activity and pause length, tuned so segments are complete clauses rather than fragments. Measure the delay each segmentation choice adds. Handle a speaker who does not pause. Trap: Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. Done when: Segments are complete clauses in normal speech, checked against a recording of real conversation. A speaker who does not pause is segmented on a maximum length rather than never. The added delay per segmentation setting is measured and recorded. Segmentation works for both languages, which may pause differently. Do not build yet: No translation, synthesis or playback. ### Phase 2 · Translating complete thoughts Build: Translate per stable segment with a confidence. Never translate a partial. Detect and flag low confidence rather than speaking it as though certain. Handle names, numbers and terms that must not be translated. Trap: Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. Done when: Only stable segments are translated, verified by checking no partial reaches the translator. Confidence is reported per translation and low confidence is signalled to the listener rather than spoken plainly. Names and numbers survive translation intact, tested with a list of both. A term list can force specific translations. Do not build yet: No synthesis or playback yet. ### Phase 3 · Speaking without falling behind Build: Synthesise per translated segment into a small queue. Detect the queue growing, which means the pipeline is slower than the conversation, and respond by shortening rather than by accumulating. Measure total latency continuously. Trap: A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. Done when: Total latency from speech to translated audio is measured and stated. A fast exchange does not accumulate an unbounded queue, tested with rapid alternating speech. Exceeding the queue bound produces a stated behaviour rather than growing lag. Playback does not overlap the original speaker's next utterance destructively. Do not build yet: No two-way handling or corrections yet. ### Phase 4 · Two people, two directions Build: Handle both directions concurrently, with separate segmentation per side. Avoid translating your own synthesised output back. Handle both people speaking at once, which is normal in conversation. Trap: Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. Done when: Both directions translate concurrently without either blocking the other. Synthesised output is never recognised and re-translated, tested by deliberately playing output into the microphone. Overlapping speech from both parties is handled rather than producing interleaved nonsense. Each side's segmentation is tuned independently. Do not build yet: No corrections or transcripts yet. ### Phase 5 · Corrections, transcripts and consent Build: Handle recognition revising a segment already translated and spoken, by issuing a correction rather than silently moving on. Offer a transcript, requiring disclosure to record. Delete conversations on a short retention by default. Trap: Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript. Done when: A revised segment produces an audible or visible correction rather than a silent transcript edit. Recording a transcript requires a disclosure record and is refused without one. Conversations delete on a short retention by default. The latency and confidence record is available afterwards for a conversation that went wrong. Do not build yet: Nothing after this. ### Out of scope (and why) - Commercial translation quality across many language pairs, which is where the hosted services have invested most and where the gap is widest for low-resource languages. - Preserving the speaker's voice in the translation, which raises consent questions about voice cloning. - Simultaneous interpretation quality, which is a professional skill involving anticipation that no pipeline of this shape reproduces. - Use in any setting where a mistranslation carries real consequence, such as medical or legal, which needs a professional interpreter. ### README must contain - That segment boundaries are the central trade between latency and translation quality - That partial segments are never translated because they produce confident nonsense - That low confidence is signalled rather than spoken with equal fluency - That the synthesis queue is bounded, because being a minute behind is worse than skipping - That a corrected segment produces a spoken or visible correction rather than a silent fix
ARCHITECTURE.md
# Architecture · ekto ## Stack | Part | Choice | Why | | --- | --- | --- | | Recognition | Streaming-shaped recognition with voice activity segmentation | the segment boundary decides both latency and translation quality | | Translation | A model run per stable segment, never per partial word | translating a half-sentence produces confident nonsense | | Synthesis | Speech generated per translated segment with a small queue | the output must not fall behind the conversation | | Latency | A stated budget per stage, measured continuously | the pipeline is only usable if the total stays under a conversational pause | | Runtime | Local models where they fit, with the trade stated | conversations are private and the alternative is sending both sides to a service | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Segmentation | Cutting speech into translatable units. | Any approach, provided the latency it adds is measured rather than assumed. | | Translation | Stable segments and confidence. | Any model, provided partials are never sent and uncertainty is signalled. | | Synthesis | Bounded output queue. | Any engine, provided the queue cannot grow unbounded. | | Duplex | Both directions and feedback prevention. | Any implementation, provided the loop is tested deliberately. | | Corrections | Handling revised segments audibly. | Any presentation, provided a silent transcript fix is not the whole response. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `LANGUAGE_A` · required · One side of the conversation. Segmentation is tuned per language. - `LANGUAGE_B` · required · The other side. Pause patterns differ, so tuning is per side. - `MAX_SEGMENT_MS` · required · A speaker who does not pause must still be translated. - `PAUSE_THRESHOLD_MS` · required · The central trade. Shorter is faster and produces fragments. - `SYNTHESIS_QUEUE_MAX` · required · Bounded. Growing lag is worse than dropping a segment. - `LATENCY_BUDGET_MS` · required · Total speech to translated audio. Beyond this a conversation stops working. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · ekto product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Streaming-shaped recognition with voice activity segmentation, A model run per stable segment, never per partial word, Speech generated per translated segment with a small queue, A stated budget per stage, measured continuously, Local models where they fit, with the trade stated. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. - Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript.
MILESTONES.md
# Delivery milestones · ekto Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Deciding where a sentence ends Speech is cut into segments that are complete enough to translate. ### Steps 1. Segment on voice activity and pause length Both. Activity alone cuts mid-clause. Files: `src/segment.py` 2. Cap segment length A speaker who does not pause must still be translated. Files: `src/segment.py` 3. Measure the delay each setting adds The trade is the project. Measure it rather than guessing. Files: `src/latency.py` ### Done when - [ ] Segments are complete clauses in real speech - [ ] A non-pausing speaker is still segmented - [ ] Delay per setting is measured - [ ] Both languages segment acceptably ### Watch out - Segmenting on short pauses gives low latency and fragments that translate into nonsense, because most languages need a complete clause before the grammar is determined. Segmenting on long pauses translates well and lags so far behind that the conversation becomes unusable. This trade is the whole project, so measure both sides of it rather than picking a number. ## M2 · Translating complete thoughts Each segment translates correctly and the pipeline knows when it is unsure. ### Steps 1. Translate only stable segments A partial clause translates into confident nonsense. Files: `src/translate.py` 2. Signal low confidence to the listener Equal fluency for a guess and a certainty is the dangerous default. Files: `src/confidence.py` 3. Protect names, numbers and terms A mistranslated number in a conversation is a real problem. Files: `src/protect.py`, `tests/test_protect.py` ### Done when - [ ] No partial segment reaches the translator - [ ] Low confidence is signalled rather than spoken plainly - [ ] Names and numbers survive intact - [ ] A term list forces specific translations ### Watch out - Speaking a low-confidence translation with the same fluency as a high-confidence one means the listener has no way to tell a good translation from a guess, and in a conversation that matters the guess is the one that causes harm. Signal uncertainty, either by saying so or by asking the speaker to repeat. ## M3 · Speaking without falling behind Translated speech plays promptly and the queue never grows unbounded. ### Steps 1. Synthesise into a bounded queue Unbounded means a minute behind during a fast exchange. Files: `src/synthesise.py` 2. Respond to queue growth by shortening Not by accumulating. Being far behind is worse than skipping. Files: `src/queue.py` 3. Measure end-to-end latency continuously It is the number the whole build is judged on. Files: `src/latency.py` ### Done when - [ ] Total latency is measured and stated - [ ] A fast exchange does not accumulate lag - [ ] Exceeding the queue bound has a stated behaviour - [ ] Playback does not destructively overlap speech ### Watch out - A synthesis queue with no bound grows during a fast exchange until the translation is a minute behind, at which point it is worse than useless because the listener is hearing an answer to a question two turns ago. Cap the queue, and when it is exceeded, drop or summarise rather than lengthening the lag. ## M4 · Two people, two directions Both sides are translated without the tool talking over itself. ### Steps 1. Run both directions concurrently Independently segmented. Languages pause differently. Files: `src/duplex.py` 2. Prevent the feedback loop Gate input during playback or identify your own output. Test it deliberately. Files: `src/feedback.py`, `tests/test_loop.py` 3. Handle simultaneous speech People talk over each other. It is not an error case. Files: `src/duplex.py` ### Done when - [ ] Both directions translate concurrently - [ ] Synthesised output is never re-translated - [ ] Overlapping speech is handled - [ ] Each side segments independently ### Watch out - Synthesised translation picked up by the microphone gets recognised, translated back and spoken again, producing a loop that fills the channel within seconds. Gate the input during playback, or identify your own output, and test it deliberately rather than discovering it in a real conversation. ## M5 · Corrections, transcripts and consent (production only) Mistakes are correctable, the record exists if wanted, and the other party knows. ### Steps 1. Issue corrections audibly The listener was told something wrong. Fixing the log does not fix that. Files: `src/corrections.py` 2. Require disclosure to record A transcript is a recording of another person's speech. Files: `src/transcript.py` 3. Delete on a short retention A conversation is not an archive unless someone chose that. Files: `src/retention.py` ### Done when - [ ] A revised segment produces a visible correction - [ ] Recording requires a disclosure record - [ ] Conversations delete on a short retention - [ ] Latency and confidence are reviewable afterwards ### Watch out - Silently correcting a segment whose wrong translation has already been spoken leaves the listener holding a false statement they were never told was wrong, and in a real conversation that is the failure with consequences. Speak the correction, or at minimum show it, rather than only fixing the transcript.
OPERATIONS.md
# Operations · ekto ## Backup Conversations delete on a short retention by default, so there is usually nothing to back up. Term lists and tuning parameters are small and worth keeping, since retuning against a real recording takes time. ## Restore Restore configuration and confirm retention resumes, since a restore that reinstates deleted conversations recreates recordings of other people's speech. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring End-to-end latency and its distribution is the health metric, along with the proportion of segments translated at low confidence. A rising low-confidence rate usually means the audio has degraded rather than the model. ## Incident checklist A feedback loop filling the channel is the most visible failure and is reproducible by playing output into the microphone. A translation that was wrong and never corrected is the most consequential, and the correction path is what prevents it. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] No partial segment is confirmed to reach the translator - [ ] Total latency is confirmed measured and within budget - [ ] A fast exchange is confirmed not to accumulate lag - [ ] Synthesised output is confirmed never re-translated - [ ] A revised segment is confirmed to produce a correction ## Launch constraint Do not market omitted ekto capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. One side of the conversation. Segmentation is tuned per language. LANGUAGE_A=en # Required. The other side. Pause patterns differ, so tuning is per side. LANGUAGE_B=pt # Required. A speaker who does not pause must still be translated. MAX_SEGMENT_MS=6000 # Required. The central trade. Shorter is faster and produces fragments. PAUSE_THRESHOLD_MS=500 # Required. Bounded. Growing lag is worse than dropping a segment. SYNTHESIS_QUEUE_MAX=2 # Required. Total speech to translated audio. Beyond this a conversation stops working. LATENCY_BUDGET_MS=2500
$ choose a build depth, inspect the files, then open the complete pack in your agent
Because voice translation is judged entirely on the seconds between someone finishing a sentence and you hearing it, and on whether it still works on minute 40. A local build nails the demo and then frays: barge-in, background noise, two people talking over each other, the phone locking. Paying gets you a phone in your pocket that handles those cases without you adding VAD thresholds mid-conversation.
xLong session reliability: memory growth, drifting segmentation and dropped turns after the first 20 minutes
xClean sentence by sentence pacing and turn detection, which is most of the perceived quality
xA mobile app, so no translating anything while standing up
xOffline or low-bandwidth behavior tuned for actual travel
xLatency budgets someone else already fought for: streaming partial results instead of waiting for a full segment
Nothing worth pointing at. That's why the prompt exists.
share on X ↗"I just replaced ekto ($29.99/mo) with one prompt"
Vibecode ekto
Kinda. The core of ekto is buildable in a weekend with the prompt on this page, but there are real gaps: Long session reliability: memory growth, drifting segmentation and dropped turns after the first 20 minutes, Clean sentence by sentence pacing and turn detection, which is most of the perceived quality. Read the honest list above before committing.
How much does ekto cost?
ekto costs about $29.99/month (Monthly Unlimited PRO, checked 2026-08-18), which is $359.88 per year.
What do I lose by replacing ekto?
Honestly: Long session reliability: memory growth, drifting segmentation and dropped turns after the first 20 minutes; Clean sentence by sentence pacing and turn detection, which is most of the perceived quality; A mobile app, so no translating anything while standing up; Offline or low-bandwidth behavior tuned for actual travel; Latency budgets someone else already fought for: streaming partial results instead of waiting for a full segment. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to ekto?
No mature open-source alternative worth pointing at, which is exactly why the one-shot prompt on this page exists.