Vibecode Screen Studio
track this build5 phases, 15 steps, beginner friendly0%The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For Screen Studio, record a Mac screen and automatically apply cursor zoom, padding, and smooth framing. The hard boundary is exceptional macos capture polish, motion design, and ongoing compatibility, plus capture polish, hosting, and collaboration.
You are building a lean indie version of Screen Studio. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Screen Studio · indie build A recorder that captures the screen flat while writing a separate high-frequency cursor and click track, then generates smooth zoom keyframes from that track in post, applies padding, rounded corners and a background, and renders a finished video you never had to edit by hand. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Language | Swift 6 and SwiftUI in Xcode | ScreenCaptureKit and the video pipeline are Apple frameworks | | Capture | ScreenCaptureKit for video, plus a separate cursor event track | the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards | | Composition | Core Image and Metal, rendered through AVAssetWriter | zoom, padding and background are per-frame transforms, which is what the GPU is for | | Interpolation | Cubic easing over generated keyframes | linear zoom looks mechanical, and the easing is the entire perceived quality of the effect | | Storage | The flat recording plus a sidecar JSON project file | keeping the source untouched means every stylistic decision stays reversible | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **A Mac running macOS 13 or newer with a discrete or Apple silicon GPU** · free if you already own a Mac - Why: ScreenCaptureKit needs 12.3 as a floor, and the composite pipeline is GPU work. An old integrated GPU will export slower than real time. - Get it: Apple menu, then About This Mac. - Verify: About This Mac reports macOS 13 or higher - [ ] **Xcode 16 or newer** · free - Why: Builds the app and provides Instruments, which you will need when the export is slow. - Get it: Mac App Store or developer.apple.com/xcode. - Verify: xcodebuild -version prints 16 or higher - [ ] **Screen Recording and Microphone permissions** · free - Why: Screen Recording is required to capture at all. Microphone is optional and only for narration. - Get it: System Settings, then Privacy and Security. Both must be re-granted during development whenever the binary changes signature or path. - Verify: A test capture returns pixels rather than a black frame - [ ] **Three real recordings to tune against** · free - Why: The thresholds in Phase 2 cannot be chosen in the abstract. You need a form being filled, a terminal session and a slow read to see where the generator over-triggers. - Get it: Record them yourself before writing the detection code, and keep them as fixtures. - Verify: Three recordings with their cursor tracks are saved as test fixtures - [ ] **Apple Developer Program membership** (optional) · 99 USD per year - Why: Product phase only. Without it there is no notarization and a Gatekeeper warning on every other Mac. - Get it: Enrol at developer.apple.com/programs. - Verify: A Developer ID Application certificate appears in Xcode settings ## Quick start ```sh xcrun notarytool submit Studio.zip --keychain-profile AC --wait xcrun stapler staple Studio.app ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - The tuned motion design judgement of the commercial product. - Hosted sharing, links and viewer analytics. - Camera overlay with background removal. - Multi-track video editing. - exceptional macOS capture polish, motion design, and ongoing compatibility - instant hosted sharing - viewer analytics - team libraries - cross-device capture If one of those is essential to you, that is the reason to keep paying for Screen Studio, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Screen Studio The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a screen recorder that automatically zooms to the cursor and renders a polished video without manual editing to replace Screen Studio. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Language: Swift 6 and SwiftUI in Xcode · ScreenCaptureKit and the video pipeline are Apple frameworks - Capture: ScreenCaptureKit for video, plus a separate cursor event track · the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards - Composition: Core Image and Metal, rendered through AVAssetWriter · zoom, padding and background are per-frame transforms, which is what the GPU is for - Interpolation: Cubic easing over generated keyframes · linear zoom looks mechanical, and the easing is the entire perceived quality of the effect - Storage: The flat recording plus a sidecar JSON project file · keeping the source untouched means every stylistic decision stays reversible ### Data model (create this before Phase 1) - recordings(id, video_path, width, height, scale, fps, started_at, duration_ms) - cursor_samples(recording_id, t_ms, x, y) · sampled at a high fixed rate, independent of video frame rate - cursor_events(recording_id, t_ms, kind) · kind is click, drag_start, drag_end, keypress or scroll - keyframes(id, recording_id, t_ms, scale, center_x, center_y, easing) · generated in Phase 2, editable afterwards - projects(id, recording_id, padding, corner_radius, background_json, trim_start_ms, trim_end_ms) - The flat recording is never modified. Every effect is a function of the source plus the project row, which is what makes re-rendering at a different size free. ### Phase 1 · Record flat, and capture the cursor track Build: Record the screen with ScreenCaptureKit at native resolution with no effects applied. In parallel, sample the cursor position at a high fixed rate and log clicks and drags with timestamps on the same clock as the video. Trap: Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. Trap: Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. Done when: A recording produces a flat video plus a cursor track whose sample count is far higher than the frame count for the same duration. Clicks in the track line up with visible clicks in the video within a frame. The video plays in QuickTime with no repair prompt. Do not build yet: No zoom, no styling, no export options. A clean source and an accurate track. ### Phase 2 · Generating zoom from the track Build: Detect regions of interest: clusters of clicks, sustained activity in one area, and typing. Emit a zoom-in keyframe shortly before the activity and a zoom-out after it. Suppress zooms for brief cursor transits, and enforce a minimum interval so the video does not pulse. Trap: Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. Done when: A recording of a form being filled produces zooms on the fields that were clicked and none on the cursor moving between them. No two zoom-ins occur closer together than the configured minimum. A recording with no clicks at all produces no keyframes rather than a default zoom. Do not build yet: No easing or styling yet, so the result will look mechanical. That is expected. ### Phase 3 · Easing and framing Build: Interpolate between keyframes with a cubic ease, not linearly. Clamp the zoom centre so the visible rectangle always stays inside the source bounds. Add padding, rounded corners and a background so the recording sits inside a frame. Trap: Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. Done when: Zooming at the far corner of the screen never shows an area outside the source. Motion between keyframes accelerates and decelerates rather than moving at constant speed. Changing the padding and corner radius re-renders without re-recording. Do not build yet: No trimming or export presets yet. ### Phase 4 · Rendering and export Build: Render the composed frames through AVAssetWriter with hardware encoding. Offer export presets with exact pixel dimensions for common destinations. Show real progress, and keep audio in sync with the composed video. Trap: Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. Done when: A two minute recording exports in less than its own duration on the machine you built it on. An export preset produces exactly the pixel dimensions it names. Audio remains in sync at the end of the export, checked against the source recording. Do not build yet: No sharing or distribution yet. ### Phase 5 · Trim and manual correction Build: A timeline showing the video, the click track and the generated keyframes. Trim the start and end. Move, delete or add a keyframe. Every change re-renders the preview from the untouched source. Trap: Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. Done when: Deleting a generated keyframe removes that zoom from the preview and from the export. Adding one by hand produces a zoom at that moment. Trimming the start does not shift the keyframe timing relative to the visible content. Re-running generation after manual edits warns before discarding them. Do not build yet: Distribution is the last phase. ### Phase 6 · Signing, notarization and updates Build: Hardened runtime, Developer ID signing, notarization and stapling, plus Sparkle for updates. Document the permission story, which here is Screen Recording and microphone. Trap: Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps. Done when: A notarized build opens on a fresh Mac with no Gatekeeper warning and requests its permissions once. An appcast update installs and relaunches. Do not build yet: Nothing after this. ### Out of scope (and why) - The motion design judgement behind the commercial product. The thresholds in Phase 2 are the whole craft, and theirs are tuned against far more footage than you will record. - Hosted sharing and viewer analytics, which need a server, storage and links that keep working. - Camera overlay with background removal, which is a separate segmentation problem. - Multi-track editing. This is a recorder with automatic motion, not a video editor, and the moment it becomes one the automation stops being the point. ### README must contain - That the recording is captured flat and every effect is applied at export, so nothing is baked in - That the cursor track is sampled far faster than the frame rate, and why - The three numbers that control the feel: minimum interval, minimum dwell and maximum zoom rate - Which permissions are needed and what refusing each one costs - That export runs on the GPU, and what to check if it becomes slower than real time ===== AGENTS.md ===== # Agent instructions · Screen Studio indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Swift 6 and SwiftUI in Xcode, ScreenCaptureKit for video, plus a separate cursor event track, Core Image and Metal, rendered through AVAssetWriter, Cubic easing over generated keyframes, The flat recording plus a sidecar JSON project file. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. ===== BUILD_PLAN.md ===== # Build plan · Screen Studio A recorder that captures the screen flat while writing a separate high-frequency cursor and click track, then generates smooth zoom keyframes from that track in post, applies padding, rounded corners and a background, and renders a finished video you never had to edit by hand. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Record flat, and capture the cursor track A plain recording exists alongside a dense, accurate record of where the cursor was and what it did. ### Steps 1. Capture the screen at native resolution SCStream configured for the display's real pixel dimensions. Write with AVAssetWriter, starting the session from the first video sample's timestamp. Files: `Recorder.swift` 2. Sample the cursor on its own timer A high-rate timer reading the global cursor position, timestamped against the same clock as the video session. Log clicks with an event tap or a global monitor. Files: `CursorTrack.swift` 3. Verify alignment before trusting it Record a deliberate click at a known screen position, then confirm the logged event timestamp matches the frame where the click is visible. Files: `test/alignment.md` ### Done when - [ ] The cursor sample count far exceeds the video frame count for the same duration - [ ] A logged click aligns with the visible click within one frame - [ ] The flat video opens in QuickTime without a repair prompt - [ ] The recording is at native pixel resolution, not scaled down ### Watch out - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. ## Phase 2 · Generating zoom from the track The zoom keyframes are computed from what the cursor did, and they are good enough that you rarely touch them. ### Steps 1. Detect intent regions Cluster click events in time and space. Treat sustained dwell within a small radius as intent too, because reading and typing matter as much as clicking. Files: `Intent.swift` 2. Emit keyframes with lead and lag Zoom in slightly before the action so the viewer arrives with it, and hold briefly after so the result is visible before pulling out. Files: `Keyframes.swift` 3. Suppress noise A minimum interval between zooms, a minimum dwell to qualify, and a maximum zoom rate. These three numbers are the whole feel of the product. Files: `Keyframes.swift` ### Done when - [ ] Clicked fields get zooms and cursor transits between them do not - [ ] No two zoom-ins are closer than the configured minimum interval - [ ] A recording with no clicks produces no keyframes - [ ] Keyframes are stored separately from the video and can be regenerated ### Watch out - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. ## Phase 3 · Easing and framing The motion looks designed rather than computed, and the frame never shows the edge of the source. ### Steps 1. Implement cubic easing Ease in and out between keyframe pairs. Compare against linear side by side once; you will not go back. Files: `Easing.swift` 2. Clamp the centre inside the source Compute the visible rectangle at the current scale and clamp its centre so the rectangle stays inside the source. Test the corners explicitly. Files: `Framing.swift` 3. Composite padding, corners and background A Core Image pipeline: scale and translate the source, round the corners, and draw over a background colour or gradient. Files: `Composite.swift` ### Done when - [ ] A zoom at a screen corner never reveals area outside the source - [ ] Motion eases rather than moving at constant speed - [ ] Padding and corner radius change on re-render without re-recording - [ ] The background renders behind the rounded corners with no fringing ### Watch out - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. ## Phase 4 · Rendering and export The finished video comes out at the size you asked for, and rendering is not slower than watching. ### Steps 1. Keep the composite on the GPU One Metal-backed CIContext reused across frames. Creating a context per frame is the single most common reason exports crawl. Files: `Renderer.swift` 2. Add export presets with exact dimensions Named presets with fixed pixel sizes. Assert the output dimensions rather than trusting the scale arithmetic. Files: `Presets.swift` 3. Carry audio through unchanged Pass the original audio track through rather than re-encoding it, and confirm sync at the end of a long export. Files: `Renderer.swift` ### Done when - [ ] A two minute recording exports in under two minutes - [ ] Each preset produces exactly the pixel dimensions it names - [ ] Audio is in sync at the end of the exported file - [ ] Cancelling an export leaves no partial file behind ### Watch out - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. ## Phase 5 · Trim and manual correction The one zoom the generator got wrong takes ten seconds to fix. ### Steps 1. Build the timeline Video thumbnails, click markers and keyframe handles on one scrubbable track. Files: `Timeline.swift` 2. Make keyframes editable Drag to move, delete, insert. Preview updates from the source, which is why nothing was baked in. Files: `Timeline.swift` 3. Protect manual edits Mark keyframes as generated or manual, and warn before regeneration discards manual ones. Files: `Keyframes.swift` ### Done when - [ ] Deleting a keyframe removes the zoom from preview and export - [ ] A manually added keyframe produces a zoom at that time - [ ] Trimming does not desynchronise keyframes from content - [ ] Regeneration warns before discarding manual edits ### Watch out - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. ## Not in this build - The tuned motion design judgement of the commercial product. - Hosted sharing, links and viewer analytics. - Camera overlay with background removal. - Multi-track video editing. ## After v1, if you want it - Keystroke overlays generated from the event track you are already recording - Automatic silence trimming using the audio track - A vertical export preset that reframes around the cursor rather than letterboxing ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where flat recordings and project files are written. These are large, so pick a disk with room. RECORDINGS_DIR=~/Movies/ScreenStudio # Required. Cursor sampling rate. Must be well above the video frame rate or the motion looks stepped. CURSOR_SAMPLE_HZ=120 # Required. Minimum time between zoom-ins. Raising this calms a busy video. ZOOM_MIN_INTERVAL_MS=2500 # Required. How long the cursor must stay in an area to count as intent. Lowering it makes the video restless. ZOOM_MIN_DWELL_MS=600 # Required. How far in the automatic zoom will go. Beyond about two and a half the source pixels start to show. ZOOM_MAX_SCALE=2.2 # Optional. Product phase only. SPARKLE_FEED_URL=https://example.com/appcast.xml
You are building a lean indie version of Screen Studio. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Screen Studio · indie build A recorder that captures the screen flat while writing a separate high-frequency cursor and click track, then generates smooth zoom keyframes from that track in post, applies padding, rounded corners and a background, and renders a finished video you never had to edit by hand. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Language | Swift 6 and SwiftUI in Xcode | ScreenCaptureKit and the video pipeline are Apple frameworks | | Capture | ScreenCaptureKit for video, plus a separate cursor event track | the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards | | Composition | Core Image and Metal, rendered through AVAssetWriter | zoom, padding and background are per-frame transforms, which is what the GPU is for | | Interpolation | Cubic easing over generated keyframes | linear zoom looks mechanical, and the easing is the entire perceived quality of the effect | | Storage | The flat recording plus a sidecar JSON project file | keeping the source untouched means every stylistic decision stays reversible | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **A Mac running macOS 13 or newer with a discrete or Apple silicon GPU** · free if you already own a Mac - Why: ScreenCaptureKit needs 12.3 as a floor, and the composite pipeline is GPU work. An old integrated GPU will export slower than real time. - Get it: Apple menu, then About This Mac. - Verify: About This Mac reports macOS 13 or higher - [ ] **Xcode 16 or newer** · free - Why: Builds the app and provides Instruments, which you will need when the export is slow. - Get it: Mac App Store or developer.apple.com/xcode. - Verify: xcodebuild -version prints 16 or higher - [ ] **Screen Recording and Microphone permissions** · free - Why: Screen Recording is required to capture at all. Microphone is optional and only for narration. - Get it: System Settings, then Privacy and Security. Both must be re-granted during development whenever the binary changes signature or path. - Verify: A test capture returns pixels rather than a black frame - [ ] **Three real recordings to tune against** · free - Why: The thresholds in Phase 2 cannot be chosen in the abstract. You need a form being filled, a terminal session and a slow read to see where the generator over-triggers. - Get it: Record them yourself before writing the detection code, and keep them as fixtures. - Verify: Three recordings with their cursor tracks are saved as test fixtures - [ ] **Apple Developer Program membership** (optional) · 99 USD per year - Why: Product phase only. Without it there is no notarization and a Gatekeeper warning on every other Mac. - Get it: Enrol at developer.apple.com/programs. - Verify: A Developer ID Application certificate appears in Xcode settings ## Quick start ```sh xcrun notarytool submit Studio.zip --keychain-profile AC --wait xcrun stapler staple Studio.app ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - The tuned motion design judgement of the commercial product. - Hosted sharing, links and viewer analytics. - Camera overlay with background removal. - Multi-track video editing. - exceptional macOS capture polish, motion design, and ongoing compatibility - instant hosted sharing - viewer analytics - team libraries - cross-device capture If one of those is essential to you, that is the reason to keep paying for Screen Studio, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Screen Studio The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a screen recorder that automatically zooms to the cursor and renders a polished video without manual editing to replace Screen Studio. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Language: Swift 6 and SwiftUI in Xcode · ScreenCaptureKit and the video pipeline are Apple frameworks - Capture: ScreenCaptureKit for video, plus a separate cursor event track · the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards - Composition: Core Image and Metal, rendered through AVAssetWriter · zoom, padding and background are per-frame transforms, which is what the GPU is for - Interpolation: Cubic easing over generated keyframes · linear zoom looks mechanical, and the easing is the entire perceived quality of the effect - Storage: The flat recording plus a sidecar JSON project file · keeping the source untouched means every stylistic decision stays reversible ### Data model (create this before Phase 1) - recordings(id, video_path, width, height, scale, fps, started_at, duration_ms) - cursor_samples(recording_id, t_ms, x, y) · sampled at a high fixed rate, independent of video frame rate - cursor_events(recording_id, t_ms, kind) · kind is click, drag_start, drag_end, keypress or scroll - keyframes(id, recording_id, t_ms, scale, center_x, center_y, easing) · generated in Phase 2, editable afterwards - projects(id, recording_id, padding, corner_radius, background_json, trim_start_ms, trim_end_ms) - The flat recording is never modified. Every effect is a function of the source plus the project row, which is what makes re-rendering at a different size free. ### Phase 1 · Record flat, and capture the cursor track Build: Record the screen with ScreenCaptureKit at native resolution with no effects applied. In parallel, sample the cursor position at a high fixed rate and log clicks and drags with timestamps on the same clock as the video. Trap: Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. Trap: Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. Done when: A recording produces a flat video plus a cursor track whose sample count is far higher than the frame count for the same duration. Clicks in the track line up with visible clicks in the video within a frame. The video plays in QuickTime with no repair prompt. Do not build yet: No zoom, no styling, no export options. A clean source and an accurate track. ### Phase 2 · Generating zoom from the track Build: Detect regions of interest: clusters of clicks, sustained activity in one area, and typing. Emit a zoom-in keyframe shortly before the activity and a zoom-out after it. Suppress zooms for brief cursor transits, and enforce a minimum interval so the video does not pulse. Trap: Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. Done when: A recording of a form being filled produces zooms on the fields that were clicked and none on the cursor moving between them. No two zoom-ins occur closer together than the configured minimum. A recording with no clicks at all produces no keyframes rather than a default zoom. Do not build yet: No easing or styling yet, so the result will look mechanical. That is expected. ### Phase 3 · Easing and framing Build: Interpolate between keyframes with a cubic ease, not linearly. Clamp the zoom centre so the visible rectangle always stays inside the source bounds. Add padding, rounded corners and a background so the recording sits inside a frame. Trap: Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. Done when: Zooming at the far corner of the screen never shows an area outside the source. Motion between keyframes accelerates and decelerates rather than moving at constant speed. Changing the padding and corner radius re-renders without re-recording. Do not build yet: No trimming or export presets yet. ### Phase 4 · Rendering and export Build: Render the composed frames through AVAssetWriter with hardware encoding. Offer export presets with exact pixel dimensions for common destinations. Show real progress, and keep audio in sync with the composed video. Trap: Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. Done when: A two minute recording exports in less than its own duration on the machine you built it on. An export preset produces exactly the pixel dimensions it names. Audio remains in sync at the end of the export, checked against the source recording. Do not build yet: No sharing or distribution yet. ### Phase 5 · Trim and manual correction Build: A timeline showing the video, the click track and the generated keyframes. Trim the start and end. Move, delete or add a keyframe. Every change re-renders the preview from the untouched source. Trap: Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. Done when: Deleting a generated keyframe removes that zoom from the preview and from the export. Adding one by hand produces a zoom at that moment. Trimming the start does not shift the keyframe timing relative to the visible content. Re-running generation after manual edits warns before discarding them. Do not build yet: Distribution is the last phase. ### Phase 6 · Signing, notarization and updates Build: Hardened runtime, Developer ID signing, notarization and stapling, plus Sparkle for updates. Document the permission story, which here is Screen Recording and microphone. Trap: Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps. Done when: A notarized build opens on a fresh Mac with no Gatekeeper warning and requests its permissions once. An appcast update installs and relaunches. Do not build yet: Nothing after this. ### Out of scope (and why) - The motion design judgement behind the commercial product. The thresholds in Phase 2 are the whole craft, and theirs are tuned against far more footage than you will record. - Hosted sharing and viewer analytics, which need a server, storage and links that keep working. - Camera overlay with background removal, which is a separate segmentation problem. - Multi-track editing. This is a recorder with automatic motion, not a video editor, and the moment it becomes one the automation stops being the point. ### README must contain - That the recording is captured flat and every effect is applied at export, so nothing is baked in - That the cursor track is sampled far faster than the frame rate, and why - The three numbers that control the feel: minimum interval, minimum dwell and maximum zoom rate - Which permissions are needed and what refusing each one costs - That export runs on the GPU, and what to check if it becomes slower than real time ===== AGENTS.md ===== # Agent instructions · Screen Studio indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Swift 6 and SwiftUI in Xcode, ScreenCaptureKit for video, plus a separate cursor event track, Core Image and Metal, rendered through AVAssetWriter, Cubic easing over generated keyframes, The flat recording plus a sidecar JSON project file. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. ===== BUILD_PLAN.md ===== # Build plan · Screen Studio A recorder that captures the screen flat while writing a separate high-frequency cursor and click track, then generates smooth zoom keyframes from that track in post, applies padding, rounded corners and a background, and renders a finished video you never had to edit by hand. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Record flat, and capture the cursor track A plain recording exists alongside a dense, accurate record of where the cursor was and what it did. ### Steps 1. Capture the screen at native resolution SCStream configured for the display's real pixel dimensions. Write with AVAssetWriter, starting the session from the first video sample's timestamp. Files: `Recorder.swift` 2. Sample the cursor on its own timer A high-rate timer reading the global cursor position, timestamped against the same clock as the video session. Log clicks with an event tap or a global monitor. Files: `CursorTrack.swift` 3. Verify alignment before trusting it Record a deliberate click at a known screen position, then confirm the logged event timestamp matches the frame where the click is visible. Files: `test/alignment.md` ### Done when - [ ] The cursor sample count far exceeds the video frame count for the same duration - [ ] A logged click aligns with the visible click within one frame - [ ] The flat video opens in QuickTime without a repair prompt - [ ] The recording is at native pixel resolution, not scaled down ### Watch out - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. ## Phase 2 · Generating zoom from the track The zoom keyframes are computed from what the cursor did, and they are good enough that you rarely touch them. ### Steps 1. Detect intent regions Cluster click events in time and space. Treat sustained dwell within a small radius as intent too, because reading and typing matter as much as clicking. Files: `Intent.swift` 2. Emit keyframes with lead and lag Zoom in slightly before the action so the viewer arrives with it, and hold briefly after so the result is visible before pulling out. Files: `Keyframes.swift` 3. Suppress noise A minimum interval between zooms, a minimum dwell to qualify, and a maximum zoom rate. These three numbers are the whole feel of the product. Files: `Keyframes.swift` ### Done when - [ ] Clicked fields get zooms and cursor transits between them do not - [ ] No two zoom-ins are closer than the configured minimum interval - [ ] A recording with no clicks produces no keyframes - [ ] Keyframes are stored separately from the video and can be regenerated ### Watch out - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. ## Phase 3 · Easing and framing The motion looks designed rather than computed, and the frame never shows the edge of the source. ### Steps 1. Implement cubic easing Ease in and out between keyframe pairs. Compare against linear side by side once; you will not go back. Files: `Easing.swift` 2. Clamp the centre inside the source Compute the visible rectangle at the current scale and clamp its centre so the rectangle stays inside the source. Test the corners explicitly. Files: `Framing.swift` 3. Composite padding, corners and background A Core Image pipeline: scale and translate the source, round the corners, and draw over a background colour or gradient. Files: `Composite.swift` ### Done when - [ ] A zoom at a screen corner never reveals area outside the source - [ ] Motion eases rather than moving at constant speed - [ ] Padding and corner radius change on re-render without re-recording - [ ] The background renders behind the rounded corners with no fringing ### Watch out - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. ## Phase 4 · Rendering and export The finished video comes out at the size you asked for, and rendering is not slower than watching. ### Steps 1. Keep the composite on the GPU One Metal-backed CIContext reused across frames. Creating a context per frame is the single most common reason exports crawl. Files: `Renderer.swift` 2. Add export presets with exact dimensions Named presets with fixed pixel sizes. Assert the output dimensions rather than trusting the scale arithmetic. Files: `Presets.swift` 3. Carry audio through unchanged Pass the original audio track through rather than re-encoding it, and confirm sync at the end of a long export. Files: `Renderer.swift` ### Done when - [ ] A two minute recording exports in under two minutes - [ ] Each preset produces exactly the pixel dimensions it names - [ ] Audio is in sync at the end of the exported file - [ ] Cancelling an export leaves no partial file behind ### Watch out - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. ## Phase 5 · Trim and manual correction The one zoom the generator got wrong takes ten seconds to fix. ### Steps 1. Build the timeline Video thumbnails, click markers and keyframe handles on one scrubbable track. Files: `Timeline.swift` 2. Make keyframes editable Drag to move, delete, insert. Preview updates from the source, which is why nothing was baked in. Files: `Timeline.swift` 3. Protect manual edits Mark keyframes as generated or manual, and warn before regeneration discards manual ones. Files: `Keyframes.swift` ### Done when - [ ] Deleting a keyframe removes the zoom from preview and export - [ ] A manually added keyframe produces a zoom at that time - [ ] Trimming does not desynchronise keyframes from content - [ ] Regeneration warns before discarding manual edits ### Watch out - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. ## Not in this build - The tuned motion design judgement of the commercial product. - Hosted sharing, links and viewer analytics. - Camera overlay with background removal. - Multi-track video editing. ## After v1, if you want it - Keystroke overlays generated from the event track you are already recording - Automatic silence trimming using the audio track - A vertical export preset that reframes around the cursor rather than letterboxing ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where flat recordings and project files are written. These are large, so pick a disk with room. RECORDINGS_DIR=~/Movies/ScreenStudio # Required. Cursor sampling rate. Must be well above the video frame rate or the motion looks stepped. CURSOR_SAMPLE_HZ=120 # Required. Minimum time between zoom-ins. Raising this calms a busy video. ZOOM_MIN_INTERVAL_MS=2500 # Required. How long the cursor must stay in an area to count as intent. Lowering it makes the video restless. ZOOM_MIN_DWELL_MS=600 # Required. How far in the automatic zoom will go. Beyond about two and a half the source pixels start to show. ZOOM_MAX_SCALE=2.2 # Optional. Product phase only. SPARKLE_FEED_URL=https://example.com/appcast.xml
You are building a production product version of Screen Studio. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Screen Studio · product brief ## Problem The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For Screen Studio, record a Mac screen and automatically apply cursor zoom, padding, and smooth framing. The hard boundary is exceptional macos capture polish, motion design, and ongoing compatibility, plus capture polish, hosting, and collaboration. ## Product outcome A recorder that turns an unedited screen capture into a video that looks edited, keeps the source untouched so every decision stays reversible, and exports faster than real time. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - macOS or Windows screen-recording permission - microphone permission - ffmpeg - local disk space ## Explicit non-goals for v1 - The tuned motion design judgement of the commercial product. - Hosted sharing, links and viewer analytics. - Camera overlay with background removal. - Multi-track video editing. - exceptional macOS capture polish, motion design, and ongoing compatibility - instant hosted sharing - viewer analytics - team libraries - cross-device capture ## Success criteria - The three fixture recordings still produce sensible keyframes after any generation change - A two minute export completes in under two minutes - Corner zooms never reveal area outside the source - Audio sync is verified at the end of a long export - spctl accepts the notarized build on a Mac that never ran it ===== BRIEF.md ===== # Build brief · Screen Studio The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a screen recorder that automatically zooms to the cursor and renders a polished video without manual editing to replace Screen Studio. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Language: Swift 6 and SwiftUI in Xcode · ScreenCaptureKit and the video pipeline are Apple frameworks - Capture: ScreenCaptureKit for video, plus a separate cursor event track · the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards - Composition: Core Image and Metal, rendered through AVAssetWriter · zoom, padding and background are per-frame transforms, which is what the GPU is for - Interpolation: Cubic easing over generated keyframes · linear zoom looks mechanical, and the easing is the entire perceived quality of the effect - Storage: The flat recording plus a sidecar JSON project file · keeping the source untouched means every stylistic decision stays reversible ### Data model (create this before Phase 1) - recordings(id, video_path, width, height, scale, fps, started_at, duration_ms) - cursor_samples(recording_id, t_ms, x, y) · sampled at a high fixed rate, independent of video frame rate - cursor_events(recording_id, t_ms, kind) · kind is click, drag_start, drag_end, keypress or scroll - keyframes(id, recording_id, t_ms, scale, center_x, center_y, easing) · generated in Phase 2, editable afterwards - projects(id, recording_id, padding, corner_radius, background_json, trim_start_ms, trim_end_ms) - The flat recording is never modified. Every effect is a function of the source plus the project row, which is what makes re-rendering at a different size free. ### Phase 1 · Record flat, and capture the cursor track Build: Record the screen with ScreenCaptureKit at native resolution with no effects applied. In parallel, sample the cursor position at a high fixed rate and log clicks and drags with timestamps on the same clock as the video. Trap: Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. Trap: Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. Done when: A recording produces a flat video plus a cursor track whose sample count is far higher than the frame count for the same duration. Clicks in the track line up with visible clicks in the video within a frame. The video plays in QuickTime with no repair prompt. Do not build yet: No zoom, no styling, no export options. A clean source and an accurate track. ### Phase 2 · Generating zoom from the track Build: Detect regions of interest: clusters of clicks, sustained activity in one area, and typing. Emit a zoom-in keyframe shortly before the activity and a zoom-out after it. Suppress zooms for brief cursor transits, and enforce a minimum interval so the video does not pulse. Trap: Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. Done when: A recording of a form being filled produces zooms on the fields that were clicked and none on the cursor moving between them. No two zoom-ins occur closer together than the configured minimum. A recording with no clicks at all produces no keyframes rather than a default zoom. Do not build yet: No easing or styling yet, so the result will look mechanical. That is expected. ### Phase 3 · Easing and framing Build: Interpolate between keyframes with a cubic ease, not linearly. Clamp the zoom centre so the visible rectangle always stays inside the source bounds. Add padding, rounded corners and a background so the recording sits inside a frame. Trap: Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. Done when: Zooming at the far corner of the screen never shows an area outside the source. Motion between keyframes accelerates and decelerates rather than moving at constant speed. Changing the padding and corner radius re-renders without re-recording. Do not build yet: No trimming or export presets yet. ### Phase 4 · Rendering and export Build: Render the composed frames through AVAssetWriter with hardware encoding. Offer export presets with exact pixel dimensions for common destinations. Show real progress, and keep audio in sync with the composed video. Trap: Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. Done when: A two minute recording exports in less than its own duration on the machine you built it on. An export preset produces exactly the pixel dimensions it names. Audio remains in sync at the end of the export, checked against the source recording. Do not build yet: No sharing or distribution yet. ### Phase 5 · Trim and manual correction Build: A timeline showing the video, the click track and the generated keyframes. Trim the start and end. Move, delete or add a keyframe. Every change re-renders the preview from the untouched source. Trap: Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. Done when: Deleting a generated keyframe removes that zoom from the preview and from the export. Adding one by hand produces a zoom at that moment. Trimming the start does not shift the keyframe timing relative to the visible content. Re-running generation after manual edits warns before discarding them. Do not build yet: Distribution is the last phase. ### Phase 6 · Signing, notarization and updates Build: Hardened runtime, Developer ID signing, notarization and stapling, plus Sparkle for updates. Document the permission story, which here is Screen Recording and microphone. Trap: Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps. Done when: A notarized build opens on a fresh Mac with no Gatekeeper warning and requests its permissions once. An appcast update installs and relaunches. Do not build yet: Nothing after this. ### Out of scope (and why) - The motion design judgement behind the commercial product. The thresholds in Phase 2 are the whole craft, and theirs are tuned against far more footage than you will record. - Hosted sharing and viewer analytics, which need a server, storage and links that keep working. - Camera overlay with background removal, which is a separate segmentation problem. - Multi-track editing. This is a recorder with automatic motion, not a video editor, and the moment it becomes one the automation stops being the point. ### README must contain - That the recording is captured flat and every effect is applied at export, so nothing is baked in - That the cursor track is sampled far faster than the frame rate, and why - The three numbers that control the feel: minimum interval, minimum dwell and maximum zoom rate - Which permissions are needed and what refusing each one costs - That export runs on the GPU, and what to check if it becomes slower than real time ===== ARCHITECTURE.md ===== # Architecture · Screen Studio ## Stack | Part | Choice | Why | | --- | --- | --- | | Language | Swift 6 and SwiftUI in Xcode | ScreenCaptureKit and the video pipeline are Apple frameworks | | Capture | ScreenCaptureKit for video, plus a separate cursor event track | the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards | | Composition | Core Image and Metal, rendered through AVAssetWriter | zoom, padding and background are per-frame transforms, which is what the GPU is for | | Interpolation | Cubic easing over generated keyframes | linear zoom looks mechanical, and the easing is the entire perceived quality of the effect | | Storage | The flat recording plus a sidecar JSON project file | keeping the source untouched means every stylistic decision stays reversible | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Capture | The flat recording and the cursor track. | ScreenCaptureKit is the only path on macOS. The cursor sampler is independent and is the part worth protecting, because the track is what everything downstream needs. | | Generation | Turning cursor data into keyframes. | This is a pure function from track to keyframes and can be rewritten entirely without touching capture or render. Keep the three fixtures as its regression suite. | | Composite | Per-frame transform, padding, corners and background. | Core Image today, a custom Metal shader if you need effects it cannot express. The interface is a frame in and a frame out. | | Render | Encoding and export presets. | AVAssetWriter is the sensible default. An ffmpeg path would add formats at the cost of hardware encoding. | | Timeline | Manual correction of generated keyframes. | Presentation only. The keyframes are data and any editor over them works. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `RECORDINGS_DIR` · required · Where flat recordings and project files are written. These are large, so pick a disk with room. - `CURSOR_SAMPLE_HZ` · required · Cursor sampling rate. Must be well above the video frame rate or the motion looks stepped. - `ZOOM_MIN_INTERVAL_MS` · required · Minimum time between zoom-ins. Raising this calms a busy video. - `ZOOM_MIN_DWELL_MS` · required · How long the cursor must stay in an area to count as intent. Lowering it makes the video restless. - `ZOOM_MAX_SCALE` · required · How far in the automatic zoom will go. Beyond about two and a half the source pixels start to show. - `SPARKLE_FEED_URL` · optional · Product phase only. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Screen Studio product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Swift 6 and SwiftUI in Xcode, ScreenCaptureKit for video, plus a separate cursor event track, Core Image and Metal, rendered through AVAssetWriter, Cubic easing over generated keyframes, The flat recording plus a sidecar JSON project file. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. - Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps. ===== MILESTONES.md ===== # Delivery milestones · Screen Studio Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Record flat, and capture the cursor track A plain recording exists alongside a dense, accurate record of where the cursor was and what it did. ### Steps 1. Capture the screen at native resolution SCStream configured for the display's real pixel dimensions. Write with AVAssetWriter, starting the session from the first video sample's timestamp. Files: `Recorder.swift` 2. Sample the cursor on its own timer A high-rate timer reading the global cursor position, timestamped against the same clock as the video session. Log clicks with an event tap or a global monitor. Files: `CursorTrack.swift` 3. Verify alignment before trusting it Record a deliberate click at a known screen position, then confirm the logged event timestamp matches the frame where the click is visible. Files: `test/alignment.md` ### Done when - [ ] The cursor sample count far exceeds the video frame count for the same duration - [ ] A logged click aligns with the visible click within one frame - [ ] The flat video opens in QuickTime without a repair prompt - [ ] The recording is at native pixel resolution, not scaled down ### Watch out - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. ## M2 · Generating zoom from the track The zoom keyframes are computed from what the cursor did, and they are good enough that you rarely touch them. ### Steps 1. Detect intent regions Cluster click events in time and space. Treat sustained dwell within a small radius as intent too, because reading and typing matter as much as clicking. Files: `Intent.swift` 2. Emit keyframes with lead and lag Zoom in slightly before the action so the viewer arrives with it, and hold briefly after so the result is visible before pulling out. Files: `Keyframes.swift` 3. Suppress noise A minimum interval between zooms, a minimum dwell to qualify, and a maximum zoom rate. These three numbers are the whole feel of the product. Files: `Keyframes.swift` ### Done when - [ ] Clicked fields get zooms and cursor transits between them do not - [ ] No two zoom-ins are closer than the configured minimum interval - [ ] A recording with no clicks produces no keyframes - [ ] Keyframes are stored separately from the video and can be regenerated ### Watch out - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. ## M3 · Easing and framing The motion looks designed rather than computed, and the frame never shows the edge of the source. ### Steps 1. Implement cubic easing Ease in and out between keyframe pairs. Compare against linear side by side once; you will not go back. Files: `Easing.swift` 2. Clamp the centre inside the source Compute the visible rectangle at the current scale and clamp its centre so the rectangle stays inside the source. Test the corners explicitly. Files: `Framing.swift` 3. Composite padding, corners and background A Core Image pipeline: scale and translate the source, round the corners, and draw over a background colour or gradient. Files: `Composite.swift` ### Done when - [ ] A zoom at a screen corner never reveals area outside the source - [ ] Motion eases rather than moving at constant speed - [ ] Padding and corner radius change on re-render without re-recording - [ ] The background renders behind the rounded corners with no fringing ### Watch out - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. ## M4 · Rendering and export The finished video comes out at the size you asked for, and rendering is not slower than watching. ### Steps 1. Keep the composite on the GPU One Metal-backed CIContext reused across frames. Creating a context per frame is the single most common reason exports crawl. Files: `Renderer.swift` 2. Add export presets with exact dimensions Named presets with fixed pixel sizes. Assert the output dimensions rather than trusting the scale arithmetic. Files: `Presets.swift` 3. Carry audio through unchanged Pass the original audio track through rather than re-encoding it, and confirm sync at the end of a long export. Files: `Renderer.swift` ### Done when - [ ] A two minute recording exports in under two minutes - [ ] Each preset produces exactly the pixel dimensions it names - [ ] Audio is in sync at the end of the exported file - [ ] Cancelling an export leaves no partial file behind ### Watch out - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. ## M5 · Trim and manual correction The one zoom the generator got wrong takes ten seconds to fix. ### Steps 1. Build the timeline Video thumbnails, click markers and keyframe handles on one scrubbable track. Files: `Timeline.swift` 2. Make keyframes editable Drag to move, delete, insert. Preview updates from the source, which is why nothing was baked in. Files: `Timeline.swift` 3. Protect manual edits Mark keyframes as generated or manual, and warn before regeneration discards manual ones. Files: `Keyframes.swift` ### Done when - [ ] Deleting a keyframe removes the zoom from preview and export - [ ] A manually added keyframe produces a zoom at that time - [ ] Trimming does not desynchronise keyframes from content - [ ] Regeneration warns before discarding manual edits ### Watch out - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. ## M6 · Signing, notarization and updates (production only) Other people can install it and receive fixes. ### Steps 1. Sign and notarize Hardened runtime, Developer ID, notarytool, stapler. Confirm the usage description strings are present or the prompts never appear. ```sh xcrun notarytool submit Studio.zip --keychain-profile AC --wait xcrun stapler staple Studio.app ``` 2. Publish updates with Sparkle EdDSA key pair, private half outside the repository, appcast over HTTPS. Files: `appcast.xml` 3. Document the permissions Which permission enables which feature, and what refusing each one costs. Files: `README.md` ### Done when - [ ] spctl reports the build accepted - [ ] A fresh Mac opens it with no warning - [ ] Permissions are requested once and the recording then works - [ ] An appcast update installs and relaunches ### Watch out - Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps. ===== OPERATIONS.md ===== # Operations · Screen Studio ## Backup Flat recordings plus their sidecar project files. The rendered exports are regenerable and do not need backing up, which is the practical benefit of never baking effects in. ## Restore Copy the recordings directory back. A project file without its recording is useless, so keep them together and say so in the interface when a project's source is missing. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Watch export speed against real time and dropped frame counts from the capture stream. A stream that silently drops frames produces a video that feels wrong without anything obviously being broken. ## Incident checklist A macOS update changing capture behaviour is the recurring risk. Keep the previous signed build available and set a minimum supported version in the appcast. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The three fixture recordings still produce sensible keyframes after any generation change - [ ] A two minute export completes in under two minutes - [ ] Corner zooms never reveal area outside the source - [ ] Audio sync is verified at the end of a long export - [ ] spctl accepts the notarized build on a Mac that never ran it ## Launch constraint Do not market omitted Screen Studio capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where flat recordings and project files are written. These are large, so pick a disk with room. RECORDINGS_DIR=~/Movies/ScreenStudio # Required. Cursor sampling rate. Must be well above the video frame rate or the motion looks stepped. CURSOR_SAMPLE_HZ=120 # Required. Minimum time between zoom-ins. Raising this calms a busy video. ZOOM_MIN_INTERVAL_MS=2500 # Required. How long the cursor must stay in an area to count as intent. Lowering it makes the video restless. ZOOM_MIN_DWELL_MS=600 # Required. How far in the automatic zoom will go. Beyond about two and a half the source pixels start to show. ZOOM_MAX_SCALE=2.2 # Optional. Product phase only. SPARKLE_FEED_URL=https://example.com/appcast.xml
# Screen Studio · indie build A recorder that captures the screen flat while writing a separate high-frequency cursor and click track, then generates smooth zoom keyframes from that track in post, applies padding, rounded corners and a background, and renders a finished video you never had to edit by hand. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Language | Swift 6 and SwiftUI in Xcode | ScreenCaptureKit and the video pipeline are Apple frameworks | | Capture | ScreenCaptureKit for video, plus a separate cursor event track | the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards | | Composition | Core Image and Metal, rendered through AVAssetWriter | zoom, padding and background are per-frame transforms, which is what the GPU is for | | Interpolation | Cubic easing over generated keyframes | linear zoom looks mechanical, and the easing is the entire perceived quality of the effect | | Storage | The flat recording plus a sidecar JSON project file | keeping the source untouched means every stylistic decision stays reversible | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **A Mac running macOS 13 or newer with a discrete or Apple silicon GPU** · free if you already own a Mac - Why: ScreenCaptureKit needs 12.3 as a floor, and the composite pipeline is GPU work. An old integrated GPU will export slower than real time. - Get it: Apple menu, then About This Mac. - Verify: About This Mac reports macOS 13 or higher - [ ] **Xcode 16 or newer** · free - Why: Builds the app and provides Instruments, which you will need when the export is slow. - Get it: Mac App Store or developer.apple.com/xcode. - Verify: xcodebuild -version prints 16 or higher - [ ] **Screen Recording and Microphone permissions** · free - Why: Screen Recording is required to capture at all. Microphone is optional and only for narration. - Get it: System Settings, then Privacy and Security. Both must be re-granted during development whenever the binary changes signature or path. - Verify: A test capture returns pixels rather than a black frame - [ ] **Three real recordings to tune against** · free - Why: The thresholds in Phase 2 cannot be chosen in the abstract. You need a form being filled, a terminal session and a slow read to see where the generator over-triggers. - Get it: Record them yourself before writing the detection code, and keep them as fixtures. - Verify: Three recordings with their cursor tracks are saved as test fixtures - [ ] **Apple Developer Program membership** (optional) · 99 USD per year - Why: Product phase only. Without it there is no notarization and a Gatekeeper warning on every other Mac. - Get it: Enrol at developer.apple.com/programs. - Verify: A Developer ID Application certificate appears in Xcode settings ## Quick start ```sh xcrun notarytool submit Studio.zip --keychain-profile AC --wait xcrun stapler staple Studio.app ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - The tuned motion design judgement of the commercial product. - Hosted sharing, links and viewer analytics. - Camera overlay with background removal. - Multi-track video editing. - exceptional macOS capture polish, motion design, and ongoing compatibility - instant hosted sharing - viewer analytics - team libraries - cross-device capture If one of those is essential to you, that is the reason to keep paying for Screen Studio, and the README should say so rather than pretend.
# Build brief · Screen Studio The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a screen recorder that automatically zooms to the cursor and renders a polished video without manual editing to replace Screen Studio. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Language: Swift 6 and SwiftUI in Xcode · ScreenCaptureKit and the video pipeline are Apple frameworks - Capture: ScreenCaptureKit for video, plus a separate cursor event track · the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards - Composition: Core Image and Metal, rendered through AVAssetWriter · zoom, padding and background are per-frame transforms, which is what the GPU is for - Interpolation: Cubic easing over generated keyframes · linear zoom looks mechanical, and the easing is the entire perceived quality of the effect - Storage: The flat recording plus a sidecar JSON project file · keeping the source untouched means every stylistic decision stays reversible ### Data model (create this before Phase 1) - recordings(id, video_path, width, height, scale, fps, started_at, duration_ms) - cursor_samples(recording_id, t_ms, x, y) · sampled at a high fixed rate, independent of video frame rate - cursor_events(recording_id, t_ms, kind) · kind is click, drag_start, drag_end, keypress or scroll - keyframes(id, recording_id, t_ms, scale, center_x, center_y, easing) · generated in Phase 2, editable afterwards - projects(id, recording_id, padding, corner_radius, background_json, trim_start_ms, trim_end_ms) - The flat recording is never modified. Every effect is a function of the source plus the project row, which is what makes re-rendering at a different size free. ### Phase 1 · Record flat, and capture the cursor track Build: Record the screen with ScreenCaptureKit at native resolution with no effects applied. In parallel, sample the cursor position at a high fixed rate and log clicks and drags with timestamps on the same clock as the video. Trap: Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. Trap: Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. Done when: A recording produces a flat video plus a cursor track whose sample count is far higher than the frame count for the same duration. Clicks in the track line up with visible clicks in the video within a frame. The video plays in QuickTime with no repair prompt. Do not build yet: No zoom, no styling, no export options. A clean source and an accurate track. ### Phase 2 · Generating zoom from the track Build: Detect regions of interest: clusters of clicks, sustained activity in one area, and typing. Emit a zoom-in keyframe shortly before the activity and a zoom-out after it. Suppress zooms for brief cursor transits, and enforce a minimum interval so the video does not pulse. Trap: Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. Done when: A recording of a form being filled produces zooms on the fields that were clicked and none on the cursor moving between them. No two zoom-ins occur closer together than the configured minimum. A recording with no clicks at all produces no keyframes rather than a default zoom. Do not build yet: No easing or styling yet, so the result will look mechanical. That is expected. ### Phase 3 · Easing and framing Build: Interpolate between keyframes with a cubic ease, not linearly. Clamp the zoom centre so the visible rectangle always stays inside the source bounds. Add padding, rounded corners and a background so the recording sits inside a frame. Trap: Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. Done when: Zooming at the far corner of the screen never shows an area outside the source. Motion between keyframes accelerates and decelerates rather than moving at constant speed. Changing the padding and corner radius re-renders without re-recording. Do not build yet: No trimming or export presets yet. ### Phase 4 · Rendering and export Build: Render the composed frames through AVAssetWriter with hardware encoding. Offer export presets with exact pixel dimensions for common destinations. Show real progress, and keep audio in sync with the composed video. Trap: Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. Done when: A two minute recording exports in less than its own duration on the machine you built it on. An export preset produces exactly the pixel dimensions it names. Audio remains in sync at the end of the export, checked against the source recording. Do not build yet: No sharing or distribution yet. ### Phase 5 · Trim and manual correction Build: A timeline showing the video, the click track and the generated keyframes. Trim the start and end. Move, delete or add a keyframe. Every change re-renders the preview from the untouched source. Trap: Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. Done when: Deleting a generated keyframe removes that zoom from the preview and from the export. Adding one by hand produces a zoom at that moment. Trimming the start does not shift the keyframe timing relative to the visible content. Re-running generation after manual edits warns before discarding them. Do not build yet: Distribution is the last phase. ### Phase 6 · Signing, notarization and updates Build: Hardened runtime, Developer ID signing, notarization and stapling, plus Sparkle for updates. Document the permission story, which here is Screen Recording and microphone. Trap: Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps. Done when: A notarized build opens on a fresh Mac with no Gatekeeper warning and requests its permissions once. An appcast update installs and relaunches. Do not build yet: Nothing after this. ### Out of scope (and why) - The motion design judgement behind the commercial product. The thresholds in Phase 2 are the whole craft, and theirs are tuned against far more footage than you will record. - Hosted sharing and viewer analytics, which need a server, storage and links that keep working. - Camera overlay with background removal, which is a separate segmentation problem. - Multi-track editing. This is a recorder with automatic motion, not a video editor, and the moment it becomes one the automation stops being the point. ### README must contain - That the recording is captured flat and every effect is applied at export, so nothing is baked in - That the cursor track is sampled far faster than the frame rate, and why - The three numbers that control the feel: minimum interval, minimum dwell and maximum zoom rate - Which permissions are needed and what refusing each one costs - That export runs on the GPU, and what to check if it becomes slower than real time
# Agent instructions · Screen Studio indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Swift 6 and SwiftUI in Xcode, ScreenCaptureKit for video, plus a separate cursor event track, Core Image and Metal, rendered through AVAssetWriter, Cubic easing over generated keyframes, The flat recording plus a sidecar JSON project file. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final.
# Build plan · Screen Studio A recorder that captures the screen flat while writing a separate high-frequency cursor and click track, then generates smooth zoom keyframes from that track in post, applies padding, rounded corners and a background, and renders a finished video you never had to edit by hand. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Record flat, and capture the cursor track A plain recording exists alongside a dense, accurate record of where the cursor was and what it did. ### Steps 1. Capture the screen at native resolution SCStream configured for the display's real pixel dimensions. Write with AVAssetWriter, starting the session from the first video sample's timestamp. Files: `Recorder.swift` 2. Sample the cursor on its own timer A high-rate timer reading the global cursor position, timestamped against the same clock as the video session. Log clicks with an event tap or a global monitor. Files: `CursorTrack.swift` 3. Verify alignment before trusting it Record a deliberate click at a known screen position, then confirm the logged event timestamp matches the frame where the click is visible. Files: `test/alignment.md` ### Done when - [ ] The cursor sample count far exceeds the video frame count for the same duration - [ ] A logged click aligns with the visible click within one frame - [ ] The flat video opens in QuickTime without a repair prompt - [ ] The recording is at native pixel resolution, not scaled down ### Watch out - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. ## Phase 2 · Generating zoom from the track The zoom keyframes are computed from what the cursor did, and they are good enough that you rarely touch them. ### Steps 1. Detect intent regions Cluster click events in time and space. Treat sustained dwell within a small radius as intent too, because reading and typing matter as much as clicking. Files: `Intent.swift` 2. Emit keyframes with lead and lag Zoom in slightly before the action so the viewer arrives with it, and hold briefly after so the result is visible before pulling out. Files: `Keyframes.swift` 3. Suppress noise A minimum interval between zooms, a minimum dwell to qualify, and a maximum zoom rate. These three numbers are the whole feel of the product. Files: `Keyframes.swift` ### Done when - [ ] Clicked fields get zooms and cursor transits between them do not - [ ] No two zoom-ins are closer than the configured minimum interval - [ ] A recording with no clicks produces no keyframes - [ ] Keyframes are stored separately from the video and can be regenerated ### Watch out - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. ## Phase 3 · Easing and framing The motion looks designed rather than computed, and the frame never shows the edge of the source. ### Steps 1. Implement cubic easing Ease in and out between keyframe pairs. Compare against linear side by side once; you will not go back. Files: `Easing.swift` 2. Clamp the centre inside the source Compute the visible rectangle at the current scale and clamp its centre so the rectangle stays inside the source. Test the corners explicitly. Files: `Framing.swift` 3. Composite padding, corners and background A Core Image pipeline: scale and translate the source, round the corners, and draw over a background colour or gradient. Files: `Composite.swift` ### Done when - [ ] A zoom at a screen corner never reveals area outside the source - [ ] Motion eases rather than moving at constant speed - [ ] Padding and corner radius change on re-render without re-recording - [ ] The background renders behind the rounded corners with no fringing ### Watch out - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. ## Phase 4 · Rendering and export The finished video comes out at the size you asked for, and rendering is not slower than watching. ### Steps 1. Keep the composite on the GPU One Metal-backed CIContext reused across frames. Creating a context per frame is the single most common reason exports crawl. Files: `Renderer.swift` 2. Add export presets with exact dimensions Named presets with fixed pixel sizes. Assert the output dimensions rather than trusting the scale arithmetic. Files: `Presets.swift` 3. Carry audio through unchanged Pass the original audio track through rather than re-encoding it, and confirm sync at the end of a long export. Files: `Renderer.swift` ### Done when - [ ] A two minute recording exports in under two minutes - [ ] Each preset produces exactly the pixel dimensions it names - [ ] Audio is in sync at the end of the exported file - [ ] Cancelling an export leaves no partial file behind ### Watch out - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. ## Phase 5 · Trim and manual correction The one zoom the generator got wrong takes ten seconds to fix. ### Steps 1. Build the timeline Video thumbnails, click markers and keyframe handles on one scrubbable track. Files: `Timeline.swift` 2. Make keyframes editable Drag to move, delete, insert. Preview updates from the source, which is why nothing was baked in. Files: `Timeline.swift` 3. Protect manual edits Mark keyframes as generated or manual, and warn before regeneration discards manual ones. Files: `Keyframes.swift` ### Done when - [ ] Deleting a keyframe removes the zoom from preview and export - [ ] A manually added keyframe produces a zoom at that time - [ ] Trimming does not desynchronise keyframes from content - [ ] Regeneration warns before discarding manual edits ### Watch out - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. ## Not in this build - The tuned motion design judgement of the commercial product. - Hosted sharing, links and viewer analytics. - Camera overlay with background removal. - Multi-track video editing. ## After v1, if you want it - Keystroke overlays generated from the event track you are already recording - Automatic silence trimming using the audio track - A vertical export preset that reframes around the cursor rather than letterboxing
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where flat recordings and project files are written. These are large, so pick a disk with room. RECORDINGS_DIR=~/Movies/ScreenStudio # Required. Cursor sampling rate. Must be well above the video frame rate or the motion looks stepped. CURSOR_SAMPLE_HZ=120 # Required. Minimum time between zoom-ins. Raising this calms a busy video. ZOOM_MIN_INTERVAL_MS=2500 # Required. How long the cursor must stay in an area to count as intent. Lowering it makes the video restless. ZOOM_MIN_DWELL_MS=600 # Required. How far in the automatic zoom will go. Beyond about two and a half the source pixels start to show. ZOOM_MAX_SCALE=2.2 # Optional. Product phase only. SPARKLE_FEED_URL=https://example.com/appcast.xml
# Screen Studio · product brief ## Problem The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For Screen Studio, record a Mac screen and automatically apply cursor zoom, padding, and smooth framing. The hard boundary is exceptional macos capture polish, motion design, and ongoing compatibility, plus capture polish, hosting, and collaboration. ## Product outcome A recorder that turns an unedited screen capture into a video that looks edited, keeps the source untouched so every decision stays reversible, and exports faster than real time. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - macOS or Windows screen-recording permission - microphone permission - ffmpeg - local disk space ## Explicit non-goals for v1 - The tuned motion design judgement of the commercial product. - Hosted sharing, links and viewer analytics. - Camera overlay with background removal. - Multi-track video editing. - exceptional macOS capture polish, motion design, and ongoing compatibility - instant hosted sharing - viewer analytics - team libraries - cross-device capture ## Success criteria - The three fixture recordings still produce sensible keyframes after any generation change - A two minute export completes in under two minutes - Corner zooms never reveal area outside the source - Audio sync is verified at the end of a long export - spctl accepts the notarized build on a Mac that never ran it
# Build brief · Screen Studio The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a screen recorder that automatically zooms to the cursor and renders a polished video without manual editing to replace Screen Studio. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Language: Swift 6 and SwiftUI in Xcode · ScreenCaptureKit and the video pipeline are Apple frameworks - Capture: ScreenCaptureKit for video, plus a separate cursor event track · the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards - Composition: Core Image and Metal, rendered through AVAssetWriter · zoom, padding and background are per-frame transforms, which is what the GPU is for - Interpolation: Cubic easing over generated keyframes · linear zoom looks mechanical, and the easing is the entire perceived quality of the effect - Storage: The flat recording plus a sidecar JSON project file · keeping the source untouched means every stylistic decision stays reversible ### Data model (create this before Phase 1) - recordings(id, video_path, width, height, scale, fps, started_at, duration_ms) - cursor_samples(recording_id, t_ms, x, y) · sampled at a high fixed rate, independent of video frame rate - cursor_events(recording_id, t_ms, kind) · kind is click, drag_start, drag_end, keypress or scroll - keyframes(id, recording_id, t_ms, scale, center_x, center_y, easing) · generated in Phase 2, editable afterwards - projects(id, recording_id, padding, corner_radius, background_json, trim_start_ms, trim_end_ms) - The flat recording is never modified. Every effect is a function of the source plus the project row, which is what makes re-rendering at a different size free. ### Phase 1 · Record flat, and capture the cursor track Build: Record the screen with ScreenCaptureKit at native resolution with no effects applied. In parallel, sample the cursor position at a high fixed rate and log clicks and drags with timestamps on the same clock as the video. Trap: Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. Trap: Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. Done when: A recording produces a flat video plus a cursor track whose sample count is far higher than the frame count for the same duration. Clicks in the track line up with visible clicks in the video within a frame. The video plays in QuickTime with no repair prompt. Do not build yet: No zoom, no styling, no export options. A clean source and an accurate track. ### Phase 2 · Generating zoom from the track Build: Detect regions of interest: clusters of clicks, sustained activity in one area, and typing. Emit a zoom-in keyframe shortly before the activity and a zoom-out after it. Suppress zooms for brief cursor transits, and enforce a minimum interval so the video does not pulse. Trap: Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. Done when: A recording of a form being filled produces zooms on the fields that were clicked and none on the cursor moving between them. No two zoom-ins occur closer together than the configured minimum. A recording with no clicks at all produces no keyframes rather than a default zoom. Do not build yet: No easing or styling yet, so the result will look mechanical. That is expected. ### Phase 3 · Easing and framing Build: Interpolate between keyframes with a cubic ease, not linearly. Clamp the zoom centre so the visible rectangle always stays inside the source bounds. Add padding, rounded corners and a background so the recording sits inside a frame. Trap: Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. Done when: Zooming at the far corner of the screen never shows an area outside the source. Motion between keyframes accelerates and decelerates rather than moving at constant speed. Changing the padding and corner radius re-renders without re-recording. Do not build yet: No trimming or export presets yet. ### Phase 4 · Rendering and export Build: Render the composed frames through AVAssetWriter with hardware encoding. Offer export presets with exact pixel dimensions for common destinations. Show real progress, and keep audio in sync with the composed video. Trap: Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. Done when: A two minute recording exports in less than its own duration on the machine you built it on. An export preset produces exactly the pixel dimensions it names. Audio remains in sync at the end of the export, checked against the source recording. Do not build yet: No sharing or distribution yet. ### Phase 5 · Trim and manual correction Build: A timeline showing the video, the click track and the generated keyframes. Trim the start and end. Move, delete or add a keyframe. Every change re-renders the preview from the untouched source. Trap: Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. Done when: Deleting a generated keyframe removes that zoom from the preview and from the export. Adding one by hand produces a zoom at that moment. Trimming the start does not shift the keyframe timing relative to the visible content. Re-running generation after manual edits warns before discarding them. Do not build yet: Distribution is the last phase. ### Phase 6 · Signing, notarization and updates Build: Hardened runtime, Developer ID signing, notarization and stapling, plus Sparkle for updates. Document the permission story, which here is Screen Recording and microphone. Trap: Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps. Done when: A notarized build opens on a fresh Mac with no Gatekeeper warning and requests its permissions once. An appcast update installs and relaunches. Do not build yet: Nothing after this. ### Out of scope (and why) - The motion design judgement behind the commercial product. The thresholds in Phase 2 are the whole craft, and theirs are tuned against far more footage than you will record. - Hosted sharing and viewer analytics, which need a server, storage and links that keep working. - Camera overlay with background removal, which is a separate segmentation problem. - Multi-track editing. This is a recorder with automatic motion, not a video editor, and the moment it becomes one the automation stops being the point. ### README must contain - That the recording is captured flat and every effect is applied at export, so nothing is baked in - That the cursor track is sampled far faster than the frame rate, and why - The three numbers that control the feel: minimum interval, minimum dwell and maximum zoom rate - Which permissions are needed and what refusing each one costs - That export runs on the GPU, and what to check if it becomes slower than real time
# Architecture · Screen Studio ## Stack | Part | Choice | Why | | --- | --- | --- | | Language | Swift 6 and SwiftUI in Xcode | ScreenCaptureKit and the video pipeline are Apple frameworks | | Capture | ScreenCaptureKit for video, plus a separate cursor event track | the motion design is generated from cursor data, and that data cannot be recovered from pixels afterwards | | Composition | Core Image and Metal, rendered through AVAssetWriter | zoom, padding and background are per-frame transforms, which is what the GPU is for | | Interpolation | Cubic easing over generated keyframes | linear zoom looks mechanical, and the easing is the entire perceived quality of the effect | | Storage | The flat recording plus a sidecar JSON project file | keeping the source untouched means every stylistic decision stays reversible | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Capture | The flat recording and the cursor track. | ScreenCaptureKit is the only path on macOS. The cursor sampler is independent and is the part worth protecting, because the track is what everything downstream needs. | | Generation | Turning cursor data into keyframes. | This is a pure function from track to keyframes and can be rewritten entirely without touching capture or render. Keep the three fixtures as its regression suite. | | Composite | Per-frame transform, padding, corners and background. | Core Image today, a custom Metal shader if you need effects it cannot express. The interface is a frame in and a frame out. | | Render | Encoding and export presets. | AVAssetWriter is the sensible default. An ffmpeg path would add formats at the cost of hardware encoding. | | Timeline | Manual correction of generated keyframes. | Presentation only. The keyframes are data and any editor over them works. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `RECORDINGS_DIR` · required · Where flat recordings and project files are written. These are large, so pick a disk with room. - `CURSOR_SAMPLE_HZ` · required · Cursor sampling rate. Must be well above the video frame rate or the motion looks stepped. - `ZOOM_MIN_INTERVAL_MS` · required · Minimum time between zoom-ins. Raising this calms a busy video. - `ZOOM_MIN_DWELL_MS` · required · How long the cursor must stay in an area to count as intent. Lowering it makes the video restless. - `ZOOM_MAX_SCALE` · required · How far in the automatic zoom will go. Beyond about two and a half the source pixels start to show. - `SPARKLE_FEED_URL` · optional · Product phase only. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
# Agent instructions · Screen Studio product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Swift 6 and SwiftUI in Xcode, ScreenCaptureKit for video, plus a separate cursor event track, Core Image and Metal, rendered through AVAssetWriter, Cubic easing over generated keyframes, The flat recording plus a sidecar JSON project file. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. - Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps.
# Delivery milestones · Screen Studio Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Record flat, and capture the cursor track A plain recording exists alongside a dense, accurate record of where the cursor was and what it did. ### Steps 1. Capture the screen at native resolution SCStream configured for the display's real pixel dimensions. Write with AVAssetWriter, starting the session from the first video sample's timestamp. Files: `Recorder.swift` 2. Sample the cursor on its own timer A high-rate timer reading the global cursor position, timestamped against the same clock as the video session. Log clicks with an event tap or a global monitor. Files: `CursorTrack.swift` 3. Verify alignment before trusting it Record a deliberate click at a known screen position, then confirm the logged event timestamp matches the frame where the click is visible. Files: `test/alignment.md` ### Done when - [ ] The cursor sample count far exceeds the video frame count for the same duration - [ ] A logged click aligns with the visible click within one frame - [ ] The flat video opens in QuickTime without a repair prompt - [ ] The recording is at native pixel resolution, not scaled down ### Watch out - Do not apply zoom while recording. A zoomed recording throws away the pixels outside the zoom, so a badly chosen zoom is permanent and a re-render is impossible. Record everything, decide later. - Sample the cursor far more often than the video frame rate. Sampling per frame gives you a stair-stepped path that produces visibly jerky motion once you smooth and amplify it. ## M2 · Generating zoom from the track The zoom keyframes are computed from what the cursor did, and they are good enough that you rarely touch them. ### Steps 1. Detect intent regions Cluster click events in time and space. Treat sustained dwell within a small radius as intent too, because reading and typing matter as much as clicking. Files: `Intent.swift` 2. Emit keyframes with lead and lag Zoom in slightly before the action so the viewer arrives with it, and hold briefly after so the result is visible before pulling out. Files: `Keyframes.swift` 3. Suppress noise A minimum interval between zooms, a minimum dwell to qualify, and a maximum zoom rate. These three numbers are the whole feel of the product. Files: `Keyframes.swift` ### Done when - [ ] Clicked fields get zooms and cursor transits between them do not - [ ] No two zoom-ins are closer than the configured minimum interval - [ ] A recording with no clicks produces no keyframes - [ ] Keyframes are stored separately from the video and can be regenerated ### Watch out - Following the cursor continuously produces motion sickness. The cursor moves constantly and most of that movement means nothing. Zoom on intent, which is clicks and sustained dwell, not on position. ## M3 · Easing and framing The motion looks designed rather than computed, and the frame never shows the edge of the source. ### Steps 1. Implement cubic easing Ease in and out between keyframe pairs. Compare against linear side by side once; you will not go back. Files: `Easing.swift` 2. Clamp the centre inside the source Compute the visible rectangle at the current scale and clamp its centre so the rectangle stays inside the source. Test the corners explicitly. Files: `Framing.swift` 3. Composite padding, corners and background A Core Image pipeline: scale and translate the source, round the corners, and draw over a background colour or gradient. Files: `Composite.swift` ### Done when - [ ] A zoom at a screen corner never reveals area outside the source - [ ] Motion eases rather than moving at constant speed - [ ] Padding and corner radius change on re-render without re-recording - [ ] The background renders behind the rounded corners with no fringing ### Watch out - Clamp the centre point, not the offset. Clamping after computing the offset lets the frame drift past the edge of the source at high zoom levels and shows background where video should be, usually at exactly the moment of most interest. ## M4 · Rendering and export The finished video comes out at the size you asked for, and rendering is not slower than watching. ### Steps 1. Keep the composite on the GPU One Metal-backed CIContext reused across frames. Creating a context per frame is the single most common reason exports crawl. Files: `Renderer.swift` 2. Add export presets with exact dimensions Named presets with fixed pixel sizes. Assert the output dimensions rather than trusting the scale arithmetic. Files: `Presets.swift` 3. Carry audio through unchanged Pass the original audio track through rather than re-encoding it, and confirm sync at the end of a long export. Files: `Renderer.swift` ### Done when - [ ] A two minute recording exports in under two minutes - [ ] Each preset produces exactly the pixel dimensions it names - [ ] Audio is in sync at the end of the exported file - [ ] Cancelling an export leaves no partial file behind ### Watch out - Compositing on the CPU makes a two minute recording take longer than two minutes to export, which turns a pleasant tool into an errand. Keep the pipeline on the GPU through Core Image with a Metal-backed context, and measure the export against real-time before adding any more effects. ## M5 · Trim and manual correction The one zoom the generator got wrong takes ten seconds to fix. ### Steps 1. Build the timeline Video thumbnails, click markers and keyframe handles on one scrubbable track. Files: `Timeline.swift` 2. Make keyframes editable Drag to move, delete, insert. Preview updates from the source, which is why nothing was baked in. Files: `Timeline.swift` 3. Protect manual edits Mark keyframes as generated or manual, and warn before regeneration discards manual ones. Files: `Keyframes.swift` ### Done when - [ ] Deleting a keyframe removes the zoom from preview and export - [ ] A manually added keyframe produces a zoom at that time - [ ] Trimming does not desynchronise keyframes from content - [ ] Regeneration warns before discarding manual edits ### Watch out - Automatic generation will be wrong occasionally, and a tool with no manual override forces a re-record for a five second mistake. The keyframes are data, so make them editable from the beginning rather than treating generation as final. ## M6 · Signing, notarization and updates (production only) Other people can install it and receive fixes. ### Steps 1. Sign and notarize Hardened runtime, Developer ID, notarytool, stapler. Confirm the usage description strings are present or the prompts never appear. ```sh xcrun notarytool submit Studio.zip --keychain-profile AC --wait xcrun stapler staple Studio.app ``` 2. Publish updates with Sparkle EdDSA key pair, private half outside the repository, appcast over HTTPS. Files: `appcast.xml` 3. Document the permissions Which permission enables which feature, and what refusing each one costs. Files: `README.md` ### Done when - [ ] spctl reports the build accepted - [ ] A fresh Mac opens it with no warning - [ ] Permissions are requested once and the recording then works - [ ] An appcast update installs and relaunches ### Watch out - Signing invalidates the Screen Recording grant testers gave the unsigned build, and macOS will not prompt again. Say so in the release notes with the removal steps.
# Operations · Screen Studio ## Backup Flat recordings plus their sidecar project files. The rendered exports are regenerable and do not need backing up, which is the practical benefit of never baking effects in. ## Restore Copy the recordings directory back. A project file without its recording is useless, so keep them together and say so in the interface when a project's source is missing. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Watch export speed against real time and dropped frame counts from the capture stream. A stream that silently drops frames produces a video that feels wrong without anything obviously being broken. ## Incident checklist A macOS update changing capture behaviour is the recurring risk. Keep the previous signed build available and set a minimum supported version in the appcast. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] The three fixture recordings still produce sensible keyframes after any generation change - [ ] A two minute export completes in under two minutes - [ ] Corner zooms never reveal area outside the source - [ ] Audio sync is verified at the end of a long export - [ ] spctl accepts the notarized build on a Mac that never ran it ## Launch constraint Do not market omitted Screen Studio capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Where flat recordings and project files are written. These are large, so pick a disk with room. RECORDINGS_DIR=~/Movies/ScreenStudio # Required. Cursor sampling rate. Must be well above the video frame rate or the motion looks stepped. CURSOR_SAMPLE_HZ=120 # Required. Minimum time between zoom-ins. Raising this calms a busy video. ZOOM_MIN_INTERVAL_MS=2500 # Required. How long the cursor must stay in an area to count as intent. Lowering it makes the video restless. ZOOM_MIN_DWELL_MS=600 # Required. How far in the automatic zoom will go. Beyond about two and a half the source pixels start to show. ZOOM_MAX_SCALE=2.2 # Optional. Product phase only. SPARKLE_FEED_URL=https://example.com/appcast.xml
$ choose a build depth, inspect the files, then open the complete pack in your agent
People still pay for Screen Studio because the recording itself is easy; the subscription removes upload, transcoding, sharing, organization, and support work. The recurring cost buys OS permissions, codecs, rendering performance, uploads, storage, links, and viewer analytics, not just the visible interface.
xexceptional macOS capture polish, motion design, and ongoing compatibility
xinstant hosted sharing
xviewer analytics
xteam libraries
xcross-device capture
Don't feel like building it? These folks already made it free.
no votes, no pay-to-list · just what's real
Screen Studio pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| screen studio | $29/user | $9/user | All recording and editing features; share links; exports up to 4K/60fps |
free tierno permanent free tier; a download/trial is offered but its numeric limits and duration are not publicly stated
billingmonthly + annual; annual rate displayed as $9/month
hidden costsCurrent access is subscription-based rather than a new lifetime license; the public page does not state storage or fair-use caps for hosted share links.
verified 2026-08-14 · source ↗
Vibecode Screen Studio
Yes. A competent AI coding agent (Claude Code, Codex, Cursor) can build a usable personal Screen Studio replacement in one session with the prompt on this page. It runs on your own machine or server with no subscription.
How much does Screen Studio cost?
Screen Studio's pricing is usage-based or varies by plan · Sold primarily as a one-time license; priceMonthly is null by design..
What do I lose by replacing Screen Studio?
Honestly: exceptional macOS capture polish, motion design, and ongoing compatibility; instant hosted sharing; viewer analytics; team libraries; cross-device capture. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Screen Studio?
Yes: Recordly (Auto-zoom demos with styled frames and installers; young enough that audio still occasionally picks a fight.) The prompt is for when you want it exactly your way.