Vibecode Parabola
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The visible visual data automation loop is buildable, but a credible replacement needs more than the first screen. Parabola earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Parabola. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Parabola · indie build A step-by-step data pipeline whose hard problems are reading tabular files without silently changing the values, keeping every step deterministic and previewable, and running on a schedule against APIs that push back without ever writing half a result. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | TypeScript on Node 22, one worker process plus one web process | the schedule has to fire whether or not a browser is open | | Storage | PostgreSQL for flow definitions and run history, files on disk for step outputs | row-level data belongs in files; run metadata belongs in a database you query | | Queue | BullMQ on Redis with one active run per flow enforced by a lock | two overlapping runs of the same flow is how a destination gets written twice | | Steps | Pure functions from one table to one table, output materialised per step | a step you cannot preview is a step you cannot debug at two in the morning | | Types | Declared on the source, never inferred per row | inference is what turns an order number into a smaller order number | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the worker and the web process. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally, a few pounds a month managed - Why: Holds flow definitions and run history, which you will query far more than you expect once something fails at three in the morning. - Get it: Docker, your package manager, or a managed instance. - Verify: psql -c 'select version()' connects - [ ] **Redis for the queue and the per-flow lock** · free locally - Why: The lock that stops two runs of one flow overlapping lives here, and overlapping runs are the duplicate-write bug. - Get it: Docker or your package manager. - Verify: redis-cli ping returns PONG - [ ] **API credentials for the one or two services you actually use** · free for most, some charge per call - Why: The whole point is moving your own data, and a pipeline with no real source is a demo. - Get it: Create a scoped token in each service, with read on the source and the narrowest write on the destination. - Verify: A single authenticated request succeeds from the command line - [ ] **The published rate limit for each service, and the date you read it** · free - Why: The request rate you choose is only defensible if it comes from the vendor's current page, and these change. - Get it: Read each vendor's rate limit documentation now and write down the figure with today's date. - Verify: Each configured rate cites a source and a date in the README - [ ] **A real messy file from your own data** · free - Why: Leading zeros, embedded newlines and an unexpected encoding are the failures that matter, and clean fixtures never show them. - Get it: Export something from the system you actually use, without tidying it up first. - Verify: It parses with a declared schema and its bad rows land in row_errors ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A connector library. - Verified OAuth applications. - Multi-user workspaces and approvals. - Automatic schema drift handling. - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors - OAuth app verification If one of those is essential to you, that is the reason to keep paying for Parabola, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Parabola The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a visual data pipeline that runs on a schedule and can account for every row it touched to replace Parabola. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: TypeScript on Node 22, one worker process plus one web process · the schedule has to fire whether or not a browser is open - Storage: PostgreSQL for flow definitions and run history, files on disk for step outputs · row-level data belongs in files; run metadata belongs in a database you query - Queue: BullMQ on Redis with one active run per flow enforced by a lock · two overlapping runs of the same flow is how a destination gets written twice - Steps: Pure functions from one table to one table, output materialised per step · a step you cannot preview is a step you cannot debug at two in the morning - Types: Declared on the source, never inferred per row · inference is what turns an order number into a smaller order number ### Data model (create this before Phase 1) - flows(id, name, current_version, schedule_cron, timezone, enabled, created_at) - flow_versions(flow_id, version, graph_json, created_at) · runs reference the version they executed, so an edit does not rewrite history - runs(id, flow_id, flow_version, trigger, state, started_at, finished_at, rows_in, rows_out, error) · trigger is schedule, manual or replay - step_runs(run_id, step_id, ordinal, state, rows_in, rows_out, rows_filtered, rows_errored, output_path, duration_ms) - row_errors(run_id, step_id, row_ordinal, column_name, raw_value, reason) · the raw value is kept exactly as it arrived - destinations(id, kind, credential_ref, write_mode, idempotency_column, last_run_id) · write_mode is replace, append or upsert - Every step run satisfies rows_in equals rows_out plus rows_filtered plus rows_errored, and the suite asserts it, because a pipeline that quietly loses rows is worse than one that fails. ### Phase 1 · Reading a file without changing it Build: Parse delimited files to the letter of RFC 4180: records separated by CRLF, fields containing a comma, a quote or a line break wrapped in double quotes, and an inner double quote escaped by doubling it. The format carries no type information and no encoding declaration, so take both from a declared schema per column plus a declared encoding, and send any value that does not parse to the row_errors table with its raw text intact. Detect and reject a byte order mark rather than folding it into the first column name. Trap: Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. Done when: A column declared as text preserves leading zeros through the whole pipeline, tested with a fixture. A quoted field containing a comma, a newline and a doubled quote round-trips exactly. A value that does not parse against its declared type lands in row_errors with its raw text and does not silently become null. A file whose encoding differs from the declared one fails loudly at the source step rather than producing replacement characters downstream. Do not build yet: No steps beyond the source, no schedule, no destinations and no API sources. ### Phase 2 · Steps you can look inside Build: Implement filter, select, rename, derive, join, group and sort as pure functions from one table to one table. Write each step's output to its own file, record rows in, out, filtered and errored in step_runs, and show a sample of the output beside the step in the editor. Make the graph an ordered list of steps with a single source, so evaluation order is not a question. Trap: Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. Done when: Every step run records rows in, out, filtered and errored, and the suite asserts the four numbers reconcile. Clicking any step shows a sample of its actual output from that run, not a recomputation. A join between a text key and an integer key fails with both types named rather than matching nothing. Running the same flow twice over the same input produces byte-identical step outputs. Do not build yet: No schedule, no API sources or destinations, no retries. ### Phase 3 · A schedule that does not run twice Build: Schedule each flow with a cron expression plus a named timezone, computing each fire time from that zone's rules for that date. Take a lock per flow so a slow run never overlaps the next tick, and record the skip rather than queueing it. On restart, mark any run left in progress as interrupted and require an explicit replay instead of resuming blindly. Trap: Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. Done when: A daily schedule fires once per local day across both clock change days, tested against hand-computed times in a zone that observes them. A run still executing when the next tick arrives causes a recorded skip rather than a second run, tested by holding a lock. Killing the worker mid-run leaves the run marked interrupted, and it is never silently resumed. A schedule landing in the missing hour resolves by a rule stated in the README. Do not build yet: No API sources or destinations, no rate limit handling, no export. ### Phase 4 · APIs that push back Build: Add an HTTP source that pages through results and an HTTP destination that writes in batches. Read each vendor's current published rate limit before choosing a request rate, honour a 429 by waiting the interval the response asks for rather than a guess, and back off on repeated failures. Stage the full result and only then write to the destination, keyed by an idempotency column so a retried batch updates rather than duplicates. Trap: Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. Done when: A source paging through a fake API with a 429 on the fourth page waits the interval the response specifies and completes, tested. A destination write that fails partway leaves the destination untouched, because the write happens only after the full result is staged. A replayed run against the same staged data produces the same destination state, tested by running it twice. The configured request rate is recorded with the date the vendor's published limit was read. Do not build yet: No replay interface, no export and no cost ceilings yet. ### Phase 5 · Reading the failure and taking the data out Build: A run page showing each step with its row counts, its errored rows with raw values, and the elapsed time. Replay a run from any step using the stored output of the previous one. Export a flow definition and any step output. Cap run duration, rows and destination writes, and stop a run that exceeds a cap rather than continuing. Alert once per failing flow, not once per run. Trap: A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration. Done when: The run page shows every step's four row counts and lists errored rows with their raw values. Replaying from step four uses the stored output of step three rather than re-reading the source, tested. A run exceeding the configured row or duration ceiling stops and is marked as capped. A flow failing five nights running produces one alert on the transition plus one on recovery, tested with a fake clock. Do not build yet: Nothing after this. ### Out of scope (and why) - Hundreds of maintained connectors, which are the actual subscription; build the two or three sources you personally use and an HTTP step for the rest. - Verified OAuth applications for consumer platforms, which need a review process and a privacy policy before a single row moves. - Multi-user workspaces, sharing and approvals, which turn every run into a permissions question. - Schema drift handling across a live vendor API, which is a maintenance commitment rather than a feature you finish. ### README must contain - That column types are declared on the source and never inferred, with the leading zero case named - That every step materialises its output and the four row counts reconcile, asserted in the suite - That schedules are cron plus a named timezone, and what happens on both clock change days - That destination writes happen only after the full result is staged, keyed on an identifier from the data - Which vendor rate limits were read, on what date, and where to recheck them ===== AGENTS.md ===== # Agent instructions · Parabola indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: TypeScript on Node 22, one worker process plus one web process, PostgreSQL for flow definitions and run history, files on disk for step outputs, BullMQ on Redis with one active run per flow enforced by a lock, Pure functions from one table to one table, output materialised per step, Declared on the source, never inferred per row. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. ===== BUILD_PLAN.md ===== # Build plan · Parabola A step-by-step data pipeline whose hard problems are reading tabular files without silently changing the values, keeping every step deterministic and previewable, and running on a schedule against APIs that push back without ever writing half a result. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Reading a file without changing it A tabular source is parsed to declared types, and anything that does not fit becomes a visible error rather than a guess. ### Steps 1. Declare the schema on the source One declared type per column, applied to every row, with no per-row inference anywhere. Files: `src/steps/source-csv.ts`, `src/schema.ts` 2. Parse quoting and line breaks by the specification Hand-rolled splitting on commas breaks on the first field containing one. Files: `src/csv/parse.ts`, `tests/csv-quoting.test.ts` 3. Route bad values to row_errors with the raw text The raw text is the only thing that lets someone fix the source file. Files: `src/steps/source-csv.ts`, `tests/leading-zeros.test.ts` ### Done when - [ ] Leading zeros survive as declared text - [ ] Quoted commas, newlines and doubled quotes round-trip - [ ] Unparseable values reach row_errors with raw text - [ ] A wrong declared encoding fails at the source step ### Watch out - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. ## Phase 2 · Steps you can look inside Every step materialises its output and accounts for every row that entered it. ### Steps 1. Materialise each step's output to its own file The debugging story for a pipeline is being able to see the table between any two steps. Files: `src/engine/run.ts`, `src/engine/materialise.ts` 2. Reconcile the row counts in the suite Rows in equals rows out plus filtered plus errored, for every step, every time. Files: `src/engine/step-run.ts`, `tests/row-accounting.test.ts` 3. Fail a join on mismatched key types A join that matches nothing looks identical to a filter that worked. Files: `src/steps/join.ts`, `tests/join-types.test.ts` ### Done when - [ ] The four row counts reconcile for every step - [ ] Each step shows a sample of that run's real output - [ ] A mismatched join key fails with both types named - [ ] A repeated run produces identical step outputs ### Watch out - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. ## Phase 3 · A schedule that does not run twice Scheduled runs fire in a named timezone, never overlap, and survive a restart mid-run. ### Steps 1. Compute fire times from the zone's rules Not by adding a fixed interval, which is wrong twice a year. Files: `src/schedule/next.ts`, `tests/dst.test.ts` 2. Take a per-flow lock and record skips A skipped tick you can see beats an overlapping run you cannot. Files: `src/queue/lock.ts` 3. Mark interrupted runs and require a replay Automatic resumption is how half a file gets written twice. Files: `src/engine/recover.ts` ### Done when - [ ] A daily schedule fires once per local day on both transitions - [ ] An overlapping tick is recorded as a skip - [ ] An interrupted run is never silently resumed - [ ] The missing hour rule is stated and tested ### Watch out - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. ## Phase 4 · APIs that push back A paged API source finishes or fails cleanly, and a destination is never half written. ### Steps 1. Honour the retry interval the response gives you Guessing a backoff when the server told you the number is how an account gets throttled for a day. Files: `src/http/client.ts`, `tests/rate-limit.test.ts` 2. Stage the whole result before writing anything A half-written destination is the failure mode with no clean recovery. Files: `src/destinations/stage.ts` 3. Key destination writes on an identifier from the data So a retry is an update and not a duplicate row. Files: `src/destinations/write.ts`, `tests/idempotent-write.test.ts` ### Done when - [ ] A 429 is honoured by waiting the stated interval - [ ] A failed write leaves the destination untouched - [ ] Running the same staged result twice is idempotent - [ ] The chosen rate cites when the limit was read ### Watch out - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. ## Not in this build - A connector library. - Verified OAuth applications. - Multi-user workspaces and approvals. - Automatic schema drift handling. ## After v1, if you want it - A dry run mode that executes every step and stops before the destination write - Column-level lineage, which the materialised outputs already make possible - A second destination kind, once one has run for a month without a duplicate ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Flows, versions, runs and row errors. DATABASE_URL=postgres://localhost:5432/pipelines # Required. The queue and the per-flow run lock. REDIS_URL=redis://localhost:6379 # Required. Materialised step outputs. Back this up with the database or a replay is impossible. DATA_DIR=./data/steps # Required. How long step outputs are kept for replay before deletion. STEP_OUTPUT_RETENTION_DAYS=30 # Required. A hard ceiling. A run that exceeds it stops and is marked capped. MAX_ROWS_PER_RUN=500000 # Required. The other ceiling, for the source that got slow rather than large. MAX_RUN_MINUTES=30 # Required · secret. Read scope only on the service you pull from. SOURCE_API_TOKEN=sk-...redacted # Optional · secret. The narrowest write scope on the destination, separate from the read token. DESTINATION_API_TOKEN=sk-...redacted
You are building a lean indie version of Parabola. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Parabola · indie build A step-by-step data pipeline whose hard problems are reading tabular files without silently changing the values, keeping every step deterministic and previewable, and running on a schedule against APIs that push back without ever writing half a result. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | TypeScript on Node 22, one worker process plus one web process | the schedule has to fire whether or not a browser is open | | Storage | PostgreSQL for flow definitions and run history, files on disk for step outputs | row-level data belongs in files; run metadata belongs in a database you query | | Queue | BullMQ on Redis with one active run per flow enforced by a lock | two overlapping runs of the same flow is how a destination gets written twice | | Steps | Pure functions from one table to one table, output materialised per step | a step you cannot preview is a step you cannot debug at two in the morning | | Types | Declared on the source, never inferred per row | inference is what turns an order number into a smaller order number | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the worker and the web process. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally, a few pounds a month managed - Why: Holds flow definitions and run history, which you will query far more than you expect once something fails at three in the morning. - Get it: Docker, your package manager, or a managed instance. - Verify: psql -c 'select version()' connects - [ ] **Redis for the queue and the per-flow lock** · free locally - Why: The lock that stops two runs of one flow overlapping lives here, and overlapping runs are the duplicate-write bug. - Get it: Docker or your package manager. - Verify: redis-cli ping returns PONG - [ ] **API credentials for the one or two services you actually use** · free for most, some charge per call - Why: The whole point is moving your own data, and a pipeline with no real source is a demo. - Get it: Create a scoped token in each service, with read on the source and the narrowest write on the destination. - Verify: A single authenticated request succeeds from the command line - [ ] **The published rate limit for each service, and the date you read it** · free - Why: The request rate you choose is only defensible if it comes from the vendor's current page, and these change. - Get it: Read each vendor's rate limit documentation now and write down the figure with today's date. - Verify: Each configured rate cites a source and a date in the README - [ ] **A real messy file from your own data** · free - Why: Leading zeros, embedded newlines and an unexpected encoding are the failures that matter, and clean fixtures never show them. - Get it: Export something from the system you actually use, without tidying it up first. - Verify: It parses with a declared schema and its bad rows land in row_errors ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A connector library. - Verified OAuth applications. - Multi-user workspaces and approvals. - Automatic schema drift handling. - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors - OAuth app verification If one of those is essential to you, that is the reason to keep paying for Parabola, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Parabola The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a visual data pipeline that runs on a schedule and can account for every row it touched to replace Parabola. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: TypeScript on Node 22, one worker process plus one web process · the schedule has to fire whether or not a browser is open - Storage: PostgreSQL for flow definitions and run history, files on disk for step outputs · row-level data belongs in files; run metadata belongs in a database you query - Queue: BullMQ on Redis with one active run per flow enforced by a lock · two overlapping runs of the same flow is how a destination gets written twice - Steps: Pure functions from one table to one table, output materialised per step · a step you cannot preview is a step you cannot debug at two in the morning - Types: Declared on the source, never inferred per row · inference is what turns an order number into a smaller order number ### Data model (create this before Phase 1) - flows(id, name, current_version, schedule_cron, timezone, enabled, created_at) - flow_versions(flow_id, version, graph_json, created_at) · runs reference the version they executed, so an edit does not rewrite history - runs(id, flow_id, flow_version, trigger, state, started_at, finished_at, rows_in, rows_out, error) · trigger is schedule, manual or replay - step_runs(run_id, step_id, ordinal, state, rows_in, rows_out, rows_filtered, rows_errored, output_path, duration_ms) - row_errors(run_id, step_id, row_ordinal, column_name, raw_value, reason) · the raw value is kept exactly as it arrived - destinations(id, kind, credential_ref, write_mode, idempotency_column, last_run_id) · write_mode is replace, append or upsert - Every step run satisfies rows_in equals rows_out plus rows_filtered plus rows_errored, and the suite asserts it, because a pipeline that quietly loses rows is worse than one that fails. ### Phase 1 · Reading a file without changing it Build: Parse delimited files to the letter of RFC 4180: records separated by CRLF, fields containing a comma, a quote or a line break wrapped in double quotes, and an inner double quote escaped by doubling it. The format carries no type information and no encoding declaration, so take both from a declared schema per column plus a declared encoding, and send any value that does not parse to the row_errors table with its raw text intact. Detect and reject a byte order mark rather than folding it into the first column name. Trap: Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. Done when: A column declared as text preserves leading zeros through the whole pipeline, tested with a fixture. A quoted field containing a comma, a newline and a doubled quote round-trips exactly. A value that does not parse against its declared type lands in row_errors with its raw text and does not silently become null. A file whose encoding differs from the declared one fails loudly at the source step rather than producing replacement characters downstream. Do not build yet: No steps beyond the source, no schedule, no destinations and no API sources. ### Phase 2 · Steps you can look inside Build: Implement filter, select, rename, derive, join, group and sort as pure functions from one table to one table. Write each step's output to its own file, record rows in, out, filtered and errored in step_runs, and show a sample of the output beside the step in the editor. Make the graph an ordered list of steps with a single source, so evaluation order is not a question. Trap: Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. Done when: Every step run records rows in, out, filtered and errored, and the suite asserts the four numbers reconcile. Clicking any step shows a sample of its actual output from that run, not a recomputation. A join between a text key and an integer key fails with both types named rather than matching nothing. Running the same flow twice over the same input produces byte-identical step outputs. Do not build yet: No schedule, no API sources or destinations, no retries. ### Phase 3 · A schedule that does not run twice Build: Schedule each flow with a cron expression plus a named timezone, computing each fire time from that zone's rules for that date. Take a lock per flow so a slow run never overlaps the next tick, and record the skip rather than queueing it. On restart, mark any run left in progress as interrupted and require an explicit replay instead of resuming blindly. Trap: Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. Done when: A daily schedule fires once per local day across both clock change days, tested against hand-computed times in a zone that observes them. A run still executing when the next tick arrives causes a recorded skip rather than a second run, tested by holding a lock. Killing the worker mid-run leaves the run marked interrupted, and it is never silently resumed. A schedule landing in the missing hour resolves by a rule stated in the README. Do not build yet: No API sources or destinations, no rate limit handling, no export. ### Phase 4 · APIs that push back Build: Add an HTTP source that pages through results and an HTTP destination that writes in batches. Read each vendor's current published rate limit before choosing a request rate, honour a 429 by waiting the interval the response asks for rather than a guess, and back off on repeated failures. Stage the full result and only then write to the destination, keyed by an idempotency column so a retried batch updates rather than duplicates. Trap: Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. Done when: A source paging through a fake API with a 429 on the fourth page waits the interval the response specifies and completes, tested. A destination write that fails partway leaves the destination untouched, because the write happens only after the full result is staged. A replayed run against the same staged data produces the same destination state, tested by running it twice. The configured request rate is recorded with the date the vendor's published limit was read. Do not build yet: No replay interface, no export and no cost ceilings yet. ### Phase 5 · Reading the failure and taking the data out Build: A run page showing each step with its row counts, its errored rows with raw values, and the elapsed time. Replay a run from any step using the stored output of the previous one. Export a flow definition and any step output. Cap run duration, rows and destination writes, and stop a run that exceeds a cap rather than continuing. Alert once per failing flow, not once per run. Trap: A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration. Done when: The run page shows every step's four row counts and lists errored rows with their raw values. Replaying from step four uses the stored output of step three rather than re-reading the source, tested. A run exceeding the configured row or duration ceiling stops and is marked as capped. A flow failing five nights running produces one alert on the transition plus one on recovery, tested with a fake clock. Do not build yet: Nothing after this. ### Out of scope (and why) - Hundreds of maintained connectors, which are the actual subscription; build the two or three sources you personally use and an HTTP step for the rest. - Verified OAuth applications for consumer platforms, which need a review process and a privacy policy before a single row moves. - Multi-user workspaces, sharing and approvals, which turn every run into a permissions question. - Schema drift handling across a live vendor API, which is a maintenance commitment rather than a feature you finish. ### README must contain - That column types are declared on the source and never inferred, with the leading zero case named - That every step materialises its output and the four row counts reconcile, asserted in the suite - That schedules are cron plus a named timezone, and what happens on both clock change days - That destination writes happen only after the full result is staged, keyed on an identifier from the data - Which vendor rate limits were read, on what date, and where to recheck them ===== AGENTS.md ===== # Agent instructions · Parabola indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: TypeScript on Node 22, one worker process plus one web process, PostgreSQL for flow definitions and run history, files on disk for step outputs, BullMQ on Redis with one active run per flow enforced by a lock, Pure functions from one table to one table, output materialised per step, Declared on the source, never inferred per row. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. ===== BUILD_PLAN.md ===== # Build plan · Parabola A step-by-step data pipeline whose hard problems are reading tabular files without silently changing the values, keeping every step deterministic and previewable, and running on a schedule against APIs that push back without ever writing half a result. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Reading a file without changing it A tabular source is parsed to declared types, and anything that does not fit becomes a visible error rather than a guess. ### Steps 1. Declare the schema on the source One declared type per column, applied to every row, with no per-row inference anywhere. Files: `src/steps/source-csv.ts`, `src/schema.ts` 2. Parse quoting and line breaks by the specification Hand-rolled splitting on commas breaks on the first field containing one. Files: `src/csv/parse.ts`, `tests/csv-quoting.test.ts` 3. Route bad values to row_errors with the raw text The raw text is the only thing that lets someone fix the source file. Files: `src/steps/source-csv.ts`, `tests/leading-zeros.test.ts` ### Done when - [ ] Leading zeros survive as declared text - [ ] Quoted commas, newlines and doubled quotes round-trip - [ ] Unparseable values reach row_errors with raw text - [ ] A wrong declared encoding fails at the source step ### Watch out - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. ## Phase 2 · Steps you can look inside Every step materialises its output and accounts for every row that entered it. ### Steps 1. Materialise each step's output to its own file The debugging story for a pipeline is being able to see the table between any two steps. Files: `src/engine/run.ts`, `src/engine/materialise.ts` 2. Reconcile the row counts in the suite Rows in equals rows out plus filtered plus errored, for every step, every time. Files: `src/engine/step-run.ts`, `tests/row-accounting.test.ts` 3. Fail a join on mismatched key types A join that matches nothing looks identical to a filter that worked. Files: `src/steps/join.ts`, `tests/join-types.test.ts` ### Done when - [ ] The four row counts reconcile for every step - [ ] Each step shows a sample of that run's real output - [ ] A mismatched join key fails with both types named - [ ] A repeated run produces identical step outputs ### Watch out - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. ## Phase 3 · A schedule that does not run twice Scheduled runs fire in a named timezone, never overlap, and survive a restart mid-run. ### Steps 1. Compute fire times from the zone's rules Not by adding a fixed interval, which is wrong twice a year. Files: `src/schedule/next.ts`, `tests/dst.test.ts` 2. Take a per-flow lock and record skips A skipped tick you can see beats an overlapping run you cannot. Files: `src/queue/lock.ts` 3. Mark interrupted runs and require a replay Automatic resumption is how half a file gets written twice. Files: `src/engine/recover.ts` ### Done when - [ ] A daily schedule fires once per local day on both transitions - [ ] An overlapping tick is recorded as a skip - [ ] An interrupted run is never silently resumed - [ ] The missing hour rule is stated and tested ### Watch out - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. ## Phase 4 · APIs that push back A paged API source finishes or fails cleanly, and a destination is never half written. ### Steps 1. Honour the retry interval the response gives you Guessing a backoff when the server told you the number is how an account gets throttled for a day. Files: `src/http/client.ts`, `tests/rate-limit.test.ts` 2. Stage the whole result before writing anything A half-written destination is the failure mode with no clean recovery. Files: `src/destinations/stage.ts` 3. Key destination writes on an identifier from the data So a retry is an update and not a duplicate row. Files: `src/destinations/write.ts`, `tests/idempotent-write.test.ts` ### Done when - [ ] A 429 is honoured by waiting the stated interval - [ ] A failed write leaves the destination untouched - [ ] Running the same staged result twice is idempotent - [ ] The chosen rate cites when the limit was read ### Watch out - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. ## Not in this build - A connector library. - Verified OAuth applications. - Multi-user workspaces and approvals. - Automatic schema drift handling. ## After v1, if you want it - A dry run mode that executes every step and stops before the destination write - Column-level lineage, which the materialised outputs already make possible - A second destination kind, once one has run for a month without a duplicate ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Flows, versions, runs and row errors. DATABASE_URL=postgres://localhost:5432/pipelines # Required. The queue and the per-flow run lock. REDIS_URL=redis://localhost:6379 # Required. Materialised step outputs. Back this up with the database or a replay is impossible. DATA_DIR=./data/steps # Required. How long step outputs are kept for replay before deletion. STEP_OUTPUT_RETENTION_DAYS=30 # Required. A hard ceiling. A run that exceeds it stops and is marked capped. MAX_ROWS_PER_RUN=500000 # Required. The other ceiling, for the source that got slow rather than large. MAX_RUN_MINUTES=30 # Required · secret. Read scope only on the service you pull from. SOURCE_API_TOKEN=sk-...redacted # Optional · secret. The narrowest write scope on the destination, separate from the read token. DESTINATION_API_TOKEN=sk-...redacted
You are building a production product version of Parabola. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Parabola · product brief ## Problem The visible visual data automation loop is buildable, but a credible replacement needs more than the first screen. Parabola earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A pipeline that reads your files without changing the values in them, shows you the table between any two steps, runs on a schedule without ever overlapping itself, and never leaves a destination half written. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - hosted PostgreSQL and Redis - API credentials for the two chosen services - Node.js 22 ## Explicit non-goals for v1 - A connector library. - Verified OAuth applications. - Multi-user workspaces and approvals. - Automatic schema drift handling. - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors - OAuth app verification ## Success criteria - Leading zeros are confirmed to survive as declared text - The four row counts are confirmed to reconcile per step - An overlapping tick is confirmed to be skipped, not queued - A failed write is confirmed to leave the destination untouched - A repeatedly failing flow is confirmed to alert once, not nightly ===== BRIEF.md ===== # Build brief · Parabola The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a visual data pipeline that runs on a schedule and can account for every row it touched to replace Parabola. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: TypeScript on Node 22, one worker process plus one web process · the schedule has to fire whether or not a browser is open - Storage: PostgreSQL for flow definitions and run history, files on disk for step outputs · row-level data belongs in files; run metadata belongs in a database you query - Queue: BullMQ on Redis with one active run per flow enforced by a lock · two overlapping runs of the same flow is how a destination gets written twice - Steps: Pure functions from one table to one table, output materialised per step · a step you cannot preview is a step you cannot debug at two in the morning - Types: Declared on the source, never inferred per row · inference is what turns an order number into a smaller order number ### Data model (create this before Phase 1) - flows(id, name, current_version, schedule_cron, timezone, enabled, created_at) - flow_versions(flow_id, version, graph_json, created_at) · runs reference the version they executed, so an edit does not rewrite history - runs(id, flow_id, flow_version, trigger, state, started_at, finished_at, rows_in, rows_out, error) · trigger is schedule, manual or replay - step_runs(run_id, step_id, ordinal, state, rows_in, rows_out, rows_filtered, rows_errored, output_path, duration_ms) - row_errors(run_id, step_id, row_ordinal, column_name, raw_value, reason) · the raw value is kept exactly as it arrived - destinations(id, kind, credential_ref, write_mode, idempotency_column, last_run_id) · write_mode is replace, append or upsert - Every step run satisfies rows_in equals rows_out plus rows_filtered plus rows_errored, and the suite asserts it, because a pipeline that quietly loses rows is worse than one that fails. ### Phase 1 · Reading a file without changing it Build: Parse delimited files to the letter of RFC 4180: records separated by CRLF, fields containing a comma, a quote or a line break wrapped in double quotes, and an inner double quote escaped by doubling it. The format carries no type information and no encoding declaration, so take both from a declared schema per column plus a declared encoding, and send any value that does not parse to the row_errors table with its raw text intact. Detect and reject a byte order mark rather than folding it into the first column name. Trap: Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. Done when: A column declared as text preserves leading zeros through the whole pipeline, tested with a fixture. A quoted field containing a comma, a newline and a doubled quote round-trips exactly. A value that does not parse against its declared type lands in row_errors with its raw text and does not silently become null. A file whose encoding differs from the declared one fails loudly at the source step rather than producing replacement characters downstream. Do not build yet: No steps beyond the source, no schedule, no destinations and no API sources. ### Phase 2 · Steps you can look inside Build: Implement filter, select, rename, derive, join, group and sort as pure functions from one table to one table. Write each step's output to its own file, record rows in, out, filtered and errored in step_runs, and show a sample of the output beside the step in the editor. Make the graph an ordered list of steps with a single source, so evaluation order is not a question. Trap: Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. Done when: Every step run records rows in, out, filtered and errored, and the suite asserts the four numbers reconcile. Clicking any step shows a sample of its actual output from that run, not a recomputation. A join between a text key and an integer key fails with both types named rather than matching nothing. Running the same flow twice over the same input produces byte-identical step outputs. Do not build yet: No schedule, no API sources or destinations, no retries. ### Phase 3 · A schedule that does not run twice Build: Schedule each flow with a cron expression plus a named timezone, computing each fire time from that zone's rules for that date. Take a lock per flow so a slow run never overlaps the next tick, and record the skip rather than queueing it. On restart, mark any run left in progress as interrupted and require an explicit replay instead of resuming blindly. Trap: Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. Done when: A daily schedule fires once per local day across both clock change days, tested against hand-computed times in a zone that observes them. A run still executing when the next tick arrives causes a recorded skip rather than a second run, tested by holding a lock. Killing the worker mid-run leaves the run marked interrupted, and it is never silently resumed. A schedule landing in the missing hour resolves by a rule stated in the README. Do not build yet: No API sources or destinations, no rate limit handling, no export. ### Phase 4 · APIs that push back Build: Add an HTTP source that pages through results and an HTTP destination that writes in batches. Read each vendor's current published rate limit before choosing a request rate, honour a 429 by waiting the interval the response asks for rather than a guess, and back off on repeated failures. Stage the full result and only then write to the destination, keyed by an idempotency column so a retried batch updates rather than duplicates. Trap: Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. Done when: A source paging through a fake API with a 429 on the fourth page waits the interval the response specifies and completes, tested. A destination write that fails partway leaves the destination untouched, because the write happens only after the full result is staged. A replayed run against the same staged data produces the same destination state, tested by running it twice. The configured request rate is recorded with the date the vendor's published limit was read. Do not build yet: No replay interface, no export and no cost ceilings yet. ### Phase 5 · Reading the failure and taking the data out Build: A run page showing each step with its row counts, its errored rows with raw values, and the elapsed time. Replay a run from any step using the stored output of the previous one. Export a flow definition and any step output. Cap run duration, rows and destination writes, and stop a run that exceeds a cap rather than continuing. Alert once per failing flow, not once per run. Trap: A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration. Done when: The run page shows every step's four row counts and lists errored rows with their raw values. Replaying from step four uses the stored output of step three rather than re-reading the source, tested. A run exceeding the configured row or duration ceiling stops and is marked as capped. A flow failing five nights running produces one alert on the transition plus one on recovery, tested with a fake clock. Do not build yet: Nothing after this. ### Out of scope (and why) - Hundreds of maintained connectors, which are the actual subscription; build the two or three sources you personally use and an HTTP step for the rest. - Verified OAuth applications for consumer platforms, which need a review process and a privacy policy before a single row moves. - Multi-user workspaces, sharing and approvals, which turn every run into a permissions question. - Schema drift handling across a live vendor API, which is a maintenance commitment rather than a feature you finish. ### README must contain - That column types are declared on the source and never inferred, with the leading zero case named - That every step materialises its output and the four row counts reconcile, asserted in the suite - That schedules are cron plus a named timezone, and what happens on both clock change days - That destination writes happen only after the full result is staged, keyed on an identifier from the data - Which vendor rate limits were read, on what date, and where to recheck them ===== ARCHITECTURE.md ===== # Architecture · Parabola ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | TypeScript on Node 22, one worker process plus one web process | the schedule has to fire whether or not a browser is open | | Storage | PostgreSQL for flow definitions and run history, files on disk for step outputs | row-level data belongs in files; run metadata belongs in a database you query | | Queue | BullMQ on Redis with one active run per flow enforced by a lock | two overlapping runs of the same flow is how a destination gets written twice | | Steps | Pure functions from one table to one table, output materialised per step | a step you cannot preview is a step you cannot debug at two in the morning | | Types | Declared on the source, never inferred per row | inference is what turns an order number into a smaller order number | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Source | Declared-type parsing and row_errors. | Any parser, provided types stay declared and bad values keep their raw text. | | Engine | Materialised steps and row accounting. | Any execution model, provided every step's output is inspectable afterwards. | | Schedule | Zone-aware fire times and the per-flow lock. | Any scheduler, provided overlapping runs are impossible. | | Destinations | Staging and idempotent writes. | Not swappable for streaming writes, which is the half-written failure. | | Run history | Replay, ceilings and transition alerts. | Any presentation, provided replay uses stored outputs. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required · Flows, versions, runs and row errors. - `REDIS_URL` · required · The queue and the per-flow run lock. - `DATA_DIR` · required · Materialised step outputs. Back this up with the database or a replay is impossible. - `STEP_OUTPUT_RETENTION_DAYS` · required · How long step outputs are kept for replay before deletion. - `MAX_ROWS_PER_RUN` · required · A hard ceiling. A run that exceeds it stops and is marked capped. - `MAX_RUN_MINUTES` · required · The other ceiling, for the source that got slow rather than large. - `SOURCE_API_TOKEN` · required, secret · Read scope only on the service you pull from. - `DESTINATION_API_TOKEN` · optional, secret · The narrowest write scope on the destination, separate from the read token. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Parabola product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: TypeScript on Node 22, one worker process plus one web process, PostgreSQL for flow definitions and run history, files on disk for step outputs, BullMQ on Redis with one active run per flow enforced by a lock, Pure functions from one table to one table, output materialised per step, Declared on the source, never inferred per row. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. - A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration. ===== MILESTONES.md ===== # Delivery milestones · Parabola Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Reading a file without changing it A tabular source is parsed to declared types, and anything that does not fit becomes a visible error rather than a guess. ### Steps 1. Declare the schema on the source One declared type per column, applied to every row, with no per-row inference anywhere. Files: `src/steps/source-csv.ts`, `src/schema.ts` 2. Parse quoting and line breaks by the specification Hand-rolled splitting on commas breaks on the first field containing one. Files: `src/csv/parse.ts`, `tests/csv-quoting.test.ts` 3. Route bad values to row_errors with the raw text The raw text is the only thing that lets someone fix the source file. Files: `src/steps/source-csv.ts`, `tests/leading-zeros.test.ts` ### Done when - [ ] Leading zeros survive as declared text - [ ] Quoted commas, newlines and doubled quotes round-trip - [ ] Unparseable values reach row_errors with raw text - [ ] A wrong declared encoding fails at the source step ### Watch out - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. ## M2 · Steps you can look inside Every step materialises its output and accounts for every row that entered it. ### Steps 1. Materialise each step's output to its own file The debugging story for a pipeline is being able to see the table between any two steps. Files: `src/engine/run.ts`, `src/engine/materialise.ts` 2. Reconcile the row counts in the suite Rows in equals rows out plus filtered plus errored, for every step, every time. Files: `src/engine/step-run.ts`, `tests/row-accounting.test.ts` 3. Fail a join on mismatched key types A join that matches nothing looks identical to a filter that worked. Files: `src/steps/join.ts`, `tests/join-types.test.ts` ### Done when - [ ] The four row counts reconcile for every step - [ ] Each step shows a sample of that run's real output - [ ] A mismatched join key fails with both types named - [ ] A repeated run produces identical step outputs ### Watch out - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. ## M3 · A schedule that does not run twice Scheduled runs fire in a named timezone, never overlap, and survive a restart mid-run. ### Steps 1. Compute fire times from the zone's rules Not by adding a fixed interval, which is wrong twice a year. Files: `src/schedule/next.ts`, `tests/dst.test.ts` 2. Take a per-flow lock and record skips A skipped tick you can see beats an overlapping run you cannot. Files: `src/queue/lock.ts` 3. Mark interrupted runs and require a replay Automatic resumption is how half a file gets written twice. Files: `src/engine/recover.ts` ### Done when - [ ] A daily schedule fires once per local day on both transitions - [ ] An overlapping tick is recorded as a skip - [ ] An interrupted run is never silently resumed - [ ] The missing hour rule is stated and tested ### Watch out - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. ## M4 · APIs that push back A paged API source finishes or fails cleanly, and a destination is never half written. ### Steps 1. Honour the retry interval the response gives you Guessing a backoff when the server told you the number is how an account gets throttled for a day. Files: `src/http/client.ts`, `tests/rate-limit.test.ts` 2. Stage the whole result before writing anything A half-written destination is the failure mode with no clean recovery. Files: `src/destinations/stage.ts` 3. Key destination writes on an identifier from the data So a retry is an update and not a duplicate row. Files: `src/destinations/write.ts`, `tests/idempotent-write.test.ts` ### Done when - [ ] A 429 is honoured by waiting the stated interval - [ ] A failed write leaves the destination untouched - [ ] Running the same staged result twice is idempotent - [ ] The chosen rate cites when the limit was read ### Watch out - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. ## M5 · Reading the failure and taking the data out (production only) Every run is explainable after the fact and nothing in the system is trapped. ### Steps 1. Show row counts and errored rows on the run page The raw value in row_errors is what the person needs to fix the source. Files: `src/web/routes/run.tsx` 2. Replay from any step using stored outputs Re-reading a source that has since changed is not a replay. Files: `src/engine/replay.ts`, `tests/replay.test.ts` 3. Cap rows, duration and writes, and alert on transitions A ceiling stops the runaway; transition alerts stop the muting. Files: `src/engine/limits.ts`, `src/alerts.ts` ### Done when - [ ] Row counts and errored raw values appear per step - [ ] Replay from a step uses the stored output - [ ] A run over its ceiling stops and is marked capped - [ ] Alerts fire on failure and recovery, not nightly ### Watch out - A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration. ===== OPERATIONS.md ===== # Operations · Parabola ## Backup The database and the step output directory together, since a run history pointing at files that no longer exist cannot be replayed. Flow definitions are the cheapest thing to lose and the most annoying to rebuild, so export them to version control as well. ## Restore Restore both, then mark every run that was in progress as interrupted before starting the worker. Never let a restored worker resume a run it did not start, because the destination may already hold part of it. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Failing flows, capped runs, the age of the last successful run per flow, and the size of the step output directory. A flow that quietly stopped being scheduled is the failure nobody notices for a month. ## Incident checklist A wrong number in a destination is traced by opening the run, finding the step where the row counts change unexpectedly, and reading the sample output on either side. That is what materialising every step is for. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Leading zeros are confirmed to survive as declared text - [ ] The four row counts are confirmed to reconcile per step - [ ] An overlapping tick is confirmed to be skipped, not queued - [ ] A failed write is confirmed to leave the destination untouched - [ ] A repeatedly failing flow is confirmed to alert once, not nightly ## Launch constraint Do not market omitted Parabola capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Flows, versions, runs and row errors. DATABASE_URL=postgres://localhost:5432/pipelines # Required. The queue and the per-flow run lock. REDIS_URL=redis://localhost:6379 # Required. Materialised step outputs. Back this up with the database or a replay is impossible. DATA_DIR=./data/steps # Required. How long step outputs are kept for replay before deletion. STEP_OUTPUT_RETENTION_DAYS=30 # Required. A hard ceiling. A run that exceeds it stops and is marked capped. MAX_ROWS_PER_RUN=500000 # Required. The other ceiling, for the source that got slow rather than large. MAX_RUN_MINUTES=30 # Required · secret. Read scope only on the service you pull from. SOURCE_API_TOKEN=sk-...redacted # Optional · secret. The narrowest write scope on the destination, separate from the read token. DESTINATION_API_TOKEN=sk-...redacted
README.md
# Parabola · indie build A step-by-step data pipeline whose hard problems are reading tabular files without silently changing the values, keeping every step deterministic and previewable, and running on a schedule against APIs that push back without ever writing half a result. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | TypeScript on Node 22, one worker process plus one web process | the schedule has to fire whether or not a browser is open | | Storage | PostgreSQL for flow definitions and run history, files on disk for step outputs | row-level data belongs in files; run metadata belongs in a database you query | | Queue | BullMQ on Redis with one active run per flow enforced by a lock | two overlapping runs of the same flow is how a destination gets written twice | | Steps | Pure functions from one table to one table, output materialised per step | a step you cannot preview is a step you cannot debug at two in the morning | | Types | Declared on the source, never inferred per row | inference is what turns an order number into a smaller order number | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the worker and the web process. - Get it: nodejs.org, or nvm install 22. - Verify: node --version prints v22 or higher - [ ] **PostgreSQL 16 or newer** · free locally, a few pounds a month managed - Why: Holds flow definitions and run history, which you will query far more than you expect once something fails at three in the morning. - Get it: Docker, your package manager, or a managed instance. - Verify: psql -c 'select version()' connects - [ ] **Redis for the queue and the per-flow lock** · free locally - Why: The lock that stops two runs of one flow overlapping lives here, and overlapping runs are the duplicate-write bug. - Get it: Docker or your package manager. - Verify: redis-cli ping returns PONG - [ ] **API credentials for the one or two services you actually use** · free for most, some charge per call - Why: The whole point is moving your own data, and a pipeline with no real source is a demo. - Get it: Create a scoped token in each service, with read on the source and the narrowest write on the destination. - Verify: A single authenticated request succeeds from the command line - [ ] **The published rate limit for each service, and the date you read it** · free - Why: The request rate you choose is only defensible if it comes from the vendor's current page, and these change. - Get it: Read each vendor's rate limit documentation now and write down the figure with today's date. - Verify: Each configured rate cites a source and a date in the README - [ ] **A real messy file from your own data** · free - Why: Leading zeros, embedded newlines and an unexpected encoding are the failures that matter, and clean fixtures never show them. - Get it: Export something from the system you actually use, without tidying it up first. - Verify: It parses with a declared schema and its bad rows land in row_errors ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A connector library. - Verified OAuth applications. - Multi-user workspaces and approvals. - Automatic schema drift handling. - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors - OAuth app verification If one of those is essential to you, that is the reason to keep paying for Parabola, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Parabola The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a visual data pipeline that runs on a schedule and can account for every row it touched to replace Parabola. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: TypeScript on Node 22, one worker process plus one web process · the schedule has to fire whether or not a browser is open - Storage: PostgreSQL for flow definitions and run history, files on disk for step outputs · row-level data belongs in files; run metadata belongs in a database you query - Queue: BullMQ on Redis with one active run per flow enforced by a lock · two overlapping runs of the same flow is how a destination gets written twice - Steps: Pure functions from one table to one table, output materialised per step · a step you cannot preview is a step you cannot debug at two in the morning - Types: Declared on the source, never inferred per row · inference is what turns an order number into a smaller order number ### Data model (create this before Phase 1) - flows(id, name, current_version, schedule_cron, timezone, enabled, created_at) - flow_versions(flow_id, version, graph_json, created_at) · runs reference the version they executed, so an edit does not rewrite history - runs(id, flow_id, flow_version, trigger, state, started_at, finished_at, rows_in, rows_out, error) · trigger is schedule, manual or replay - step_runs(run_id, step_id, ordinal, state, rows_in, rows_out, rows_filtered, rows_errored, output_path, duration_ms) - row_errors(run_id, step_id, row_ordinal, column_name, raw_value, reason) · the raw value is kept exactly as it arrived - destinations(id, kind, credential_ref, write_mode, idempotency_column, last_run_id) · write_mode is replace, append or upsert - Every step run satisfies rows_in equals rows_out plus rows_filtered plus rows_errored, and the suite asserts it, because a pipeline that quietly loses rows is worse than one that fails. ### Phase 1 · Reading a file without changing it Build: Parse delimited files to the letter of RFC 4180: records separated by CRLF, fields containing a comma, a quote or a line break wrapped in double quotes, and an inner double quote escaped by doubling it. The format carries no type information and no encoding declaration, so take both from a declared schema per column plus a declared encoding, and send any value that does not parse to the row_errors table with its raw text intact. Detect and reject a byte order mark rather than folding it into the first column name. Trap: Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. Done when: A column declared as text preserves leading zeros through the whole pipeline, tested with a fixture. A quoted field containing a comma, a newline and a doubled quote round-trips exactly. A value that does not parse against its declared type lands in row_errors with its raw text and does not silently become null. A file whose encoding differs from the declared one fails loudly at the source step rather than producing replacement characters downstream. Do not build yet: No steps beyond the source, no schedule, no destinations and no API sources. ### Phase 2 · Steps you can look inside Build: Implement filter, select, rename, derive, join, group and sort as pure functions from one table to one table. Write each step's output to its own file, record rows in, out, filtered and errored in step_runs, and show a sample of the output beside the step in the editor. Make the graph an ordered list of steps with a single source, so evaluation order is not a question. Trap: Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. Done when: Every step run records rows in, out, filtered and errored, and the suite asserts the four numbers reconcile. Clicking any step shows a sample of its actual output from that run, not a recomputation. A join between a text key and an integer key fails with both types named rather than matching nothing. Running the same flow twice over the same input produces byte-identical step outputs. Do not build yet: No schedule, no API sources or destinations, no retries. ### Phase 3 · A schedule that does not run twice Build: Schedule each flow with a cron expression plus a named timezone, computing each fire time from that zone's rules for that date. Take a lock per flow so a slow run never overlaps the next tick, and record the skip rather than queueing it. On restart, mark any run left in progress as interrupted and require an explicit replay instead of resuming blindly. Trap: Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. Done when: A daily schedule fires once per local day across both clock change days, tested against hand-computed times in a zone that observes them. A run still executing when the next tick arrives causes a recorded skip rather than a second run, tested by holding a lock. Killing the worker mid-run leaves the run marked interrupted, and it is never silently resumed. A schedule landing in the missing hour resolves by a rule stated in the README. Do not build yet: No API sources or destinations, no rate limit handling, no export. ### Phase 4 · APIs that push back Build: Add an HTTP source that pages through results and an HTTP destination that writes in batches. Read each vendor's current published rate limit before choosing a request rate, honour a 429 by waiting the interval the response asks for rather than a guess, and back off on repeated failures. Stage the full result and only then write to the destination, keyed by an idempotency column so a retried batch updates rather than duplicates. Trap: Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. Done when: A source paging through a fake API with a 429 on the fourth page waits the interval the response specifies and completes, tested. A destination write that fails partway leaves the destination untouched, because the write happens only after the full result is staged. A replayed run against the same staged data produces the same destination state, tested by running it twice. The configured request rate is recorded with the date the vendor's published limit was read. Do not build yet: No replay interface, no export and no cost ceilings yet. ### Phase 5 · Reading the failure and taking the data out Build: A run page showing each step with its row counts, its errored rows with raw values, and the elapsed time. Replay a run from any step using the stored output of the previous one. Export a flow definition and any step output. Cap run duration, rows and destination writes, and stop a run that exceeds a cap rather than continuing. Alert once per failing flow, not once per run. Trap: A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration. Done when: The run page shows every step's four row counts and lists errored rows with their raw values. Replaying from step four uses the stored output of step three rather than re-reading the source, tested. A run exceeding the configured row or duration ceiling stops and is marked as capped. A flow failing five nights running produces one alert on the transition plus one on recovery, tested with a fake clock. Do not build yet: Nothing after this. ### Out of scope (and why) - Hundreds of maintained connectors, which are the actual subscription; build the two or three sources you personally use and an HTTP step for the rest. - Verified OAuth applications for consumer platforms, which need a review process and a privacy policy before a single row moves. - Multi-user workspaces, sharing and approvals, which turn every run into a permissions question. - Schema drift handling across a live vendor API, which is a maintenance commitment rather than a feature you finish. ### README must contain - That column types are declared on the source and never inferred, with the leading zero case named - That every step materialises its output and the four row counts reconcile, asserted in the suite - That schedules are cron plus a named timezone, and what happens on both clock change days - That destination writes happen only after the full result is staged, keyed on an identifier from the data - Which vendor rate limits were read, on what date, and where to recheck them
AGENTS.md
# Agent instructions · Parabola indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: TypeScript on Node 22, one worker process plus one web process, PostgreSQL for flow definitions and run history, files on disk for step outputs, BullMQ on Redis with one active run per flow enforced by a lock, Pure functions from one table to one table, output materialised per step, Declared on the source, never inferred per row. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update.
BUILD_PLAN.md
# Build plan · Parabola A step-by-step data pipeline whose hard problems are reading tabular files without silently changing the values, keeping every step deterministic and previewable, and running on a schedule against APIs that push back without ever writing half a result. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Reading a file without changing it A tabular source is parsed to declared types, and anything that does not fit becomes a visible error rather than a guess. ### Steps 1. Declare the schema on the source One declared type per column, applied to every row, with no per-row inference anywhere. Files: `src/steps/source-csv.ts`, `src/schema.ts` 2. Parse quoting and line breaks by the specification Hand-rolled splitting on commas breaks on the first field containing one. Files: `src/csv/parse.ts`, `tests/csv-quoting.test.ts` 3. Route bad values to row_errors with the raw text The raw text is the only thing that lets someone fix the source file. Files: `src/steps/source-csv.ts`, `tests/leading-zeros.test.ts` ### Done when - [ ] Leading zeros survive as declared text - [ ] Quoted commas, newlines and doubled quotes round-trip - [ ] Unparseable values reach row_errors with raw text - [ ] A wrong declared encoding fails at the source step ### Watch out - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. ## Phase 2 · Steps you can look inside Every step materialises its output and accounts for every row that entered it. ### Steps 1. Materialise each step's output to its own file The debugging story for a pipeline is being able to see the table between any two steps. Files: `src/engine/run.ts`, `src/engine/materialise.ts` 2. Reconcile the row counts in the suite Rows in equals rows out plus filtered plus errored, for every step, every time. Files: `src/engine/step-run.ts`, `tests/row-accounting.test.ts` 3. Fail a join on mismatched key types A join that matches nothing looks identical to a filter that worked. Files: `src/steps/join.ts`, `tests/join-types.test.ts` ### Done when - [ ] The four row counts reconcile for every step - [ ] Each step shows a sample of that run's real output - [ ] A mismatched join key fails with both types named - [ ] A repeated run produces identical step outputs ### Watch out - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. ## Phase 3 · A schedule that does not run twice Scheduled runs fire in a named timezone, never overlap, and survive a restart mid-run. ### Steps 1. Compute fire times from the zone's rules Not by adding a fixed interval, which is wrong twice a year. Files: `src/schedule/next.ts`, `tests/dst.test.ts` 2. Take a per-flow lock and record skips A skipped tick you can see beats an overlapping run you cannot. Files: `src/queue/lock.ts` 3. Mark interrupted runs and require a replay Automatic resumption is how half a file gets written twice. Files: `src/engine/recover.ts` ### Done when - [ ] A daily schedule fires once per local day on both transitions - [ ] An overlapping tick is recorded as a skip - [ ] An interrupted run is never silently resumed - [ ] The missing hour rule is stated and tested ### Watch out - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. ## Phase 4 · APIs that push back A paged API source finishes or fails cleanly, and a destination is never half written. ### Steps 1. Honour the retry interval the response gives you Guessing a backoff when the server told you the number is how an account gets throttled for a day. Files: `src/http/client.ts`, `tests/rate-limit.test.ts` 2. Stage the whole result before writing anything A half-written destination is the failure mode with no clean recovery. Files: `src/destinations/stage.ts` 3. Key destination writes on an identifier from the data So a retry is an update and not a duplicate row. Files: `src/destinations/write.ts`, `tests/idempotent-write.test.ts` ### Done when - [ ] A 429 is honoured by waiting the stated interval - [ ] A failed write leaves the destination untouched - [ ] Running the same staged result twice is idempotent - [ ] The chosen rate cites when the limit was read ### Watch out - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. ## Not in this build - A connector library. - Verified OAuth applications. - Multi-user workspaces and approvals. - Automatic schema drift handling. ## After v1, if you want it - A dry run mode that executes every step and stops before the destination write - Column-level lineage, which the materialised outputs already make possible - A second destination kind, once one has run for a month without a duplicate
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Flows, versions, runs and row errors. DATABASE_URL=postgres://localhost:5432/pipelines # Required. The queue and the per-flow run lock. REDIS_URL=redis://localhost:6379 # Required. Materialised step outputs. Back this up with the database or a replay is impossible. DATA_DIR=./data/steps # Required. How long step outputs are kept for replay before deletion. STEP_OUTPUT_RETENTION_DAYS=30 # Required. A hard ceiling. A run that exceeds it stops and is marked capped. MAX_ROWS_PER_RUN=500000 # Required. The other ceiling, for the source that got slow rather than large. MAX_RUN_MINUTES=30 # Required · secret. Read scope only on the service you pull from. SOURCE_API_TOKEN=sk-...redacted # Optional · secret. The narrowest write scope on the destination, separate from the read token. DESTINATION_API_TOKEN=sk-...redacted
PRODUCT.md
# Parabola · product brief ## Problem The visible visual data automation loop is buildable, but a credible replacement needs more than the first screen. Parabola earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A pipeline that reads your files without changing the values in them, shows you the table between any two steps, runs on a schedule without ever overlapping itself, and never leaves a destination half written. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - hosted PostgreSQL and Redis - API credentials for the two chosen services - Node.js 22 ## Explicit non-goals for v1 - A connector library. - Verified OAuth applications. - Multi-user workspaces and approvals. - Automatic schema drift handling. - durable execution at scale - schema drift handling and enterprise controls - hundreds of maintained connectors - OAuth app verification ## Success criteria - Leading zeros are confirmed to survive as declared text - The four row counts are confirmed to reconcile per step - An overlapping tick is confirmed to be skipped, not queued - A failed write is confirmed to leave the destination untouched - A repeatedly failing flow is confirmed to alert once, not nightly
BRIEF.md
# Build brief · Parabola The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a visual data pipeline that runs on a schedule and can account for every row it touched to replace Parabola. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: TypeScript on Node 22, one worker process plus one web process · the schedule has to fire whether or not a browser is open - Storage: PostgreSQL for flow definitions and run history, files on disk for step outputs · row-level data belongs in files; run metadata belongs in a database you query - Queue: BullMQ on Redis with one active run per flow enforced by a lock · two overlapping runs of the same flow is how a destination gets written twice - Steps: Pure functions from one table to one table, output materialised per step · a step you cannot preview is a step you cannot debug at two in the morning - Types: Declared on the source, never inferred per row · inference is what turns an order number into a smaller order number ### Data model (create this before Phase 1) - flows(id, name, current_version, schedule_cron, timezone, enabled, created_at) - flow_versions(flow_id, version, graph_json, created_at) · runs reference the version they executed, so an edit does not rewrite history - runs(id, flow_id, flow_version, trigger, state, started_at, finished_at, rows_in, rows_out, error) · trigger is schedule, manual or replay - step_runs(run_id, step_id, ordinal, state, rows_in, rows_out, rows_filtered, rows_errored, output_path, duration_ms) - row_errors(run_id, step_id, row_ordinal, column_name, raw_value, reason) · the raw value is kept exactly as it arrived - destinations(id, kind, credential_ref, write_mode, idempotency_column, last_run_id) · write_mode is replace, append or upsert - Every step run satisfies rows_in equals rows_out plus rows_filtered plus rows_errored, and the suite asserts it, because a pipeline that quietly loses rows is worse than one that fails. ### Phase 1 · Reading a file without changing it Build: Parse delimited files to the letter of RFC 4180: records separated by CRLF, fields containing a comma, a quote or a line break wrapped in double quotes, and an inner double quote escaped by doubling it. The format carries no type information and no encoding declaration, so take both from a declared schema per column plus a declared encoding, and send any value that does not parse to the row_errors table with its raw text intact. Detect and reject a byte order mark rather than folding it into the first column name. Trap: Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. Done when: A column declared as text preserves leading zeros through the whole pipeline, tested with a fixture. A quoted field containing a comma, a newline and a doubled quote round-trips exactly. A value that does not parse against its declared type lands in row_errors with its raw text and does not silently become null. A file whose encoding differs from the declared one fails loudly at the source step rather than producing replacement characters downstream. Do not build yet: No steps beyond the source, no schedule, no destinations and no API sources. ### Phase 2 · Steps you can look inside Build: Implement filter, select, rename, derive, join, group and sort as pure functions from one table to one table. Write each step's output to its own file, record rows in, out, filtered and errored in step_runs, and show a sample of the output beside the step in the editor. Make the graph an ordered list of steps with a single source, so evaluation order is not a question. Trap: Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. Done when: Every step run records rows in, out, filtered and errored, and the suite asserts the four numbers reconcile. Clicking any step shows a sample of its actual output from that run, not a recomputation. A join between a text key and an integer key fails with both types named rather than matching nothing. Running the same flow twice over the same input produces byte-identical step outputs. Do not build yet: No schedule, no API sources or destinations, no retries. ### Phase 3 · A schedule that does not run twice Build: Schedule each flow with a cron expression plus a named timezone, computing each fire time from that zone's rules for that date. Take a lock per flow so a slow run never overlaps the next tick, and record the skip rather than queueing it. On restart, mark any run left in progress as interrupted and require an explicit replay instead of resuming blindly. Trap: Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. Done when: A daily schedule fires once per local day across both clock change days, tested against hand-computed times in a zone that observes them. A run still executing when the next tick arrives causes a recorded skip rather than a second run, tested by holding a lock. Killing the worker mid-run leaves the run marked interrupted, and it is never silently resumed. A schedule landing in the missing hour resolves by a rule stated in the README. Do not build yet: No API sources or destinations, no rate limit handling, no export. ### Phase 4 · APIs that push back Build: Add an HTTP source that pages through results and an HTTP destination that writes in batches. Read each vendor's current published rate limit before choosing a request rate, honour a 429 by waiting the interval the response asks for rather than a guess, and back off on repeated failures. Stage the full result and only then write to the destination, keyed by an idempotency column so a retried batch updates rather than duplicates. Trap: Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. Done when: A source paging through a fake API with a 429 on the fourth page waits the interval the response specifies and completes, tested. A destination write that fails partway leaves the destination untouched, because the write happens only after the full result is staged. A replayed run against the same staged data produces the same destination state, tested by running it twice. The configured request rate is recorded with the date the vendor's published limit was read. Do not build yet: No replay interface, no export and no cost ceilings yet. ### Phase 5 · Reading the failure and taking the data out Build: A run page showing each step with its row counts, its errored rows with raw values, and the elapsed time. Replay a run from any step using the stored output of the previous one. Export a flow definition and any step output. Cap run duration, rows and destination writes, and stop a run that exceeds a cap rather than continuing. Alert once per failing flow, not once per run. Trap: A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration. Done when: The run page shows every step's four row counts and lists errored rows with their raw values. Replaying from step four uses the stored output of step three rather than re-reading the source, tested. A run exceeding the configured row or duration ceiling stops and is marked as capped. A flow failing five nights running produces one alert on the transition plus one on recovery, tested with a fake clock. Do not build yet: Nothing after this. ### Out of scope (and why) - Hundreds of maintained connectors, which are the actual subscription; build the two or three sources you personally use and an HTTP step for the rest. - Verified OAuth applications for consumer platforms, which need a review process and a privacy policy before a single row moves. - Multi-user workspaces, sharing and approvals, which turn every run into a permissions question. - Schema drift handling across a live vendor API, which is a maintenance commitment rather than a feature you finish. ### README must contain - That column types are declared on the source and never inferred, with the leading zero case named - That every step materialises its output and the four row counts reconcile, asserted in the suite - That schedules are cron plus a named timezone, and what happens on both clock change days - That destination writes happen only after the full result is staged, keyed on an identifier from the data - Which vendor rate limits were read, on what date, and where to recheck them
ARCHITECTURE.md
# Architecture · Parabola ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | TypeScript on Node 22, one worker process plus one web process | the schedule has to fire whether or not a browser is open | | Storage | PostgreSQL for flow definitions and run history, files on disk for step outputs | row-level data belongs in files; run metadata belongs in a database you query | | Queue | BullMQ on Redis with one active run per flow enforced by a lock | two overlapping runs of the same flow is how a destination gets written twice | | Steps | Pure functions from one table to one table, output materialised per step | a step you cannot preview is a step you cannot debug at two in the morning | | Types | Declared on the source, never inferred per row | inference is what turns an order number into a smaller order number | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Source | Declared-type parsing and row_errors. | Any parser, provided types stay declared and bad values keep their raw text. | | Engine | Materialised steps and row accounting. | Any execution model, provided every step's output is inspectable afterwards. | | Schedule | Zone-aware fire times and the per-flow lock. | Any scheduler, provided overlapping runs are impossible. | | Destinations | Staging and idempotent writes. | Not swappable for streaming writes, which is the half-written failure. | | Run history | Replay, ceilings and transition alerts. | Any presentation, provided replay uses stored outputs. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required · Flows, versions, runs and row errors. - `REDIS_URL` · required · The queue and the per-flow run lock. - `DATA_DIR` · required · Materialised step outputs. Back this up with the database or a replay is impossible. - `STEP_OUTPUT_RETENTION_DAYS` · required · How long step outputs are kept for replay before deletion. - `MAX_ROWS_PER_RUN` · required · A hard ceiling. A run that exceeds it stops and is marked capped. - `MAX_RUN_MINUTES` · required · The other ceiling, for the source that got slow rather than large. - `SOURCE_API_TOKEN` · required, secret · Read scope only on the service you pull from. - `DESTINATION_API_TOKEN` · optional, secret · The narrowest write scope on the destination, separate from the read token. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Parabola product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: TypeScript on Node 22, one worker process plus one web process, PostgreSQL for flow definitions and run history, files on disk for step outputs, BullMQ on Redis with one active run per flow enforced by a lock, Pure functions from one table to one table, output materialised per step, Declared on the source, never inferred per row. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. - A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration.
MILESTONES.md
# Delivery milestones · Parabola Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Reading a file without changing it A tabular source is parsed to declared types, and anything that does not fit becomes a visible error rather than a guess. ### Steps 1. Declare the schema on the source One declared type per column, applied to every row, with no per-row inference anywhere. Files: `src/steps/source-csv.ts`, `src/schema.ts` 2. Parse quoting and line breaks by the specification Hand-rolled splitting on commas breaks on the first field containing one. Files: `src/csv/parse.ts`, `tests/csv-quoting.test.ts` 3. Route bad values to row_errors with the raw text The raw text is the only thing that lets someone fix the source file. Files: `src/steps/source-csv.ts`, `tests/leading-zeros.test.ts` ### Done when - [ ] Leading zeros survive as declared text - [ ] Quoted commas, newlines and doubled quotes round-trip - [ ] Unparseable values reach row_errors with raw text - [ ] A wrong declared encoding fails at the source step ### Watch out - Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing. ## M2 · Steps you can look inside Every step materialises its output and accounts for every row that entered it. ### Steps 1. Materialise each step's output to its own file The debugging story for a pipeline is being able to see the table between any two steps. Files: `src/engine/run.ts`, `src/engine/materialise.ts` 2. Reconcile the row counts in the suite Rows in equals rows out plus filtered plus errored, for every step, every time. Files: `src/engine/step-run.ts`, `tests/row-accounting.test.ts` 3. Fail a join on mismatched key types A join that matches nothing looks identical to a filter that worked. Files: `src/steps/join.ts`, `tests/join-types.test.ts` ### Done when - [ ] The four row counts reconcile for every step - [ ] Each step shows a sample of that run's real output - [ ] A mismatched join key fails with both types named - [ ] A repeated run produces identical step outputs ### Watch out - Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them. ## M3 · A schedule that does not run twice Scheduled runs fire in a named timezone, never overlap, and survive a restart mid-run. ### Steps 1. Compute fire times from the zone's rules Not by adding a fixed interval, which is wrong twice a year. Files: `src/schedule/next.ts`, `tests/dst.test.ts` 2. Take a per-flow lock and record skips A skipped tick you can see beats an overlapping run you cannot. Files: `src/queue/lock.ts` 3. Mark interrupted runs and require a replay Automatic resumption is how half a file gets written twice. Files: `src/engine/recover.ts` ### Done when - [ ] A daily schedule fires once per local day on both transitions - [ ] An overlapping tick is recorded as a skip - [ ] An interrupted run is never silently resumed - [ ] The missing hour rule is stated and tested ### Watch out - Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does. ## M4 · APIs that push back A paged API source finishes or fails cleanly, and a destination is never half written. ### Steps 1. Honour the retry interval the response gives you Guessing a backoff when the server told you the number is how an account gets throttled for a day. Files: `src/http/client.ts`, `tests/rate-limit.test.ts` 2. Stage the whole result before writing anything A half-written destination is the failure mode with no clean recovery. Files: `src/destinations/stage.ts` 3. Key destination writes on an identifier from the data So a retry is an update and not a duplicate row. Files: `src/destinations/write.ts`, `tests/idempotent-write.test.ts` ### Done when - [ ] A 429 is honoured by waiting the stated interval - [ ] A failed write leaves the destination untouched - [ ] Running the same staged result twice is idempotent - [ ] The chosen rate cites when the limit was read ### Watch out - Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update. ## M5 · Reading the failure and taking the data out (production only) Every run is explainable after the fact and nothing in the system is trapped. ### Steps 1. Show row counts and errored rows on the run page The raw value in row_errors is what the person needs to fix the source. Files: `src/web/routes/run.tsx` 2. Replay from any step using stored outputs Re-reading a source that has since changed is not a replay. Files: `src/engine/replay.ts`, `tests/replay.test.ts` 3. Cap rows, duration and writes, and alert on transitions A ceiling stops the runaway; transition alerts stop the muting. Files: `src/engine/limits.ts`, `src/alerts.ts` ### Done when - [ ] Row counts and errored raw values appear per step - [ ] Replay from a step uses the stored output - [ ] A run over its ceiling stops and is marked capped - [ ] Alerts fire on failure and recovery, not nightly ### Watch out - A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration.
OPERATIONS.md
# Operations · Parabola ## Backup The database and the step output directory together, since a run history pointing at files that no longer exist cannot be replayed. Flow definitions are the cheapest thing to lose and the most annoying to rebuild, so export them to version control as well. ## Restore Restore both, then mark every run that was in progress as interrupted before starting the worker. Never let a restored worker resume a run it did not start, because the destination may already hold part of it. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Failing flows, capped runs, the age of the last successful run per flow, and the size of the step output directory. A flow that quietly stopped being scheduled is the failure nobody notices for a month. ## Incident checklist A wrong number in a destination is traced by opening the run, finding the step where the row counts change unexpectedly, and reading the sample output on either side. That is what materialising every step is for. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Leading zeros are confirmed to survive as declared text - [ ] The four row counts are confirmed to reconcile per step - [ ] An overlapping tick is confirmed to be skipped, not queued - [ ] A failed write is confirmed to leave the destination untouched - [ ] A repeatedly failing flow is confirmed to alert once, not nightly ## Launch constraint Do not market omitted Parabola capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Flows, versions, runs and row errors. DATABASE_URL=postgres://localhost:5432/pipelines # Required. The queue and the per-flow run lock. REDIS_URL=redis://localhost:6379 # Required. Materialised step outputs. Back this up with the database or a replay is impossible. DATA_DIR=./data/steps # Required. How long step outputs are kept for replay before deletion. STEP_OUTPUT_RETENTION_DAYS=30 # Required. A hard ceiling. A run that exceeds it stops and is marked capped. MAX_ROWS_PER_RUN=500000 # Required. The other ceiling, for the source that got slow rather than large. MAX_RUN_MINUTES=30 # Required · secret. Read scope only on the service you pull from. SOURCE_API_TOKEN=sk-...redacted # Optional · secret. The narrowest write scope on the destination, separate from the read token. DESTINATION_API_TOKEN=sk-...redacted
$ choose a build depth, inspect the files, then open the complete pack in your agent
Parabola: One automation is easy. Subscribers pay for maintained integrations, credentials, retries, observability, and someone else owning breakage.
xdurable execution at scale
xschema drift handling and enterprise controls
xhundreds of maintained connectors
xOAuth app verification
share on X ↗"I just replaced Parabola with one prompt"
Don't feel like building it? These folks already made it free.
KNIME Analytics PlatformA visual data workbench with hundreds of connectors; scheduled cloud runs are where KNIME starts charging.open source↗no votes, no pay-to-list · just what's real
Parabola pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| basic | $0/workspace | $0/workspace | 1 user; 1,000 credits/month; unlimited flows; limited AI usage; hard stop at the monthly cap. |
| explorer | $20/workspace | — | 1 user; 1,500 credits/month; 250 MB table storage; daily scheduling. |
| collaborator | $400/workspace | — | Up to 3 users; 30,000 credits/month; custom scheduling. |
| business | custom | — | Unlimited users; custom credit allotment, storage, scheduling and security. |
free tier1 user, 1,000 credits/month, unlimited flows and a hard stop at the cap
billingself-service plans are monthly only; Business is custom and may be annual
hidden costsExplorer charges $0.18 per additional 10 credits. Credits are charged on flow runs; paid plans move into pay-per-credit usage, while failed or zero-row steps may not count. Cancellation can immediately finalize accrued overage.
verified 2026-08-11 · source ↗
Vibecode Parabola
Kinda. The core of Parabola is buildable in a weekend with the prompt on this page, but there are real gaps: durable execution at scale, schema drift handling and enterprise controls. Read the honest list above before committing.
How much does Parabola cost?
Parabola's pricing is usage-based or varies by plan · The official pricing URL is recorded, but a stable current amount was not safely recoverable in this pass. Do not publish a number until rechecked..
What do I lose by replacing Parabola?
Honestly: durable execution at scale; schema drift handling and enterprise controls; hundreds of maintained connectors; OAuth app verification. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Parabola?
Yes: KNIME Analytics Platform (A visual data workbench with hundreds of connectors; scheduled cloud runs are where KNIME starts charging.) The prompt is for when you want it exactly your way.