Build Parabola

0%0 of 18 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A step-by-step data pipeline whose hard problems are reading tabular files without silently changing the values, keeping every step deterministic and previewable, and running on a schedule against APIs that push back without ever writing half a result.

estimated effort multi-daythe files for this build are in the project pack

RuntimeTypeScript on Node 22, one worker process plus one web processStoragePostgreSQL for flow definitions and run history, files on disk for step outputsQueueBullMQ on Redis with one active run per flow enforced by a lockStepsPure functions from one table to one table, output materialised per stepTypesDeclared on the source, never inferred per row

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the worker and the web process.
Get it nodejs.org, or nvm install 22. open ↗
Verify node --version prints v22 or higher
installfree locally, a few pounds a month managed
Why Holds flow definitions and run history, which you will query far more than you expect once something fails at three in the morning.
Get it Docker, your package manager, or a managed instance. open ↗
Verify psql -c 'select version()' connects
installfree locally
Why The lock that stops two runs of one flow overlapping lives here, and overlapping runs are the duplicate-write bug.
Get it Docker or your package manager. open ↗
Verify redis-cli ping returns PONG
API keyfree for most, some charge per call
Why The whole point is moving your own data, and a pipeline with no real source is a demo.
Get it Create a scoped token in each service, with read on the source and the narrowest write on the destination.
Verify A single authenticated request succeeds from the command line
decidefree
Why The request rate you choose is only defensible if it comes from the vendor's current page, and these change.
Get it Read each vendor's rate limit documentation now and write down the figure with today's date.
Verify Each configured rate cites a source and a date in the README
have readyfree
Why Leading zeros, embedded newlines and an unexpected encoding are the failures that matter, and clean fixtures never show them.
Get it Export something from the system you actually use, without tidying it up first.
Verify It parses with a declared schema and its bad rows land in row_errors

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- flows(id, name, current_version, schedule_cron, timezone, enabled, created_at)
- flow_versions(flow_id, version, graph_json, created_at) · runs reference the version they executed, so an edit does not rewrite history
- runs(id, flow_id, flow_version, trigger, state, started_at, finished_at, rows_in, rows_out, error) · trigger is schedule, manual or replay
- step_runs(run_id, step_id, ordinal, state, rows_in, rows_out, rows_filtered, rows_errored, output_path, duration_ms)
- row_errors(run_id, step_id, row_ordinal, column_name, raw_value, reason) · the raw value is kept exactly as it arrived
- destinations(id, kind, credential_ref, write_mode, idempotency_column, last_run_id) · write_mode is replace, append or upsert
- Every step run satisfies rows_in equals rows_out plus rows_filtered plus rows_errored, and the suite asserts it, because a pipeline that quietly loses rows is worse than one that fails.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_URL`	required	`postgres://localhost:5432/pipelines`	Flows, versions, runs and row errors.
`REDIS_URL`	required	`redis://localhost:6379`	The queue and the per-flow run lock.
`DATA_DIR`	required	`./data/steps`	Materialised step outputs. Back this up with the database or a replay is impossible.
`STEP_OUTPUT_RETENTION_DAYS`	required	`30`	How long step outputs are kept for replay before deletion.
`MAX_ROWS_PER_RUN`	required	`500000`	A hard ceiling. A run that exceeds it stops and is marked capped.
`MAX_RUN_MINUTES`	required	`30`	The other ceiling, for the source that got slow rather than large.
`SOURCE_API_TOKEN`secret	required	`sk-...redacted`	Read scope only on the service you pull from.
`DESTINATION_API_TOKEN`secret	optional	`sk-...redacted`	The narrowest write scope on the destination, separate from the read token.

The build, in order

1
Reading a file without changing it
A tabular source is parsed to declared types, and anything that does not fit becomes a visible error rather than a guess.
1. One declared type per column, applied to every row, with no per-row inference anywhere.
  Files src/steps/source-csv.tssrc/schema.ts
2. Hand-rolled splitting on commas breaks on the first field containing one.
  Files src/csv/parse.tstests/csv-quoting.test.ts
3. The raw text is the only thing that lets someone fix the source file.
  Files src/steps/source-csv.tstests/leading-zeros.test.ts
done when · tick each as it passes
watch out
- Inferring column types from the first few rows reads the order number 007634 as the integer 7634 and a postcode as a float, and nobody notices until a customer cannot be found; the same inference reads a date column as text in one file and as a date in the next because row four happened to be blank. Declare the type per column on the source, parse strictly against it, and route failures to row_errors instead of coercing.
next phase
2
Steps you can look inside
Every step materialises its output and accounts for every row that entered it.
1. The debugging story for a pipeline is being able to see the table between any two steps.
  Files src/engine/run.tssrc/engine/materialise.ts
2. Rows in equals rows out plus filtered plus errored, for every step, every time.
  Files src/engine/step-run.tstests/row-accounting.test.ts
3. A join that matches nothing looks identical to a filter that worked.
  Files src/steps/join.tstests/join-types.test.ts
done when · tick each as it passes
watch out
- Chaining steps in memory and only showing the final table means a join that silently drops nine tenths of the rows because one side's key is text and the other's is an integer looks exactly like a join that worked. Materialise every step, show its row counts next to it, and fail a join whose key types differ rather than coercing them.
next phase
3
A schedule that does not run twice
Scheduled runs fire in a named timezone, never overlap, and survive a restart mid-run.
1. Not by adding a fixed interval, which is wrong twice a year.
  Files src/schedule/next.tstests/dst.test.ts
2. A skipped tick you can see beats an overlapping run you cannot.
  Files src/queue/lock.ts
3. Automatic resumption is how half a file gets written twice.
  Files src/engine/recover.ts
done when · tick each as it passes
watch out
- Computing the next fire time by adding twenty four hours to the last one drifts by an hour twice a year and fires twice on the repeated hour, and a daily export that ran twice means a destination with every row duplicated. Compute each fire from the zone's rules for that date, and decide explicitly what a schedule landing in the missing hour does.
next phase
4
APIs that push back
A paged API source finishes or fails cleanly, and a destination is never half written.
1. Guessing a backoff when the server told you the number is how an account gets throttled for a day.
  Files src/http/client.tstests/rate-limit.test.ts
2. A half-written destination is the failure mode with no clean recovery.
  Files src/destinations/stage.ts
3. So a retry is an update and not a duplicate row.
  Files src/destinations/write.tstests/idempotent-write.test.ts
done when · tick each as it passes
watch out
- Writing each page to the destination as it arrives means a failure on page seven leaves six pages of new rows in a live system and no safe way to retry, since rerunning writes the first six again. Stage the whole result first, write once, and key the write on an identifier from the data so a repeat is an update.
5
Reading the failure and taking the data outproduct builder
Every run is explainable after the fact and nothing in the system is trapped.
1. The raw value in row_errors is what the person needs to fix the source.
  Files src/web/routes/run.tsx
2. Re-reading a source that has since changed is not a replay.
  Files src/engine/replay.tstests/replay.test.ts
3. A ceiling stops the runaway; transition alerts stop the muting.
  Files src/engine/limits.tssrc/alerts.ts
done when · tick each as it passes
watch out
- A nightly flow that starts failing sends an identical alert every night until the person mutes the channel, and then the real failure is invisible; meanwhile a source file that doubled in size runs for six hours and nobody stops it. Alert on the transition into failure and on recovery, and give every run a hard ceiling on rows and duration.

what this build does not replace

A connector library.
Verified OAuth applications.
Multi-user workspaces and approvals.
Automatic schema drift handling.
durable execution at scale
schema drift handling and enterprise controls
hundreds of maintained connectors
OAuth app verification

after v1, if you want it

A dry run mode that executes every step and stops before the destination write
Column-level lineage, which the materialised outputs already make possible
A second destination kind, once one has run for a month without a duplicate

Need the files? The project pack on the verdict page hands your agent the whole brief · more automation.