Build Relevance AI

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A runner for one automation whose hard problems are that a queue gives you at-least-once execution while the actions you are automating are not repeatable, that a model proposing a tool call is not the same as a tool call being valid, and that an agent with credentials and no ceiling is an unbounded liability.

estimated effort weekendthe files for this build are in the project pack

RuntimeNode 22, one web process and one worker processQueueBullMQ on Redis, with stalled jobs treated as a design inputStoragePostgreSQL holding runs, steps, attempts and tool callsToolsA registry of typed tools with JSON Schema on both sidesSecretsHeld by the adapter, never placed in a prompt or a stored inputCostPer-run accounting against a hard monthly ceiling

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the web process and the worker.
Get it nodejs.org or your version manager. open ↗
Verify node --version prints v22 or higher
installfree locally
Why BullMQ needs it, and the stalled-job behaviour that shapes phase three is a property of this queue rather than an abstraction.
Get it Your package manager, or a small managed instance. open ↗
Verify A worker picks up a queued job and a killed worker's job is re-delivered
installfree locally
Why Holds runs, steps, attempts and approvals, which need to survive every restart in this design.
Get it Your package manager or a managed instance.
Verify Migrations apply and a waiting run survives a restart
API keypay per call; the ceiling in this build is what keeps it small
Why The proposal step needs structured output, and a provider without it turns phase two into string parsing, which is the thing that phase exists to prevent.
Get it Create a key and set a provider-side spend limit as well as the ceiling in this build.
Verify One proposal returns arguments that validate against a tool schema
decidefree
Why Every design decision here is about a fixed list of typed steps, and a runner built without one specific workflow in mind becomes a connector library nobody finishes.
Get it Write the trigger, the steps, their inputs and which of them are irreversible.
Verify The written list matches the registry, including the reversible flags

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- workflows(id, name, definition_json, version, created_at) · a run pins the version it started under, so editing the workflow never rewrites history
- runs(id, workflow_id, workflow_version, trigger_kind, trigger_ref, state, started_at, ended_at, cost_minor)
- steps(id, run_id, ordinal, tool_name, input_json, output_json, state, idempotency_key, started_at, ended_at)
- attempts(id, step_id, attempt, outcome, error_code, error_excerpt, duration_ms, at) · including the attempts that stalled and were re-run by another worker
- tool_calls(id, step_id, request_id, external_ref, external_status, at) · the other side's identifier, which is the only proof the effect already happened
- approvals(step_id, requested_at, decided_at, decided_by, decision, reason) · irreversible steps stop here and wait
- secret_refs(name, provider, last_rotated_at) · names and metadata only, never a value, and never joined into a prompt
- A run is reconstructable from these rows alone. If you need a log file to explain what an agent did, the schema is wrong.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`DATABASE_URL`secret	required	`postgres://runner@localhost/runner`	Runs, steps, attempts, approvals. The whole audit trail.
`REDIS_URL`	required	`redis://localhost:6379`	The job queue. Losing it loses in-flight jobs, not history.
`MODEL_API_KEY`secret	required	`sk-...redacted`	The proposal step. Set a provider-side limit as well.
`MODEL_ID`	required	`a model identifier supporting tool calls`	Recorded on every run, because a different model proposes differently.
`MONTHLY_COST_CEILING`	required	`10.00`	New runs are refused at this figure. In-flight runs finish.
`WORKER_CONCURRENCY`	required	`1`	Start at one. Concurrency is how you discover your idempotency was wishful.
`APPROVAL_TIMEOUT_HOURS`	required	`48`	A waiting run expires here rather than sitting forever holding a claim.

The build, in order

1
The run log is the product
Every run writes enough rows that somebody can explain what happened without reading a log file.
1. A crash is exactly when you most want to know where it got to.
  Files src/runner/run.tsmigrations/002_runs.sql
2. Otherwise editing the workflow rewrites the past.
  Files src/runner/version.tstests/version-pin.test.ts
3. Inputs get exported, and exports get shared.
  Files src/runner/redact.tstests/redact.test.ts
done when · tick each as it passes
watch out
- Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run.
next phase
2
The model proposes, the runner decides
A tool only runs when its arguments validate against a schema the runner owns.
1. Output validation catches the API that changed under you.
  Files src/tools/registry.tssrc/tools/schema.ts
2. Two is a loop and a loop is a bill.
  Files src/runner/propose.tstests/repair-once.test.ts
3. The web page your agent reads is written by somebody else.
  Files src/runner/prompt.tstests/injection.test.ts
done when · tick each as it passes
watch out
- Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop.
next phase
3
Effects that happen once on a queue that runs twice
A step that sends, charges or writes to somebody else's system does it once even when the job is re-run.
1. A key allocated after the effect protects nothing.
  Files src/runner/idempotency.tstests/key-stability.test.ts
2. It is the only evidence the effect already happened.
  Files src/tools/adapter.ts
3. The failure you do not simulate is the one you ship.
  Files tests/stalled-replay.test.ts
done when · tick each as it passes
watch out
- BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not.
next phase
4
The steps that stop and ask
Anything irreversible waits for a human, and the wait is a first-class state rather than a paused script.
1. The classification is the point; the gate is easy once it exists.
  Files src/tools/registry.ts
2. Decisions arrive hours later, from other machines.
  Files src/runner/approvals.tstests/resume-after-restart.test.ts
3. Approving a template is not approving anything.
  Files src/app/approvals/page.tsx
done when · tick each as it passes
watch out
- Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine.
5
A ceiling, a replay and a way outproduct builder
The runner cannot spend without limit, any run can be replayed without repeating its effects, and everything exports.
1. A run that died halfway still cost money.
  Files src/runner/cost.tstests/cost-ceiling.test.ts
2. Replaying with live adapters is a self-inflicted incident.
  Files src/tools/registry.tstests/replay-safety.test.ts
3. Including attempts, which is where the arguments are.
  Files src/export/dump.ts
done when · tick each as it passes
watch out
- A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter.

what this build does not replace

A connector library.
A visual workflow builder.
Multi-user access with roles.
Agents that choose their own goals.
hundreds of maintained connectors
OAuth app verification
durable execution at scale
schema drift handling and enterprise controls

after v1, if you want it

A second workflow, which is the real test of whether the registry and the run schema were general enough
A diff view between two runs of the same workflow version, which finds prompt drift faster than reading outputs
Scheduled triggers with a jitter window, once you trust the idempotency enough to let it run unattended

Need the files? The project pack on the verdict page hands your agent the whole brief · more automation.