Vibecode Relevance AI
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The visible AI workforce platform loop is buildable, but a credible replacement needs more than the first screen. Relevance AI earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Relevance AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Relevance AI · indie build A runner for one automation whose hard problems are that a queue gives you at-least-once execution while the actions you are automating are not repeatable, that a model proposing a tool call is not the same as a tool call being valid, and that an agent with credentials and no ceiling is an unbounded liability. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22, one web process and one worker process | the worker is where everything interesting and everything dangerous happens | | Queue | BullMQ on Redis, with stalled jobs treated as a design input | a job whose worker died is re-run by another worker, and your build has to survive that | | Storage | PostgreSQL holding runs, steps, attempts and tool calls | if reading the database does not explain a run, the run cannot be defended | | Tools | A registry of typed tools with JSON Schema on both sides | the model proposes and the runner validates, which is the whole safety story | | Secrets | Held by the adapter, never placed in a prompt or a stored input | a credential inside a model context is a credential in a log somewhere | | Cost | Per-run accounting against a hard monthly ceiling | an agent in a loop is the most expensive bug available to you | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the web process and the worker. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **Redis** · free locally - Why: BullMQ needs it, and the stalled-job behaviour that shapes phase three is a property of this queue rather than an abstraction. - Get it: Your package manager, or a small managed instance. - Verify: A worker picks up a queued job and a killed worker's job is re-delivered - [ ] **PostgreSQL 16 or newer** · free locally - Why: Holds runs, steps, attempts and approvals, which need to survive every restart in this design. - Get it: Your package manager or a managed instance. - Verify: Migrations apply and a waiting run survives a restart - [ ] **An API key for a model that supports structured tool calling** · pay per call; the ceiling in this build is what keeps it small - Why: The proposal step needs structured output, and a provider without it turns phase two into string parsing, which is the thing that phase exists to prevent. - Get it: Create a key and set a provider-side spend limit as well as the ceiling in this build. - Verify: One proposal returns arguments that validate against a tool schema - [ ] **The one workflow this runner automates, written down before any code** · free - Why: Every design decision here is about a fixed list of typed steps, and a runner built without one specific workflow in mind becomes a connector library nobody finishes. - Get it: Write the trigger, the steps, their inputs and which of them are irreversible. - Verify: The written list matches the registry, including the reversible flags ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A connector library. - A visual workflow builder. - Multi-user access with roles. - Agents that choose their own goals. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls If one of those is essential to you, that is the reason to keep paying for Relevance AI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Relevance AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a single-workflow agent runner whose runs are replayable from the database, whose side effects happen once, and whose spending has a ceiling to replace Relevance AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22, one web process and one worker process · the worker is where everything interesting and everything dangerous happens - Queue: BullMQ on Redis, with stalled jobs treated as a design input · a job whose worker died is re-run by another worker, and your build has to survive that - Storage: PostgreSQL holding runs, steps, attempts and tool calls · if reading the database does not explain a run, the run cannot be defended - Tools: A registry of typed tools with JSON Schema on both sides · the model proposes and the runner validates, which is the whole safety story - Secrets: Held by the adapter, never placed in a prompt or a stored input · a credential inside a model context is a credential in a log somewhere - Cost: Per-run accounting against a hard monthly ceiling · an agent in a loop is the most expensive bug available to you ### Data model (create this before Phase 1) - workflows(id, name, definition_json, version, created_at) · a run pins the version it started under, so editing the workflow never rewrites history - runs(id, workflow_id, workflow_version, trigger_kind, trigger_ref, state, started_at, ended_at, cost_minor) - steps(id, run_id, ordinal, tool_name, input_json, output_json, state, idempotency_key, started_at, ended_at) - attempts(id, step_id, attempt, outcome, error_code, error_excerpt, duration_ms, at) · including the attempts that stalled and were re-run by another worker - tool_calls(id, step_id, request_id, external_ref, external_status, at) · the other side's identifier, which is the only proof the effect already happened - approvals(step_id, requested_at, decided_at, decided_by, decision, reason) · irreversible steps stop here and wait - secret_refs(name, provider, last_rotated_at) · names and metadata only, never a value, and never joined into a prompt - A run is reconstructable from these rows alone. If you need a log file to explain what an agent did, the schema is wrong. ### Phase 1 · The run log is the product Build: Define one workflow as data: an ordered list of typed steps with inputs mapped from earlier outputs. On trigger, create a run pinned to the current workflow version, then one step row per stage as it begins. Record each attempt separately with its outcome, duration and a truncated error. Keep inputs and outputs as JSON on the step, redacting anything named as a secret reference. Trap: Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. Done when: Editing a workflow does not change how an existing run reads back, tested by editing the definition and replaying the query. A run interrupted halfway leaves step rows in a started state rather than no rows at all, tested by killing the worker. Every attempt appears as its own row with an outcome and a duration, including failures. No value referenced through secret_refs appears in any stored input or output, tested with a fixture containing one. Do not build yet: No tool validation, no idempotency, no approvals, no ceiling yet. ### Phase 2 · The model proposes, the runner decides Build: Give every tool a JSON Schema for its input and its output. Ask the model for a structured tool call, validate it against the schema, and on failure return the validation errors for exactly one repair attempt before failing the step. Never execute a tool whose arguments did not validate, and never accept a tool name that is not in the registry. Trap: Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. Done when: A tool call with an unknown tool name is refused without an HTTP request, tested. A call failing schema validation triggers exactly one repair attempt and then fails the step, tested with a stub that never returns valid arguments. Output validation runs too, and a tool returning a shape that does not match its schema fails the step rather than passing a surprise to the next one, tested. Text arriving from a fetched document is never treated as an instruction, checked by a test containing an injected instruction that must be ignored. Do not build yet: No idempotency, no approvals, no replay, no ceiling yet. ### Phase 3 · Effects that happen once on a queue that runs twice Build: Derive a deterministic idempotency key per step from the run identifier, the step ordinal and a hash of the resolved input. Store it before the adapter is called. Each adapter either passes the key to the remote API or records the remote identifier in tool_calls and checks for it before acting. On restart, a step with a recorded external reference is complete, not pending. Trap: BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. Done when: A step interrupted between the outbound call and the database write does not repeat its effect when re-run, tested by killing the worker at that point against a stub that counts calls. The idempotency key is a pure function of run, ordinal and input hash, tested for stability across process restarts. Adapters for irreversible actions refuse to run without either a remote idempotency mechanism or a pre-check, asserted in the suite. A stalled job that returns after maxStalledCount is failed with its history intact rather than silently dropped. Do not build yet: No approvals, no replay interface, no export, no ceiling yet. ### Phase 4 · The steps that stop and ask Build: Mark tools in the registry as reversible or not. An irreversible step creates an approval row and the run enters a waiting state that survives a restart. Approving resumes from that step with the same idempotency key. Rejecting ends the run with the reason recorded. Include what the step will actually do, rendered from the validated arguments, in the approval request. Trap: Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. Done when: A run waiting for approval survives a full restart of the web and worker processes and is still resumable, tested. The approval request shows the resolved arguments rather than the template, checked in the interface test. Approving resumes with the same idempotency key so an approved step cannot double-act, tested. Rejecting records the reason and ends the run, and no later step executes, tested. Do not build yet: No spend ceiling, no replay tooling, no export yet. ### Phase 5 · A ceiling, a replay and a way out Build: Account model tokens and per-call costs onto the run as it goes. When the monthly total reaches the ceiling, refuse to start new runs and say so, leaving in-flight runs to finish. Add a replay that re-executes a run against recorded inputs in a mode where every irreversible adapter is a stub. Export runs, steps, attempts and tool calls as newline-delimited JSON. Trap: A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter. Done when: Reaching the monthly ceiling refuses new runs with a named reason and lets in-flight runs complete, tested with a moved counter. Replay substitutes every irreversible adapter and a live adapter reached during replay raises rather than acts, tested. Per-run cost is recorded as the run proceeds rather than computed at the end, so a run killed halfway still shows what it spent, tested. Export round-trips into an empty database and the replayed outputs match the recorded ones for deterministic tools. Do not build yet: Nothing after this. ### Out of scope (and why) - A connector library, because each integration is an OAuth application, a review process and a maintenance commitment, and two hand-written adapters beat forty half-working ones. - A visual workflow builder, which is a substantial interface project and adds nothing to the reliability problems this build is actually about. - Multi-user access with roles, since an agent runner holding credentials needs a real permissions model rather than a bolted-on one. - Autonomous agents that choose their own goals, because every safety property here rests on the workflow being a fixed list of typed steps you wrote down. ### README must contain - That the queue provides at-least-once execution and that exactly-once effects come from per-step idempotency keys, not from the queue - That every tool has a schema on both sides and that a model proposing a call is never enough to execute one - Which tools are marked irreversible, and that those steps wait for a written approval that survives a restart - That secrets live in adapters and never enter a prompt, a stored input or an export - The monthly spend ceiling, what happens when it is reached, and how replay avoids repeating effects ===== AGENTS.md ===== # Agent instructions · Relevance AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22, one web process and one worker process, BullMQ on Redis, with stalled jobs treated as a design input, PostgreSQL holding runs, steps, attempts and tool calls, A registry of typed tools with JSON Schema on both sides, Held by the adapter, never placed in a prompt or a stored input, Per-run accounting against a hard monthly ceiling. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. ===== BUILD_PLAN.md ===== # Build plan · Relevance AI A runner for one automation whose hard problems are that a queue gives you at-least-once execution while the actions you are automating are not repeatable, that a model proposing a tool call is not the same as a tool call being valid, and that an agent with credentials and no ceiling is an unbounded liability. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · The run log is the product Every run writes enough rows that somebody can explain what happened without reading a log file. ### Steps 1. Write step rows at start, not at finish A crash is exactly when you most want to know where it got to. Files: `src/runner/run.ts`, `migrations/002_runs.sql` 2. Pin the workflow version on the run Otherwise editing the workflow rewrites the past. Files: `src/runner/version.ts`, `tests/version-pin.test.ts` 3. Redact secret references from stored inputs Inputs get exported, and exports get shared. Files: `src/runner/redact.ts`, `tests/redact.test.ts` ### Done when - [ ] Editing a workflow leaves old runs unchanged - [ ] An interrupted run leaves a started step row - [ ] Every attempt is its own row with an outcome - [ ] No secret value appears in a stored input or output ### Watch out - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. ## Phase 2 · The model proposes, the runner decides A tool only runs when its arguments validate against a schema the runner owns. ### Steps 1. Schema on both sides of every tool Output validation catches the API that changed under you. Files: `src/tools/registry.ts`, `src/tools/schema.ts` 2. One repair attempt, with the errors quoted Two is a loop and a loop is a bill. Files: `src/runner/propose.ts`, `tests/repair-once.test.ts` 3. Treat fetched content as data, never as instructions The web page your agent reads is written by somebody else. Files: `src/runner/prompt.ts`, `tests/injection.test.ts` ### Done when - [ ] An unknown tool name never reaches the network - [ ] Invalid arguments get one repair, then fail - [ ] A tool output that breaks its schema fails the step - [ ] Injected instructions in fetched text are ignored ### Watch out - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. ## Phase 3 · Effects that happen once on a queue that runs twice A step that sends, charges or writes to somebody else's system does it once even when the job is re-run. ### Steps 1. Derive the key before the adapter is called A key allocated after the effect protects nothing. Files: `src/runner/idempotency.ts`, `tests/key-stability.test.ts` 2. Record the remote identifier in tool_calls It is the only evidence the effect already happened. Files: `src/tools/adapter.ts` 3. Kill the worker mid-step in a test The failure you do not simulate is the one you ship. Files: `tests/stalled-replay.test.ts` ### Done when - [ ] A killed worker does not repeat a step's effect - [ ] Keys are stable across restarts - [ ] Irreversible adapters refuse without a pre-check - [ ] An exhausted stalled job fails with its history ### Watch out - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. ## Phase 4 · The steps that stop and ask Anything irreversible waits for a human, and the wait is a first-class state rather than a paused script. ### Steps 1. Mark tools reversible or not in the registry The classification is the point; the gate is easy once it exists. Files: `src/tools/registry.ts` 2. Make waiting a database state, not a sleep Decisions arrive hours later, from other machines. Files: `src/runner/approvals.ts`, `tests/resume-after-restart.test.ts` 3. Show the resolved arguments in the request Approving a template is not approving anything. Files: `src/app/approvals/page.tsx` ### Done when - [ ] A waiting run survives a restart - [ ] The request shows resolved arguments - [ ] Approval resumes with the same idempotency key - [ ] Rejection records a reason and stops the run ### Watch out - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. ## Not in this build - A connector library. - A visual workflow builder. - Multi-user access with roles. - Agents that choose their own goals. ## After v1, if you want it - A second workflow, which is the real test of whether the registry and the run schema were general enough - A diff view between two runs of the same workflow version, which finds prompt drift faster than reading outputs - Scheduled triggers with a jitter window, once you trust the idempotency enough to let it run unattended ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Runs, steps, attempts, approvals. The whole audit trail. DATABASE_URL=postgres://runner@localhost/runner # Required. The job queue. Losing it loses in-flight jobs, not history. REDIS_URL=redis://localhost:6379 # Required · secret. The proposal step. Set a provider-side limit as well. MODEL_API_KEY=sk-...redacted # Required. Recorded on every run, because a different model proposes differently. MODEL_ID=a model identifier supporting tool calls # Required. New runs are refused at this figure. In-flight runs finish. MONTHLY_COST_CEILING=10.00 # Required. Start at one. Concurrency is how you discover your idempotency was wishful. WORKER_CONCURRENCY=1 # Required. A waiting run expires here rather than sitting forever holding a claim. APPROVAL_TIMEOUT_HOURS=48
You are building a lean indie version of Relevance AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Relevance AI · indie build A runner for one automation whose hard problems are that a queue gives you at-least-once execution while the actions you are automating are not repeatable, that a model proposing a tool call is not the same as a tool call being valid, and that an agent with credentials and no ceiling is an unbounded liability. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22, one web process and one worker process | the worker is where everything interesting and everything dangerous happens | | Queue | BullMQ on Redis, with stalled jobs treated as a design input | a job whose worker died is re-run by another worker, and your build has to survive that | | Storage | PostgreSQL holding runs, steps, attempts and tool calls | if reading the database does not explain a run, the run cannot be defended | | Tools | A registry of typed tools with JSON Schema on both sides | the model proposes and the runner validates, which is the whole safety story | | Secrets | Held by the adapter, never placed in a prompt or a stored input | a credential inside a model context is a credential in a log somewhere | | Cost | Per-run accounting against a hard monthly ceiling | an agent in a loop is the most expensive bug available to you | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the web process and the worker. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **Redis** · free locally - Why: BullMQ needs it, and the stalled-job behaviour that shapes phase three is a property of this queue rather than an abstraction. - Get it: Your package manager, or a small managed instance. - Verify: A worker picks up a queued job and a killed worker's job is re-delivered - [ ] **PostgreSQL 16 or newer** · free locally - Why: Holds runs, steps, attempts and approvals, which need to survive every restart in this design. - Get it: Your package manager or a managed instance. - Verify: Migrations apply and a waiting run survives a restart - [ ] **An API key for a model that supports structured tool calling** · pay per call; the ceiling in this build is what keeps it small - Why: The proposal step needs structured output, and a provider without it turns phase two into string parsing, which is the thing that phase exists to prevent. - Get it: Create a key and set a provider-side spend limit as well as the ceiling in this build. - Verify: One proposal returns arguments that validate against a tool schema - [ ] **The one workflow this runner automates, written down before any code** · free - Why: Every design decision here is about a fixed list of typed steps, and a runner built without one specific workflow in mind becomes a connector library nobody finishes. - Get it: Write the trigger, the steps, their inputs and which of them are irreversible. - Verify: The written list matches the registry, including the reversible flags ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A connector library. - A visual workflow builder. - Multi-user access with roles. - Agents that choose their own goals. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls If one of those is essential to you, that is the reason to keep paying for Relevance AI, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Relevance AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a single-workflow agent runner whose runs are replayable from the database, whose side effects happen once, and whose spending has a ceiling to replace Relevance AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22, one web process and one worker process · the worker is where everything interesting and everything dangerous happens - Queue: BullMQ on Redis, with stalled jobs treated as a design input · a job whose worker died is re-run by another worker, and your build has to survive that - Storage: PostgreSQL holding runs, steps, attempts and tool calls · if reading the database does not explain a run, the run cannot be defended - Tools: A registry of typed tools with JSON Schema on both sides · the model proposes and the runner validates, which is the whole safety story - Secrets: Held by the adapter, never placed in a prompt or a stored input · a credential inside a model context is a credential in a log somewhere - Cost: Per-run accounting against a hard monthly ceiling · an agent in a loop is the most expensive bug available to you ### Data model (create this before Phase 1) - workflows(id, name, definition_json, version, created_at) · a run pins the version it started under, so editing the workflow never rewrites history - runs(id, workflow_id, workflow_version, trigger_kind, trigger_ref, state, started_at, ended_at, cost_minor) - steps(id, run_id, ordinal, tool_name, input_json, output_json, state, idempotency_key, started_at, ended_at) - attempts(id, step_id, attempt, outcome, error_code, error_excerpt, duration_ms, at) · including the attempts that stalled and were re-run by another worker - tool_calls(id, step_id, request_id, external_ref, external_status, at) · the other side's identifier, which is the only proof the effect already happened - approvals(step_id, requested_at, decided_at, decided_by, decision, reason) · irreversible steps stop here and wait - secret_refs(name, provider, last_rotated_at) · names and metadata only, never a value, and never joined into a prompt - A run is reconstructable from these rows alone. If you need a log file to explain what an agent did, the schema is wrong. ### Phase 1 · The run log is the product Build: Define one workflow as data: an ordered list of typed steps with inputs mapped from earlier outputs. On trigger, create a run pinned to the current workflow version, then one step row per stage as it begins. Record each attempt separately with its outcome, duration and a truncated error. Keep inputs and outputs as JSON on the step, redacting anything named as a secret reference. Trap: Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. Done when: Editing a workflow does not change how an existing run reads back, tested by editing the definition and replaying the query. A run interrupted halfway leaves step rows in a started state rather than no rows at all, tested by killing the worker. Every attempt appears as its own row with an outcome and a duration, including failures. No value referenced through secret_refs appears in any stored input or output, tested with a fixture containing one. Do not build yet: No tool validation, no idempotency, no approvals, no ceiling yet. ### Phase 2 · The model proposes, the runner decides Build: Give every tool a JSON Schema for its input and its output. Ask the model for a structured tool call, validate it against the schema, and on failure return the validation errors for exactly one repair attempt before failing the step. Never execute a tool whose arguments did not validate, and never accept a tool name that is not in the registry. Trap: Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. Done when: A tool call with an unknown tool name is refused without an HTTP request, tested. A call failing schema validation triggers exactly one repair attempt and then fails the step, tested with a stub that never returns valid arguments. Output validation runs too, and a tool returning a shape that does not match its schema fails the step rather than passing a surprise to the next one, tested. Text arriving from a fetched document is never treated as an instruction, checked by a test containing an injected instruction that must be ignored. Do not build yet: No idempotency, no approvals, no replay, no ceiling yet. ### Phase 3 · Effects that happen once on a queue that runs twice Build: Derive a deterministic idempotency key per step from the run identifier, the step ordinal and a hash of the resolved input. Store it before the adapter is called. Each adapter either passes the key to the remote API or records the remote identifier in tool_calls and checks for it before acting. On restart, a step with a recorded external reference is complete, not pending. Trap: BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. Done when: A step interrupted between the outbound call and the database write does not repeat its effect when re-run, tested by killing the worker at that point against a stub that counts calls. The idempotency key is a pure function of run, ordinal and input hash, tested for stability across process restarts. Adapters for irreversible actions refuse to run without either a remote idempotency mechanism or a pre-check, asserted in the suite. A stalled job that returns after maxStalledCount is failed with its history intact rather than silently dropped. Do not build yet: No approvals, no replay interface, no export, no ceiling yet. ### Phase 4 · The steps that stop and ask Build: Mark tools in the registry as reversible or not. An irreversible step creates an approval row and the run enters a waiting state that survives a restart. Approving resumes from that step with the same idempotency key. Rejecting ends the run with the reason recorded. Include what the step will actually do, rendered from the validated arguments, in the approval request. Trap: Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. Done when: A run waiting for approval survives a full restart of the web and worker processes and is still resumable, tested. The approval request shows the resolved arguments rather than the template, checked in the interface test. Approving resumes with the same idempotency key so an approved step cannot double-act, tested. Rejecting records the reason and ends the run, and no later step executes, tested. Do not build yet: No spend ceiling, no replay tooling, no export yet. ### Phase 5 · A ceiling, a replay and a way out Build: Account model tokens and per-call costs onto the run as it goes. When the monthly total reaches the ceiling, refuse to start new runs and say so, leaving in-flight runs to finish. Add a replay that re-executes a run against recorded inputs in a mode where every irreversible adapter is a stub. Export runs, steps, attempts and tool calls as newline-delimited JSON. Trap: A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter. Done when: Reaching the monthly ceiling refuses new runs with a named reason and lets in-flight runs complete, tested with a moved counter. Replay substitutes every irreversible adapter and a live adapter reached during replay raises rather than acts, tested. Per-run cost is recorded as the run proceeds rather than computed at the end, so a run killed halfway still shows what it spent, tested. Export round-trips into an empty database and the replayed outputs match the recorded ones for deterministic tools. Do not build yet: Nothing after this. ### Out of scope (and why) - A connector library, because each integration is an OAuth application, a review process and a maintenance commitment, and two hand-written adapters beat forty half-working ones. - A visual workflow builder, which is a substantial interface project and adds nothing to the reliability problems this build is actually about. - Multi-user access with roles, since an agent runner holding credentials needs a real permissions model rather than a bolted-on one. - Autonomous agents that choose their own goals, because every safety property here rests on the workflow being a fixed list of typed steps you wrote down. ### README must contain - That the queue provides at-least-once execution and that exactly-once effects come from per-step idempotency keys, not from the queue - That every tool has a schema on both sides and that a model proposing a call is never enough to execute one - Which tools are marked irreversible, and that those steps wait for a written approval that survives a restart - That secrets live in adapters and never enter a prompt, a stored input or an export - The monthly spend ceiling, what happens when it is reached, and how replay avoids repeating effects ===== AGENTS.md ===== # Agent instructions · Relevance AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22, one web process and one worker process, BullMQ on Redis, with stalled jobs treated as a design input, PostgreSQL holding runs, steps, attempts and tool calls, A registry of typed tools with JSON Schema on both sides, Held by the adapter, never placed in a prompt or a stored input, Per-run accounting against a hard monthly ceiling. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. ===== BUILD_PLAN.md ===== # Build plan · Relevance AI A runner for one automation whose hard problems are that a queue gives you at-least-once execution while the actions you are automating are not repeatable, that a model proposing a tool call is not the same as a tool call being valid, and that an agent with credentials and no ceiling is an unbounded liability. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · The run log is the product Every run writes enough rows that somebody can explain what happened without reading a log file. ### Steps 1. Write step rows at start, not at finish A crash is exactly when you most want to know where it got to. Files: `src/runner/run.ts`, `migrations/002_runs.sql` 2. Pin the workflow version on the run Otherwise editing the workflow rewrites the past. Files: `src/runner/version.ts`, `tests/version-pin.test.ts` 3. Redact secret references from stored inputs Inputs get exported, and exports get shared. Files: `src/runner/redact.ts`, `tests/redact.test.ts` ### Done when - [ ] Editing a workflow leaves old runs unchanged - [ ] An interrupted run leaves a started step row - [ ] Every attempt is its own row with an outcome - [ ] No secret value appears in a stored input or output ### Watch out - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. ## Phase 2 · The model proposes, the runner decides A tool only runs when its arguments validate against a schema the runner owns. ### Steps 1. Schema on both sides of every tool Output validation catches the API that changed under you. Files: `src/tools/registry.ts`, `src/tools/schema.ts` 2. One repair attempt, with the errors quoted Two is a loop and a loop is a bill. Files: `src/runner/propose.ts`, `tests/repair-once.test.ts` 3. Treat fetched content as data, never as instructions The web page your agent reads is written by somebody else. Files: `src/runner/prompt.ts`, `tests/injection.test.ts` ### Done when - [ ] An unknown tool name never reaches the network - [ ] Invalid arguments get one repair, then fail - [ ] A tool output that breaks its schema fails the step - [ ] Injected instructions in fetched text are ignored ### Watch out - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. ## Phase 3 · Effects that happen once on a queue that runs twice A step that sends, charges or writes to somebody else's system does it once even when the job is re-run. ### Steps 1. Derive the key before the adapter is called A key allocated after the effect protects nothing. Files: `src/runner/idempotency.ts`, `tests/key-stability.test.ts` 2. Record the remote identifier in tool_calls It is the only evidence the effect already happened. Files: `src/tools/adapter.ts` 3. Kill the worker mid-step in a test The failure you do not simulate is the one you ship. Files: `tests/stalled-replay.test.ts` ### Done when - [ ] A killed worker does not repeat a step's effect - [ ] Keys are stable across restarts - [ ] Irreversible adapters refuse without a pre-check - [ ] An exhausted stalled job fails with its history ### Watch out - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. ## Phase 4 · The steps that stop and ask Anything irreversible waits for a human, and the wait is a first-class state rather than a paused script. ### Steps 1. Mark tools reversible or not in the registry The classification is the point; the gate is easy once it exists. Files: `src/tools/registry.ts` 2. Make waiting a database state, not a sleep Decisions arrive hours later, from other machines. Files: `src/runner/approvals.ts`, `tests/resume-after-restart.test.ts` 3. Show the resolved arguments in the request Approving a template is not approving anything. Files: `src/app/approvals/page.tsx` ### Done when - [ ] A waiting run survives a restart - [ ] The request shows resolved arguments - [ ] Approval resumes with the same idempotency key - [ ] Rejection records a reason and stops the run ### Watch out - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. ## Not in this build - A connector library. - A visual workflow builder. - Multi-user access with roles. - Agents that choose their own goals. ## After v1, if you want it - A second workflow, which is the real test of whether the registry and the run schema were general enough - A diff view between two runs of the same workflow version, which finds prompt drift faster than reading outputs - Scheduled triggers with a jitter window, once you trust the idempotency enough to let it run unattended ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Runs, steps, attempts, approvals. The whole audit trail. DATABASE_URL=postgres://runner@localhost/runner # Required. The job queue. Losing it loses in-flight jobs, not history. REDIS_URL=redis://localhost:6379 # Required · secret. The proposal step. Set a provider-side limit as well. MODEL_API_KEY=sk-...redacted # Required. Recorded on every run, because a different model proposes differently. MODEL_ID=a model identifier supporting tool calls # Required. New runs are refused at this figure. In-flight runs finish. MONTHLY_COST_CEILING=10.00 # Required. Start at one. Concurrency is how you discover your idempotency was wishful. WORKER_CONCURRENCY=1 # Required. A waiting run expires here rather than sitting forever holding a claim. APPROVAL_TIMEOUT_HOURS=48
You are building a production product version of Relevance AI. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Relevance AI · product brief ## Problem The visible AI workforce platform loop is buildable, but a credible replacement needs more than the first screen. Relevance AI earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A runner for one automation that can be explained from its own tables, that validates every tool call before executing it, that sends the email once even when the worker died, that stops and asks before anything irreversible, and that cannot spend past a number you set. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - hosted PostgreSQL and Redis - API credentials for the two chosen services - Node.js 22 ## Explicit non-goals for v1 - A connector library. - A visual workflow builder. - Multi-user access with roles. - Agents that choose their own goals. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls ## Success criteria - A killed worker is confirmed not to repeat a step's effect - An invalid tool call is confirmed never to reach the network - A waiting approval is confirmed to survive a restart - Replay is confirmed unable to reach a live irreversible adapter - The monthly ceiling is confirmed to refuse new runs ===== BRIEF.md ===== # Build brief · Relevance AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a single-workflow agent runner whose runs are replayable from the database, whose side effects happen once, and whose spending has a ceiling to replace Relevance AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22, one web process and one worker process · the worker is where everything interesting and everything dangerous happens - Queue: BullMQ on Redis, with stalled jobs treated as a design input · a job whose worker died is re-run by another worker, and your build has to survive that - Storage: PostgreSQL holding runs, steps, attempts and tool calls · if reading the database does not explain a run, the run cannot be defended - Tools: A registry of typed tools with JSON Schema on both sides · the model proposes and the runner validates, which is the whole safety story - Secrets: Held by the adapter, never placed in a prompt or a stored input · a credential inside a model context is a credential in a log somewhere - Cost: Per-run accounting against a hard monthly ceiling · an agent in a loop is the most expensive bug available to you ### Data model (create this before Phase 1) - workflows(id, name, definition_json, version, created_at) · a run pins the version it started under, so editing the workflow never rewrites history - runs(id, workflow_id, workflow_version, trigger_kind, trigger_ref, state, started_at, ended_at, cost_minor) - steps(id, run_id, ordinal, tool_name, input_json, output_json, state, idempotency_key, started_at, ended_at) - attempts(id, step_id, attempt, outcome, error_code, error_excerpt, duration_ms, at) · including the attempts that stalled and were re-run by another worker - tool_calls(id, step_id, request_id, external_ref, external_status, at) · the other side's identifier, which is the only proof the effect already happened - approvals(step_id, requested_at, decided_at, decided_by, decision, reason) · irreversible steps stop here and wait - secret_refs(name, provider, last_rotated_at) · names and metadata only, never a value, and never joined into a prompt - A run is reconstructable from these rows alone. If you need a log file to explain what an agent did, the schema is wrong. ### Phase 1 · The run log is the product Build: Define one workflow as data: an ordered list of typed steps with inputs mapped from earlier outputs. On trigger, create a run pinned to the current workflow version, then one step row per stage as it begins. Record each attempt separately with its outcome, duration and a truncated error. Keep inputs and outputs as JSON on the step, redacting anything named as a secret reference. Trap: Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. Done when: Editing a workflow does not change how an existing run reads back, tested by editing the definition and replaying the query. A run interrupted halfway leaves step rows in a started state rather than no rows at all, tested by killing the worker. Every attempt appears as its own row with an outcome and a duration, including failures. No value referenced through secret_refs appears in any stored input or output, tested with a fixture containing one. Do not build yet: No tool validation, no idempotency, no approvals, no ceiling yet. ### Phase 2 · The model proposes, the runner decides Build: Give every tool a JSON Schema for its input and its output. Ask the model for a structured tool call, validate it against the schema, and on failure return the validation errors for exactly one repair attempt before failing the step. Never execute a tool whose arguments did not validate, and never accept a tool name that is not in the registry. Trap: Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. Done when: A tool call with an unknown tool name is refused without an HTTP request, tested. A call failing schema validation triggers exactly one repair attempt and then fails the step, tested with a stub that never returns valid arguments. Output validation runs too, and a tool returning a shape that does not match its schema fails the step rather than passing a surprise to the next one, tested. Text arriving from a fetched document is never treated as an instruction, checked by a test containing an injected instruction that must be ignored. Do not build yet: No idempotency, no approvals, no replay, no ceiling yet. ### Phase 3 · Effects that happen once on a queue that runs twice Build: Derive a deterministic idempotency key per step from the run identifier, the step ordinal and a hash of the resolved input. Store it before the adapter is called. Each adapter either passes the key to the remote API or records the remote identifier in tool_calls and checks for it before acting. On restart, a step with a recorded external reference is complete, not pending. Trap: BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. Done when: A step interrupted between the outbound call and the database write does not repeat its effect when re-run, tested by killing the worker at that point against a stub that counts calls. The idempotency key is a pure function of run, ordinal and input hash, tested for stability across process restarts. Adapters for irreversible actions refuse to run without either a remote idempotency mechanism or a pre-check, asserted in the suite. A stalled job that returns after maxStalledCount is failed with its history intact rather than silently dropped. Do not build yet: No approvals, no replay interface, no export, no ceiling yet. ### Phase 4 · The steps that stop and ask Build: Mark tools in the registry as reversible or not. An irreversible step creates an approval row and the run enters a waiting state that survives a restart. Approving resumes from that step with the same idempotency key. Rejecting ends the run with the reason recorded. Include what the step will actually do, rendered from the validated arguments, in the approval request. Trap: Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. Done when: A run waiting for approval survives a full restart of the web and worker processes and is still resumable, tested. The approval request shows the resolved arguments rather than the template, checked in the interface test. Approving resumes with the same idempotency key so an approved step cannot double-act, tested. Rejecting records the reason and ends the run, and no later step executes, tested. Do not build yet: No spend ceiling, no replay tooling, no export yet. ### Phase 5 · A ceiling, a replay and a way out Build: Account model tokens and per-call costs onto the run as it goes. When the monthly total reaches the ceiling, refuse to start new runs and say so, leaving in-flight runs to finish. Add a replay that re-executes a run against recorded inputs in a mode where every irreversible adapter is a stub. Export runs, steps, attempts and tool calls as newline-delimited JSON. Trap: A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter. Done when: Reaching the monthly ceiling refuses new runs with a named reason and lets in-flight runs complete, tested with a moved counter. Replay substitutes every irreversible adapter and a live adapter reached during replay raises rather than acts, tested. Per-run cost is recorded as the run proceeds rather than computed at the end, so a run killed halfway still shows what it spent, tested. Export round-trips into an empty database and the replayed outputs match the recorded ones for deterministic tools. Do not build yet: Nothing after this. ### Out of scope (and why) - A connector library, because each integration is an OAuth application, a review process and a maintenance commitment, and two hand-written adapters beat forty half-working ones. - A visual workflow builder, which is a substantial interface project and adds nothing to the reliability problems this build is actually about. - Multi-user access with roles, since an agent runner holding credentials needs a real permissions model rather than a bolted-on one. - Autonomous agents that choose their own goals, because every safety property here rests on the workflow being a fixed list of typed steps you wrote down. ### README must contain - That the queue provides at-least-once execution and that exactly-once effects come from per-step idempotency keys, not from the queue - That every tool has a schema on both sides and that a model proposing a call is never enough to execute one - Which tools are marked irreversible, and that those steps wait for a written approval that survives a restart - That secrets live in adapters and never enter a prompt, a stored input or an export - The monthly spend ceiling, what happens when it is reached, and how replay avoids repeating effects ===== ARCHITECTURE.md ===== # Architecture · Relevance AI ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22, one web process and one worker process | the worker is where everything interesting and everything dangerous happens | | Queue | BullMQ on Redis, with stalled jobs treated as a design input | a job whose worker died is re-run by another worker, and your build has to survive that | | Storage | PostgreSQL holding runs, steps, attempts and tool calls | if reading the database does not explain a run, the run cannot be defended | | Tools | A registry of typed tools with JSON Schema on both sides | the model proposes and the runner validates, which is the whole safety story | | Secrets | Held by the adapter, never placed in a prompt or a stored input | a credential inside a model context is a credential in a log somewhere | | Cost | Per-run accounting against a hard monthly ceiling | an agent in a loop is the most expensive bug available to you | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Runner | Runs, steps, attempts and state transitions. | Any orchestrator, provided step rows are written at start. | | Registry | Tool schemas, reversibility flags and replay mode. | Not swappable for free-form function calling, which removes the validation. | | Idempotency | Derived keys and remote reference checks. | Any scheme, provided the key exists before the adapter call. | | Approvals | Waiting states and recorded decisions. | Any interface, provided waiting is a database state. | | Cost | Incremental accounting and the ceiling. | Any accounting, provided it is written as the run proceeds. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Runs, steps, attempts, approvals. The whole audit trail. - `REDIS_URL` · required · The job queue. Losing it loses in-flight jobs, not history. - `MODEL_API_KEY` · required, secret · The proposal step. Set a provider-side limit as well. - `MODEL_ID` · required · Recorded on every run, because a different model proposes differently. - `MONTHLY_COST_CEILING` · required · New runs are refused at this figure. In-flight runs finish. - `WORKER_CONCURRENCY` · required · Start at one. Concurrency is how you discover your idempotency was wishful. - `APPROVAL_TIMEOUT_HOURS` · required · A waiting run expires here rather than sitting forever holding a claim. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Relevance AI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22, one web process and one worker process, BullMQ on Redis, with stalled jobs treated as a design input, PostgreSQL holding runs, steps, attempts and tool calls, A registry of typed tools with JSON Schema on both sides, Held by the adapter, never placed in a prompt or a stored input, Per-run accounting against a hard monthly ceiling. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. - A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter. ===== MILESTONES.md ===== # Delivery milestones · Relevance AI Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · The run log is the product Every run writes enough rows that somebody can explain what happened without reading a log file. ### Steps 1. Write step rows at start, not at finish A crash is exactly when you most want to know where it got to. Files: `src/runner/run.ts`, `migrations/002_runs.sql` 2. Pin the workflow version on the run Otherwise editing the workflow rewrites the past. Files: `src/runner/version.ts`, `tests/version-pin.test.ts` 3. Redact secret references from stored inputs Inputs get exported, and exports get shared. Files: `src/runner/redact.ts`, `tests/redact.test.ts` ### Done when - [ ] Editing a workflow leaves old runs unchanged - [ ] An interrupted run leaves a started step row - [ ] Every attempt is its own row with an outcome - [ ] No secret value appears in a stored input or output ### Watch out - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. ## M2 · The model proposes, the runner decides A tool only runs when its arguments validate against a schema the runner owns. ### Steps 1. Schema on both sides of every tool Output validation catches the API that changed under you. Files: `src/tools/registry.ts`, `src/tools/schema.ts` 2. One repair attempt, with the errors quoted Two is a loop and a loop is a bill. Files: `src/runner/propose.ts`, `tests/repair-once.test.ts` 3. Treat fetched content as data, never as instructions The web page your agent reads is written by somebody else. Files: `src/runner/prompt.ts`, `tests/injection.test.ts` ### Done when - [ ] An unknown tool name never reaches the network - [ ] Invalid arguments get one repair, then fail - [ ] A tool output that breaks its schema fails the step - [ ] Injected instructions in fetched text are ignored ### Watch out - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. ## M3 · Effects that happen once on a queue that runs twice A step that sends, charges or writes to somebody else's system does it once even when the job is re-run. ### Steps 1. Derive the key before the adapter is called A key allocated after the effect protects nothing. Files: `src/runner/idempotency.ts`, `tests/key-stability.test.ts` 2. Record the remote identifier in tool_calls It is the only evidence the effect already happened. Files: `src/tools/adapter.ts` 3. Kill the worker mid-step in a test The failure you do not simulate is the one you ship. Files: `tests/stalled-replay.test.ts` ### Done when - [ ] A killed worker does not repeat a step's effect - [ ] Keys are stable across restarts - [ ] Irreversible adapters refuse without a pre-check - [ ] An exhausted stalled job fails with its history ### Watch out - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. ## M4 · The steps that stop and ask Anything irreversible waits for a human, and the wait is a first-class state rather than a paused script. ### Steps 1. Mark tools reversible or not in the registry The classification is the point; the gate is easy once it exists. Files: `src/tools/registry.ts` 2. Make waiting a database state, not a sleep Decisions arrive hours later, from other machines. Files: `src/runner/approvals.ts`, `tests/resume-after-restart.test.ts` 3. Show the resolved arguments in the request Approving a template is not approving anything. Files: `src/app/approvals/page.tsx` ### Done when - [ ] A waiting run survives a restart - [ ] The request shows resolved arguments - [ ] Approval resumes with the same idempotency key - [ ] Rejection records a reason and stops the run ### Watch out - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. ## M5 · A ceiling, a replay and a way out (production only) The runner cannot spend without limit, any run can be replayed without repeating its effects, and everything exports. ### Steps 1. Account cost as the run proceeds A run that died halfway still cost money. Files: `src/runner/cost.ts`, `tests/cost-ceiling.test.ts` 2. Make replay a registry mode that cannot leak Replaying with live adapters is a self-inflicted incident. Files: `src/tools/registry.ts`, `tests/replay-safety.test.ts` 3. Export every table as newline-delimited JSON Including attempts, which is where the arguments are. Files: `src/export/dump.ts` ### Done when - [ ] The monthly ceiling refuses new runs and explains why - [ ] Replay cannot reach a live irreversible adapter - [ ] Cost is recorded incrementally, not at the end - [ ] Export and reimport round-trips every table ### Watch out - A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter. ===== OPERATIONS.md ===== # Operations · Relevance AI ## Backup PostgreSQL nightly. Redis is not backed up, because it holds work in progress rather than history, and that distinction should be written down so nobody restores a queue and replays last week. ## Restore Restore PostgreSQL, start the web process before the worker, and confirm that waiting approvals are still listed and resumable. Starting the worker first against a restored database can resume runs whose effects already happened. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Stalled job count, the ratio of repair attempts to proposals, approvals older than the timeout, monthly spend against the ceiling, and steps whose external reference is missing. A rising stall count means the worker is doing blocking work it should not. ## Incident checklist The incident is a side effect that happened twice or happened without approval. Both are answered from steps, attempts and tool_calls, which is the entire reason those tables record more than the final answer. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A killed worker is confirmed not to repeat a step's effect - [ ] An invalid tool call is confirmed never to reach the network - [ ] A waiting approval is confirmed to survive a restart - [ ] Replay is confirmed unable to reach a live irreversible adapter - [ ] The monthly ceiling is confirmed to refuse new runs ## Launch constraint Do not market omitted Relevance AI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Runs, steps, attempts, approvals. The whole audit trail. DATABASE_URL=postgres://runner@localhost/runner # Required. The job queue. Losing it loses in-flight jobs, not history. REDIS_URL=redis://localhost:6379 # Required · secret. The proposal step. Set a provider-side limit as well. MODEL_API_KEY=sk-...redacted # Required. Recorded on every run, because a different model proposes differently. MODEL_ID=a model identifier supporting tool calls # Required. New runs are refused at this figure. In-flight runs finish. MONTHLY_COST_CEILING=10.00 # Required. Start at one. Concurrency is how you discover your idempotency was wishful. WORKER_CONCURRENCY=1 # Required. A waiting run expires here rather than sitting forever holding a claim. APPROVAL_TIMEOUT_HOURS=48
README.md
# Relevance AI · indie build A runner for one automation whose hard problems are that a queue gives you at-least-once execution while the actions you are automating are not repeatable, that a model proposing a tool call is not the same as a tool call being valid, and that an agent with credentials and no ceiling is an unbounded liability. Estimated effort: **weekend**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22, one web process and one worker process | the worker is where everything interesting and everything dangerous happens | | Queue | BullMQ on Redis, with stalled jobs treated as a design input | a job whose worker died is re-run by another worker, and your build has to survive that | | Storage | PostgreSQL holding runs, steps, attempts and tool calls | if reading the database does not explain a run, the run cannot be defended | | Tools | A registry of typed tools with JSON Schema on both sides | the model proposes and the runner validates, which is the whole safety story | | Secrets | Held by the adapter, never placed in a prompt or a stored input | a credential inside a model context is a credential in a log somewhere | | Cost | Per-run accounting against a hard monthly ceiling | an agent in a loop is the most expensive bug available to you | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Runs the web process and the worker. - Get it: nodejs.org or your version manager. - Verify: node --version prints v22 or higher - [ ] **Redis** · free locally - Why: BullMQ needs it, and the stalled-job behaviour that shapes phase three is a property of this queue rather than an abstraction. - Get it: Your package manager, or a small managed instance. - Verify: A worker picks up a queued job and a killed worker's job is re-delivered - [ ] **PostgreSQL 16 or newer** · free locally - Why: Holds runs, steps, attempts and approvals, which need to survive every restart in this design. - Get it: Your package manager or a managed instance. - Verify: Migrations apply and a waiting run survives a restart - [ ] **An API key for a model that supports structured tool calling** · pay per call; the ceiling in this build is what keeps it small - Why: The proposal step needs structured output, and a provider without it turns phase two into string parsing, which is the thing that phase exists to prevent. - Get it: Create a key and set a provider-side spend limit as well as the ceiling in this build. - Verify: One proposal returns arguments that validate against a tool schema - [ ] **The one workflow this runner automates, written down before any code** · free - Why: Every design decision here is about a fixed list of typed steps, and a runner built without one specific workflow in mind becomes a connector library nobody finishes. - Get it: Write the trigger, the steps, their inputs and which of them are irreversible. - Verify: The written list matches the registry, including the reversible flags ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - A connector library. - A visual workflow builder. - Multi-user access with roles. - Agents that choose their own goals. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls If one of those is essential to you, that is the reason to keep paying for Relevance AI, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Relevance AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a single-workflow agent runner whose runs are replayable from the database, whose side effects happen once, and whose spending has a ceiling to replace Relevance AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22, one web process and one worker process · the worker is where everything interesting and everything dangerous happens - Queue: BullMQ on Redis, with stalled jobs treated as a design input · a job whose worker died is re-run by another worker, and your build has to survive that - Storage: PostgreSQL holding runs, steps, attempts and tool calls · if reading the database does not explain a run, the run cannot be defended - Tools: A registry of typed tools with JSON Schema on both sides · the model proposes and the runner validates, which is the whole safety story - Secrets: Held by the adapter, never placed in a prompt or a stored input · a credential inside a model context is a credential in a log somewhere - Cost: Per-run accounting against a hard monthly ceiling · an agent in a loop is the most expensive bug available to you ### Data model (create this before Phase 1) - workflows(id, name, definition_json, version, created_at) · a run pins the version it started under, so editing the workflow never rewrites history - runs(id, workflow_id, workflow_version, trigger_kind, trigger_ref, state, started_at, ended_at, cost_minor) - steps(id, run_id, ordinal, tool_name, input_json, output_json, state, idempotency_key, started_at, ended_at) - attempts(id, step_id, attempt, outcome, error_code, error_excerpt, duration_ms, at) · including the attempts that stalled and were re-run by another worker - tool_calls(id, step_id, request_id, external_ref, external_status, at) · the other side's identifier, which is the only proof the effect already happened - approvals(step_id, requested_at, decided_at, decided_by, decision, reason) · irreversible steps stop here and wait - secret_refs(name, provider, last_rotated_at) · names and metadata only, never a value, and never joined into a prompt - A run is reconstructable from these rows alone. If you need a log file to explain what an agent did, the schema is wrong. ### Phase 1 · The run log is the product Build: Define one workflow as data: an ordered list of typed steps with inputs mapped from earlier outputs. On trigger, create a run pinned to the current workflow version, then one step row per stage as it begins. Record each attempt separately with its outcome, duration and a truncated error. Keep inputs and outputs as JSON on the step, redacting anything named as a secret reference. Trap: Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. Done when: Editing a workflow does not change how an existing run reads back, tested by editing the definition and replaying the query. A run interrupted halfway leaves step rows in a started state rather than no rows at all, tested by killing the worker. Every attempt appears as its own row with an outcome and a duration, including failures. No value referenced through secret_refs appears in any stored input or output, tested with a fixture containing one. Do not build yet: No tool validation, no idempotency, no approvals, no ceiling yet. ### Phase 2 · The model proposes, the runner decides Build: Give every tool a JSON Schema for its input and its output. Ask the model for a structured tool call, validate it against the schema, and on failure return the validation errors for exactly one repair attempt before failing the step. Never execute a tool whose arguments did not validate, and never accept a tool name that is not in the registry. Trap: Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. Done when: A tool call with an unknown tool name is refused without an HTTP request, tested. A call failing schema validation triggers exactly one repair attempt and then fails the step, tested with a stub that never returns valid arguments. Output validation runs too, and a tool returning a shape that does not match its schema fails the step rather than passing a surprise to the next one, tested. Text arriving from a fetched document is never treated as an instruction, checked by a test containing an injected instruction that must be ignored. Do not build yet: No idempotency, no approvals, no replay, no ceiling yet. ### Phase 3 · Effects that happen once on a queue that runs twice Build: Derive a deterministic idempotency key per step from the run identifier, the step ordinal and a hash of the resolved input. Store it before the adapter is called. Each adapter either passes the key to the remote API or records the remote identifier in tool_calls and checks for it before acting. On restart, a step with a recorded external reference is complete, not pending. Trap: BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. Done when: A step interrupted between the outbound call and the database write does not repeat its effect when re-run, tested by killing the worker at that point against a stub that counts calls. The idempotency key is a pure function of run, ordinal and input hash, tested for stability across process restarts. Adapters for irreversible actions refuse to run without either a remote idempotency mechanism or a pre-check, asserted in the suite. A stalled job that returns after maxStalledCount is failed with its history intact rather than silently dropped. Do not build yet: No approvals, no replay interface, no export, no ceiling yet. ### Phase 4 · The steps that stop and ask Build: Mark tools in the registry as reversible or not. An irreversible step creates an approval row and the run enters a waiting state that survives a restart. Approving resumes from that step with the same idempotency key. Rejecting ends the run with the reason recorded. Include what the step will actually do, rendered from the validated arguments, in the approval request. Trap: Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. Done when: A run waiting for approval survives a full restart of the web and worker processes and is still resumable, tested. The approval request shows the resolved arguments rather than the template, checked in the interface test. Approving resumes with the same idempotency key so an approved step cannot double-act, tested. Rejecting records the reason and ends the run, and no later step executes, tested. Do not build yet: No spend ceiling, no replay tooling, no export yet. ### Phase 5 · A ceiling, a replay and a way out Build: Account model tokens and per-call costs onto the run as it goes. When the monthly total reaches the ceiling, refuse to start new runs and say so, leaving in-flight runs to finish. Add a replay that re-executes a run against recorded inputs in a mode where every irreversible adapter is a stub. Export runs, steps, attempts and tool calls as newline-delimited JSON. Trap: A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter. Done when: Reaching the monthly ceiling refuses new runs with a named reason and lets in-flight runs complete, tested with a moved counter. Replay substitutes every irreversible adapter and a live adapter reached during replay raises rather than acts, tested. Per-run cost is recorded as the run proceeds rather than computed at the end, so a run killed halfway still shows what it spent, tested. Export round-trips into an empty database and the replayed outputs match the recorded ones for deterministic tools. Do not build yet: Nothing after this. ### Out of scope (and why) - A connector library, because each integration is an OAuth application, a review process and a maintenance commitment, and two hand-written adapters beat forty half-working ones. - A visual workflow builder, which is a substantial interface project and adds nothing to the reliability problems this build is actually about. - Multi-user access with roles, since an agent runner holding credentials needs a real permissions model rather than a bolted-on one. - Autonomous agents that choose their own goals, because every safety property here rests on the workflow being a fixed list of typed steps you wrote down. ### README must contain - That the queue provides at-least-once execution and that exactly-once effects come from per-step idempotency keys, not from the queue - That every tool has a schema on both sides and that a model proposing a call is never enough to execute one - Which tools are marked irreversible, and that those steps wait for a written approval that survives a restart - That secrets live in adapters and never enter a prompt, a stored input or an export - The monthly spend ceiling, what happens when it is reached, and how replay avoids repeating effects
AGENTS.md
# Agent instructions · Relevance AI indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Node 22, one web process and one worker process, BullMQ on Redis, with stalled jobs treated as a design input, PostgreSQL holding runs, steps, attempts and tool calls, A registry of typed tools with JSON Schema on both sides, Held by the adapter, never placed in a prompt or a stored input, Per-run accounting against a hard monthly ceiling. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine.
BUILD_PLAN.md
# Build plan · Relevance AI A runner for one automation whose hard problems are that a queue gives you at-least-once execution while the actions you are automating are not repeatable, that a model proposing a tool call is not the same as a tool call being valid, and that an agent with credentials and no ceiling is an unbounded liability. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · The run log is the product Every run writes enough rows that somebody can explain what happened without reading a log file. ### Steps 1. Write step rows at start, not at finish A crash is exactly when you most want to know where it got to. Files: `src/runner/run.ts`, `migrations/002_runs.sql` 2. Pin the workflow version on the run Otherwise editing the workflow rewrites the past. Files: `src/runner/version.ts`, `tests/version-pin.test.ts` 3. Redact secret references from stored inputs Inputs get exported, and exports get shared. Files: `src/runner/redact.ts`, `tests/redact.test.ts` ### Done when - [ ] Editing a workflow leaves old runs unchanged - [ ] An interrupted run leaves a started step row - [ ] Every attempt is its own row with an outcome - [ ] No secret value appears in a stored input or output ### Watch out - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. ## Phase 2 · The model proposes, the runner decides A tool only runs when its arguments validate against a schema the runner owns. ### Steps 1. Schema on both sides of every tool Output validation catches the API that changed under you. Files: `src/tools/registry.ts`, `src/tools/schema.ts` 2. One repair attempt, with the errors quoted Two is a loop and a loop is a bill. Files: `src/runner/propose.ts`, `tests/repair-once.test.ts` 3. Treat fetched content as data, never as instructions The web page your agent reads is written by somebody else. Files: `src/runner/prompt.ts`, `tests/injection.test.ts` ### Done when - [ ] An unknown tool name never reaches the network - [ ] Invalid arguments get one repair, then fail - [ ] A tool output that breaks its schema fails the step - [ ] Injected instructions in fetched text are ignored ### Watch out - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. ## Phase 3 · Effects that happen once on a queue that runs twice A step that sends, charges or writes to somebody else's system does it once even when the job is re-run. ### Steps 1. Derive the key before the adapter is called A key allocated after the effect protects nothing. Files: `src/runner/idempotency.ts`, `tests/key-stability.test.ts` 2. Record the remote identifier in tool_calls It is the only evidence the effect already happened. Files: `src/tools/adapter.ts` 3. Kill the worker mid-step in a test The failure you do not simulate is the one you ship. Files: `tests/stalled-replay.test.ts` ### Done when - [ ] A killed worker does not repeat a step's effect - [ ] Keys are stable across restarts - [ ] Irreversible adapters refuse without a pre-check - [ ] An exhausted stalled job fails with its history ### Watch out - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. ## Phase 4 · The steps that stop and ask Anything irreversible waits for a human, and the wait is a first-class state rather than a paused script. ### Steps 1. Mark tools reversible or not in the registry The classification is the point; the gate is easy once it exists. Files: `src/tools/registry.ts` 2. Make waiting a database state, not a sleep Decisions arrive hours later, from other machines. Files: `src/runner/approvals.ts`, `tests/resume-after-restart.test.ts` 3. Show the resolved arguments in the request Approving a template is not approving anything. Files: `src/app/approvals/page.tsx` ### Done when - [ ] A waiting run survives a restart - [ ] The request shows resolved arguments - [ ] Approval resumes with the same idempotency key - [ ] Rejection records a reason and stops the run ### Watch out - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. ## Not in this build - A connector library. - A visual workflow builder. - Multi-user access with roles. - Agents that choose their own goals. ## After v1, if you want it - A second workflow, which is the real test of whether the registry and the run schema were general enough - A diff view between two runs of the same workflow version, which finds prompt drift faster than reading outputs - Scheduled triggers with a jitter window, once you trust the idempotency enough to let it run unattended
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Runs, steps, attempts, approvals. The whole audit trail. DATABASE_URL=postgres://runner@localhost/runner # Required. The job queue. Losing it loses in-flight jobs, not history. REDIS_URL=redis://localhost:6379 # Required · secret. The proposal step. Set a provider-side limit as well. MODEL_API_KEY=sk-...redacted # Required. Recorded on every run, because a different model proposes differently. MODEL_ID=a model identifier supporting tool calls # Required. New runs are refused at this figure. In-flight runs finish. MONTHLY_COST_CEILING=10.00 # Required. Start at one. Concurrency is how you discover your idempotency was wishful. WORKER_CONCURRENCY=1 # Required. A waiting run expires here rather than sitting forever holding a claim. APPROVAL_TIMEOUT_HOURS=48
PRODUCT.md
# Relevance AI · product brief ## Problem The visible AI workforce platform loop is buildable, but a credible replacement needs more than the first screen. Relevance AI earns its keep through connectors, auth, reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A runner for one automation that can be explained from its own tables, that validates every tool call before executing it, that sends the email once even when the worker died, that stops and asks before anything irreversible, and that cannot spend past a number you set. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - hosted PostgreSQL and Redis - API credentials for the two chosen services - Node.js 22 ## Explicit non-goals for v1 - A connector library. - A visual workflow builder. - Multi-user access with roles. - Agents that choose their own goals. - hundreds of maintained connectors - OAuth app verification - durable execution at scale - schema drift handling and enterprise controls ## Success criteria - A killed worker is confirmed not to repeat a step's effect - An invalid tool call is confirmed never to reach the network - A waiting approval is confirmed to survive a restart - Replay is confirmed unable to reach a live irreversible adapter - The monthly ceiling is confirmed to refuse new runs
BRIEF.md
# Build brief · Relevance AI The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me a single-workflow agent runner whose runs are replayable from the database, whose side effects happen once, and whose spending has a ceiling to replace Relevance AI. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Runtime: Node 22, one web process and one worker process · the worker is where everything interesting and everything dangerous happens - Queue: BullMQ on Redis, with stalled jobs treated as a design input · a job whose worker died is re-run by another worker, and your build has to survive that - Storage: PostgreSQL holding runs, steps, attempts and tool calls · if reading the database does not explain a run, the run cannot be defended - Tools: A registry of typed tools with JSON Schema on both sides · the model proposes and the runner validates, which is the whole safety story - Secrets: Held by the adapter, never placed in a prompt or a stored input · a credential inside a model context is a credential in a log somewhere - Cost: Per-run accounting against a hard monthly ceiling · an agent in a loop is the most expensive bug available to you ### Data model (create this before Phase 1) - workflows(id, name, definition_json, version, created_at) · a run pins the version it started under, so editing the workflow never rewrites history - runs(id, workflow_id, workflow_version, trigger_kind, trigger_ref, state, started_at, ended_at, cost_minor) - steps(id, run_id, ordinal, tool_name, input_json, output_json, state, idempotency_key, started_at, ended_at) - attempts(id, step_id, attempt, outcome, error_code, error_excerpt, duration_ms, at) · including the attempts that stalled and were re-run by another worker - tool_calls(id, step_id, request_id, external_ref, external_status, at) · the other side's identifier, which is the only proof the effect already happened - approvals(step_id, requested_at, decided_at, decided_by, decision, reason) · irreversible steps stop here and wait - secret_refs(name, provider, last_rotated_at) · names and metadata only, never a value, and never joined into a prompt - A run is reconstructable from these rows alone. If you need a log file to explain what an agent did, the schema is wrong. ### Phase 1 · The run log is the product Build: Define one workflow as data: an ordered list of typed steps with inputs mapped from earlier outputs. On trigger, create a run pinned to the current workflow version, then one step row per stage as it begins. Record each attempt separately with its outcome, duration and a truncated error. Keep inputs and outputs as JSON on the step, redacting anything named as a secret reference. Trap: Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. Done when: Editing a workflow does not change how an existing run reads back, tested by editing the definition and replaying the query. A run interrupted halfway leaves step rows in a started state rather than no rows at all, tested by killing the worker. Every attempt appears as its own row with an outcome and a duration, including failures. No value referenced through secret_refs appears in any stored input or output, tested with a fixture containing one. Do not build yet: No tool validation, no idempotency, no approvals, no ceiling yet. ### Phase 2 · The model proposes, the runner decides Build: Give every tool a JSON Schema for its input and its output. Ask the model for a structured tool call, validate it against the schema, and on failure return the validation errors for exactly one repair attempt before failing the step. Never execute a tool whose arguments did not validate, and never accept a tool name that is not in the registry. Trap: Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. Done when: A tool call with an unknown tool name is refused without an HTTP request, tested. A call failing schema validation triggers exactly one repair attempt and then fails the step, tested with a stub that never returns valid arguments. Output validation runs too, and a tool returning a shape that does not match its schema fails the step rather than passing a surprise to the next one, tested. Text arriving from a fetched document is never treated as an instruction, checked by a test containing an injected instruction that must be ignored. Do not build yet: No idempotency, no approvals, no replay, no ceiling yet. ### Phase 3 · Effects that happen once on a queue that runs twice Build: Derive a deterministic idempotency key per step from the run identifier, the step ordinal and a hash of the resolved input. Store it before the adapter is called. Each adapter either passes the key to the remote API or records the remote identifier in tool_calls and checks for it before acting. On restart, a step with a recorded external reference is complete, not pending. Trap: BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. Done when: A step interrupted between the outbound call and the database write does not repeat its effect when re-run, tested by killing the worker at that point against a stub that counts calls. The idempotency key is a pure function of run, ordinal and input hash, tested for stability across process restarts. Adapters for irreversible actions refuse to run without either a remote idempotency mechanism or a pre-check, asserted in the suite. A stalled job that returns after maxStalledCount is failed with its history intact rather than silently dropped. Do not build yet: No approvals, no replay interface, no export, no ceiling yet. ### Phase 4 · The steps that stop and ask Build: Mark tools in the registry as reversible or not. An irreversible step creates an approval row and the run enters a waiting state that survives a restart. Approving resumes from that step with the same idempotency key. Rejecting ends the run with the reason recorded. Include what the step will actually do, rendered from the validated arguments, in the approval request. Trap: Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. Done when: A run waiting for approval survives a full restart of the web and worker processes and is still resumable, tested. The approval request shows the resolved arguments rather than the template, checked in the interface test. Approving resumes with the same idempotency key so an approved step cannot double-act, tested. Rejecting records the reason and ends the run, and no later step executes, tested. Do not build yet: No spend ceiling, no replay tooling, no export yet. ### Phase 5 · A ceiling, a replay and a way out Build: Account model tokens and per-call costs onto the run as it goes. When the monthly total reaches the ceiling, refuse to start new runs and say so, leaving in-flight runs to finish. Add a replay that re-executes a run against recorded inputs in a mode where every irreversible adapter is a stub. Export runs, steps, attempts and tool calls as newline-delimited JSON. Trap: A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter. Done when: Reaching the monthly ceiling refuses new runs with a named reason and lets in-flight runs complete, tested with a moved counter. Replay substitutes every irreversible adapter and a live adapter reached during replay raises rather than acts, tested. Per-run cost is recorded as the run proceeds rather than computed at the end, so a run killed halfway still shows what it spent, tested. Export round-trips into an empty database and the replayed outputs match the recorded ones for deterministic tools. Do not build yet: Nothing after this. ### Out of scope (and why) - A connector library, because each integration is an OAuth application, a review process and a maintenance commitment, and two hand-written adapters beat forty half-working ones. - A visual workflow builder, which is a substantial interface project and adds nothing to the reliability problems this build is actually about. - Multi-user access with roles, since an agent runner holding credentials needs a real permissions model rather than a bolted-on one. - Autonomous agents that choose their own goals, because every safety property here rests on the workflow being a fixed list of typed steps you wrote down. ### README must contain - That the queue provides at-least-once execution and that exactly-once effects come from per-step idempotency keys, not from the queue - That every tool has a schema on both sides and that a model proposing a call is never enough to execute one - Which tools are marked irreversible, and that those steps wait for a written approval that survives a restart - That secrets live in adapters and never enter a prompt, a stored input or an export - The monthly spend ceiling, what happens when it is reached, and how replay avoids repeating effects
ARCHITECTURE.md
# Architecture · Relevance AI ## Stack | Part | Choice | Why | | --- | --- | --- | | Runtime | Node 22, one web process and one worker process | the worker is where everything interesting and everything dangerous happens | | Queue | BullMQ on Redis, with stalled jobs treated as a design input | a job whose worker died is re-run by another worker, and your build has to survive that | | Storage | PostgreSQL holding runs, steps, attempts and tool calls | if reading the database does not explain a run, the run cannot be defended | | Tools | A registry of typed tools with JSON Schema on both sides | the model proposes and the runner validates, which is the whole safety story | | Secrets | Held by the adapter, never placed in a prompt or a stored input | a credential inside a model context is a credential in a log somewhere | | Cost | Per-run accounting against a hard monthly ceiling | an agent in a loop is the most expensive bug available to you | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Runner | Runs, steps, attempts and state transitions. | Any orchestrator, provided step rows are written at start. | | Registry | Tool schemas, reversibility flags and replay mode. | Not swappable for free-form function calling, which removes the validation. | | Idempotency | Derived keys and remote reference checks. | Any scheme, provided the key exists before the adapter call. | | Approvals | Waiting states and recorded decisions. | Any interface, provided waiting is a database state. | | Cost | Incremental accounting and the ceiling. | Any accounting, provided it is written as the run proceeds. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Runs, steps, attempts, approvals. The whole audit trail. - `REDIS_URL` · required · The job queue. Losing it loses in-flight jobs, not history. - `MODEL_API_KEY` · required, secret · The proposal step. Set a provider-side limit as well. - `MODEL_ID` · required · Recorded on every run, because a different model proposes differently. - `MONTHLY_COST_CEILING` · required · New runs are refused at this figure. In-flight runs finish. - `WORKER_CONCURRENCY` · required · Start at one. Concurrency is how you discover your idempotency was wishful. - `APPROVAL_TIMEOUT_HOURS` · required · A waiting run expires here rather than sitting forever holding a claim. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Relevance AI product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Node 22, one web process and one worker process, BullMQ on Redis, with stalled jobs treated as a design input, PostgreSQL holding runs, steps, attempts and tool calls, A registry of typed tools with JSON Schema on both sides, Held by the adapter, never placed in a prompt or a stored input, Per-run accounting against a hard monthly ceiling. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. - A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter.
MILESTONES.md
# Delivery milestones · Relevance AI Estimated effort: **weekend** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · The run log is the product Every run writes enough rows that somebody can explain what happened without reading a log file. ### Steps 1. Write step rows at start, not at finish A crash is exactly when you most want to know where it got to. Files: `src/runner/run.ts`, `migrations/002_runs.sql` 2. Pin the workflow version on the run Otherwise editing the workflow rewrites the past. Files: `src/runner/version.ts`, `tests/version-pin.test.ts` 3. Redact secret references from stored inputs Inputs get exported, and exports get shared. Files: `src/runner/redact.ts`, `tests/redact.test.ts` ### Done when - [ ] Editing a workflow leaves old runs unchanged - [ ] An interrupted run leaves a started step row - [ ] Every attempt is its own row with an outcome - [ ] No secret value appears in a stored input or output ### Watch out - Storing only the final output of a run means the first time an automation does something wrong you have a result, no inputs, no intermediate values and no idea which version of the workflow produced it, so the only available response is to run it again and hope. Write the step rows as they start rather than when they finish, and pin the workflow version on the run. ## M2 · The model proposes, the runner decides A tool only runs when its arguments validate against a schema the runner owns. ### Steps 1. Schema on both sides of every tool Output validation catches the API that changed under you. Files: `src/tools/registry.ts`, `src/tools/schema.ts` 2. One repair attempt, with the errors quoted Two is a loop and a loop is a bill. Files: `src/runner/propose.ts`, `tests/repair-once.test.ts` 3. Treat fetched content as data, never as instructions The web page your agent reads is written by somebody else. Files: `src/runner/prompt.ts`, `tests/injection.test.ts` ### Done when - [ ] An unknown tool name never reaches the network - [ ] Invalid arguments get one repair, then fail - [ ] A tool output that breaks its schema fails the step - [ ] Injected instructions in fetched text are ignored ### Watch out - Pulling a tool call out of model output with a regular expression and passing whatever appeared into the adapter means a hallucinated field name reaches your HTTP client and a hallucinated identifier reaches somebody else's database, and the failure looks like a bug in their API rather than in your parsing. Validate against the schema first, repair once with the errors quoted back, then stop. ## M3 · Effects that happen once on a queue that runs twice A step that sends, charges or writes to somebody else's system does it once even when the job is re-run. ### Steps 1. Derive the key before the adapter is called A key allocated after the effect protects nothing. Files: `src/runner/idempotency.ts`, `tests/key-stability.test.ts` 2. Record the remote identifier in tool_calls It is the only evidence the effect already happened. Files: `src/tools/adapter.ts` 3. Kill the worker mid-step in a test The failure you do not simulate is the one you ship. Files: `tests/stalled-replay.test.ts` ### Done when - [ ] A killed worker does not repeat a step's effect - [ ] Keys are stable across restarts - [ ] Irreversible adapters refuse without a pre-check - [ ] An exhausted stalled job fails with its history ### Watch out - BullMQ's own documentation is explicit that a worker which fails to renew its lock has its job marked stalled, moved back to waiting and processed again by another worker, so a step that sends an email will send it twice the first time your worker gets busy enough to miss a heartbeat. Derive the key before the call, let the adapter check the remote side for it, and never treat the queue's delivery as the exactly-once guarantee it is not. ## M4 · The steps that stop and ask Anything irreversible waits for a human, and the wait is a first-class state rather than a paused script. ### Steps 1. Mark tools reversible or not in the registry The classification is the point; the gate is easy once it exists. Files: `src/tools/registry.ts` 2. Make waiting a database state, not a sleep Decisions arrive hours later, from other machines. Files: `src/runner/approvals.ts`, `tests/resume-after-restart.test.ts` 3. Show the resolved arguments in the request Approving a template is not approving anything. Files: `src/app/approvals/page.tsx` ### Done when - [ ] A waiting run survives a restart - [ ] The request shows resolved arguments - [ ] Approval resumes with the same idempotency key - [ ] Rejection records a reason and stops the run ### Watch out - Implementing approval as a prompt in the terminal or a sleep inside the job means a worker restart loses the pending decision and the run either dies or, worse, resumes and acts without one. Make waiting a state in the database with the resolved arguments stored beside it, so approval is a row somebody writes hours later from a different machine. ## M5 · A ceiling, a replay and a way out (production only) The runner cannot spend without limit, any run can be replayed without repeating its effects, and everything exports. ### Steps 1. Account cost as the run proceeds A run that died halfway still cost money. Files: `src/runner/cost.ts`, `tests/cost-ceiling.test.ts` 2. Make replay a registry mode that cannot leak Replaying with live adapters is a self-inflicted incident. Files: `src/tools/registry.ts`, `tests/replay-safety.test.ts` 3. Export every table as newline-delimited JSON Including attempts, which is where the arguments are. Files: `src/export/dump.ts` ### Done when - [ ] The monthly ceiling refuses new runs and explains why - [ ] Replay cannot reach a live irreversible adapter - [ ] Cost is recorded incrementally, not at the end - [ ] Export and reimport round-trips every table ### Watch out - A replay that re-runs the real adapters will send the emails again, and somebody will discover this by replaying last Tuesday to debug it. Make replay a mode the adapter registry itself enforces, where irreversible tools are substituted for recorders, and fail loudly if a replay ever reaches a live adapter.
OPERATIONS.md
# Operations · Relevance AI ## Backup PostgreSQL nightly. Redis is not backed up, because it holds work in progress rather than history, and that distinction should be written down so nobody restores a queue and replays last week. ## Restore Restore PostgreSQL, start the web process before the worker, and confirm that waiting approvals are still listed and resumable. Starting the worker first against a restored database can resume runs whose effects already happened. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Stalled job count, the ratio of repair attempts to proposals, approvals older than the timeout, monthly spend against the ceiling, and steps whose external reference is missing. A rising stall count means the worker is doing blocking work it should not. ## Incident checklist The incident is a side effect that happened twice or happened without approval. Both are answered from steps, attempts and tool_calls, which is the entire reason those tables record more than the final answer. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] A killed worker is confirmed not to repeat a step's effect - [ ] An invalid tool call is confirmed never to reach the network - [ ] A waiting approval is confirmed to survive a restart - [ ] Replay is confirmed unable to reach a live irreversible adapter - [ ] The monthly ceiling is confirmed to refuse new runs ## Launch constraint Do not market omitted Relevance AI capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Runs, steps, attempts, approvals. The whole audit trail. DATABASE_URL=postgres://runner@localhost/runner # Required. The job queue. Losing it loses in-flight jobs, not history. REDIS_URL=redis://localhost:6379 # Required · secret. The proposal step. Set a provider-side limit as well. MODEL_API_KEY=sk-...redacted # Required. Recorded on every run, because a different model proposes differently. MODEL_ID=a model identifier supporting tool calls # Required. New runs are refused at this figure. In-flight runs finish. MONTHLY_COST_CEILING=10.00 # Required. Start at one. Concurrency is how you discover your idempotency was wishful. WORKER_CONCURRENCY=1 # Required. A waiting run expires here rather than sitting forever holding a claim. APPROVAL_TIMEOUT_HOURS=48
$ choose a build depth, inspect the files, then open the complete pack in your agent
Relevance AI: One automation is easy. Subscribers pay for maintained integrations, credentials, retries, observability, and someone else owning breakage.
xhundreds of maintained connectors
xOAuth app verification
xdurable execution at scale
xschema drift handling and enterprise controls
share on X ↗"I just replaced Relevance AI with one prompt"
Don't feel like building it? These folks already made it free.
LangflowA desktop-or-server visual agent builder with tools, knowledge, and multi-agent flows.open source↗
Dify Community EditionThe broadest finished free agent platform here; the Docker stack is the tax.free↗no votes, no pay-to-list · just what's real
Relevance AI pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| free | $0/workspace | $0/workspace | 200 Actions/month; 1,000 vendor credits one time (stated value $2); 1 workforce, 1 user, 1 project and 30-day history. |
| pro | $29/workspace | $19/workspace | 2,500 Actions/month or 30,000/year; $20 vendor credits/month or $240/year; 2 builder users. |
| team | $349/workspace | $234/workspace | 7,000 Actions/month or 84,000/year; $70 vendor credits/month or $840/year; 5 builder users, 45 end users and 5 projects. |
| enterprise | custom | — | Custom Actions, vendor credits, users, projects, security and support. |
free tier200 Actions/month, 1,000 one-time vendor credits, 1 workforce, 1 user, 1 project and 30-day history
billingmonthly + annual; annual plans carry the stated yearly Action and vendor-credit pools
hidden costsExtra Actions cost $80 per 1,000; vendor credits cost $20 per 10,000. The Free plan cannot top up. Included Actions reset, while purchased Actions and vendor-credit balances can roll while the subscription remains active.
verified 2026-08-11 · source ↗
Vibecode Relevance AI
Kinda. The core of Relevance AI is buildable in a weekend with the prompt on this page, but there are real gaps: hundreds of maintained connectors, OAuth app verification. Read the honest list above before committing.
How much does Relevance AI cost?
Relevance AI's pricing is usage-based or varies by plan · The official pricing URL is recorded, but a stable current amount was not safely recoverable in this pass. Do not publish a number until rechecked..
What do I lose by replacing Relevance AI?
Honestly: hundreds of maintained connectors; OAuth app verification; durable execution at scale; schema drift handling and enterprise controls. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Relevance AI?
Yes: Langflow (A desktop-or-server visual agent builder with tools, knowledge, and multi-agent flows.) Dify Community Edition (The broadest finished free agent platform here; the Docker stack is the tax.) The prompt is for when you want it exactly your way.