Build ZeroLeaks

KINDAreplaces $79/mosaves $948/yrback to the verdict

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

A local extraction-probe harness whose hard problems are enforcing authorisation at the transport rather than in documentation, scoring a finding by what actually leaked rather than by whether a prompt came back, and judging transcripts without posting your own secrets to somebody else's model.

estimated effort weekendthe files for this build are in the project pack

RuntimeA Python command line tool writing to a local run directory, no service and no dashboardTargetingOne target file naming exactly one host, enforced inside the HTTP transportProbesA versioned directory of declarative probe files with a pack version and a dateDetectionDeterministic matchers first, a model judge second, deterministic always winningReportingA self-contained HTML report plus JSON, with full transcripts and a diff between runs

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the command line tool.
Get it python.org or your package manager. open ↗
Verify python3 --version prints 3.12 or higher
have readyfree, and non-negotiable
Why Authorisation is what separates this from an offence, and it is a required record in the tool rather than an assumption, so you need it before anything runs.
Get it An email or signed note from whoever owns the endpoint, naming the host, the scope and the dates.
Verify The authorisation record exists, names the host and has not expired
accountwhatever your own endpoint costs to call
Why The target file describes one endpoint, its request shape and where the reply text sits in the response, and getting that wrong means every probe scores as a refusal.
Get it Point it at a staging deployment of your own application if you have one.
Verify A single probe returns extracted reply text rather than a raw response body
have readyfree
Why The deterministic matchers are the reliable half of this build and they are worthless without the actual strings to match against.
Get it Copy the current prompt and tool schema, and write patterns for every key format you use.
Verify A crafted response containing each signal produces a finding
API keypennies per run
Why The judge catches paraphrased leakage the matchers miss, but it is optional by design because a run without it must still produce every deterministic finding.
Get it Any provider whose model returns structured output reliably.
Verify A redacted transcript returns strict JSON with a verdict

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- authorisations(id, target_host, granted_by, granted_at, expires_at, evidence_path) · a run with no current row here does not start
- targets(id, name, url, method, header_names_json, body_template, reply_path) · header values come from the environment and never from this file
- signals(id, kind, value_ref, description) · kind is prompt_text, tool_name or secret_pattern, and the values live outside the repository
- probes(id, pack_version, family, severity_hint, turns_json, added_at, source_note)
- runs(id, target_id, pack_version, started_at, ended_at, authorisation_id, judge_model_id, probe_count, request_count)
- attempts(id, run_id, probe_id, turn_ordinal, request_json, response_json, latency_ms, http_status) · verbatim, because this is the evidence
- findings(id, run_id, probe_id, detector, matched_signal_id, severity, judge_json, first_seen_run_id) · severity is computed from which signal matched, never from which probe fired

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`TARGET_FILE`	required	`./target.yaml`	The one endpoint. Its host is the only host the transport will speak to.
`AUTHORISATION_FILE`	required	`./authorisation.yaml`	Who granted permission, when, and until when. Printed into every report.
`SIGNALS_FILE`	required	`./signals.yaml`	Your prompt text, tool names and secret patterns. Never committed.
`TARGET_AUTH_HEADER`secret	required	`Bearer ...redacted`	The credential for your own endpoint. Referenced by name from the target file.
`JUDGE_API_KEY`secret	optional	`sk-...redacted`	Optional. Without it the run reports every deterministic finding and no judge verdicts.
`MAX_REQUESTS_PER_RUN`	required	`600`	A hard cap. The run stops here whatever the pack says.
`CONCURRENCY`	required	`2`	Low by default. This is pointed at something somebody depends on.
`RUNS_DIR`	required	`./runs`	Transcripts and reports. These contain whatever leaked, so treat them as secret.

The build, in order

1
A target you are allowed to test
The tool cannot send a request to any host other than the one you recorded permission for.
1. Configuration-time checks have three documented ways around them.
  Files leakprobe/transport.pytests/test_host_guard.py
2. A redirect is the easiest way to point this at a stranger.
  Files leakprobe/transport.py
3. It goes in the report too, because that is where it matters.
  Files leakprobe/auth.pytests/test_auth_required.py
done when · tick each as it passes
watch out
- Checking the host once when the configuration loads leaves three ways out: a redirect, a proxy setting and a templated URL, any of which sends your adversarial traffic at somebody who never agreed to receive it and turns a test into an incident; put the check in the transport so it runs on every single request, turn off automatic redirect following, and make a cross-host attempt end the run rather than skip the request, so it is impossible to miss.
next phase
2
A probe pack that is data, not code
Probes can be added, reviewed and dated without touching the program.
1. A bad probe stops the run; it does not get skipped.
  Files leakprobe/probes/schema.pytests/test_probe_schema.py
2. A report without a pack date is a report with no shelf life.
  Files leakprobe/pack.py
3. No eval, no import, no subprocess, and a test that says so.
  Files tests/test_loader_inert.py
done when · tick each as it passes
watch out
- Encoding probes as Python functions means the pack cannot be updated without a release, cannot be reviewed by anyone who does not read your code, and eventually contains a probe that does something other than send a message, at which point your red-team harness has an arbitrary code execution path in it aimed at your own infrastructure; keep probes declarative, validate them at load, and make the loader structurally incapable of running anything.
next phase
3
Detection in the right order
A deterministic match is the finding, and the judge never sees your secrets.
1. The matchers are cheap and they are the reliable half.
  Files leakprobe/detect.pytests/test_detectors.py
2. Then record which markers stood in for what.
  Files leakprobe/judge.pytests/test_judge_redaction.py
3. A prompt with nothing in it is not a critical finding.
  Files leakprobe/severity.py
done when · tick each as it passes
watch out
- Sending the full transcript to a third-party judge model means the tool built to find out whether your system prompt and your keys leak has just posted your system prompt and your keys into somebody else's inference logs, which is the exact outcome you were testing for; match deterministically on your own machine, replace every matched span with a marker before the judge sees anything, record which markers were substituted, and make the judge optional so a run with no judge key still produces every deterministic finding.
next phase
4
Multi-turn, and not knocking the target over
A run explores multi-turn attacks while staying gentle enough to point at something real.
1. This is pointed at something somebody depends on.
  Files leakprobe/runner.pytests/test_cap.py
2. Ignoring it is how a rate limit becomes an outage.
  Files leakprobe/backoff.py
3. A thousand failed requests is not more data.
  Files leakprobe/runner.py
done when · tick each as it passes
watch out
- A harness with cheerful default concurrency pointed at your own production endpoint is an unplanned load test, and one that retries a rate-limit response immediately converts a rate limit into an outage you caused yourself on the system you were trying to protect; default the concurrency low, honour the interval the server asks for, cap total requests per run, and abort with a summary when the error rate crosses a threshold rather than grinding on and blaming the target.
5
A report that does not overclaimproduct builder
The output is usable as a regression gate and cannot be mistaken for an assurance.
1. It is the context that decides what the result means.
  Files leakprobe/report.pytemplates/report.html.j2
2. Someone will paste this into a questionnaire.
  Files tests/test_no_assurance.py
3. That is what turns this into a regression gate.
  Files leakprobe/diff.pytests/test_diff.py
done when · tick each as it passes
watch out
- A summary line reading 'no leaks found' will be pasted into a security questionnaire within a month, and it is not true: what happened is that this pack, on this date, against this endpoint, found nothing, while the jailbreak families it knows about are a snapshot of what its author understood when they wrote it; put the pack date, probe count and covered families in the summary, phrase the result as what was tested rather than what is safe, and state in the report itself that a self-run harness is not a third-party assessment.

what this build does not replace

Testing anything not named in the target file.
Denial of service, scanning or authentication bypass.
A hosted dashboard or shared findings database.
A numeric security score or badge.
A curated, maintained probe corpus that tracks new jailbreak families instead of whatever the agent remembered on build day
Multi-turn and multi-model attack strategies, including crescendo and encoding tricks, done properly rather than as single-shot prompts
A third-party report with a date on it that you can hand to a customer or an auditor
Severity triage and remediation guidance written by someone who has seen a lot of these
Regression runs on every model or prompt change without you remembering to trigger them

after v1, if you want it

A scheduled run against staging on every prompt change, which is the only use of this that catches anything before a customer does
A second signals file for a different deployment, which tests whether severity really was computed from signals
A written note beside each probe family saying where it came from and when, so the pack ages visibly rather than silently

Need the files? The project pack on the verdict page hands your agent the whole brief · more security.