# Detector performance and release budget

Current September 11, 2026. Performance matters alongside detection and benign impact. More rules, additional full-text passes and costly regex near-misses can increase CPU cost. Input-size bounds limit exposure but do not make every scan equally fast or rule growth free.

## Reproducible gate

Run `python tools/performance_guard.py`. `python deploy/preflight.py` runs it after the test suite and refuses a successful preflight if it fails. The benchmark imports only the offline detector. It makes no live scans, uses no account/key, and stores only synthetic-workload identifiers and timing aggregates.

The pinned baseline is commit 16c3ae8 (v0.4.4). Both baseline and candidate run on the same host, in isolated per-workload workers, with a warmup and nine alternating-order measurements each. Cases include 1/8/32/64 KiB prose, repeated transfer near-misses, separated letters, Unicode, encoded content, eight 4 KiB messages and nested tool metadata. Payload sizes describe workload content; this is not a wire-body-limit test. It does not exhaust the space of regex worst cases.

The report records engine hashes, workload hash, Python/platform, sample count, local median/p95/max and failed budgets. With nine runs, nearest-rank p95 is the maximum; it is a small-sample alarm, not a statistically stable production percentile. A crashed/timed-out worker fails the gate instead of silently dropping the case.

Policy in evals/performance-policy.json:

- Fail when median exceeds baseline by both 20% and 1 ms.
- Fail when sampled p95 exceeds baseline by both 50% and 25 ms.
- Fail any candidate observation above 2,000 ms, even when the baseline host is also slow.
- Fail a workload worker after 30 seconds; preflight also limits overall benchmark time.

These are initial **internal review budgets**, not customer promises. Keep the fixed baseline to catch cumulative growth across several small releases. If noisy infrastructure causes failure, investigate the host, then make one controlled repeat and retain both reports. Do not rerun until green, change workloads, reduce rounds, move the baseline or loosen budgets merely to pass. An intentional budget change needs a documented owner decision with detection and benign-impact evidence.

## Current observations and gaps

The retained evals/performance-v044.json compares v0.4.3 with v0.4.4 on four synthetic inputs: roughly 16–24% slower local medians. That motivated the gate. New reports go to performance-latest.json unless --output selects a dated artifact. Historical reports are not overwritten. They are observations of the named host, not API response-time guarantees.

The gate excludes HTTP/JSON parsing, metering, network distance, cold starts, gateway overhead, concurrency and queueing. Fly may suspend the machine while idle. A production capacity/load study remains separate and must be scoped to the dedicated Trismag deployment and coordinated before applying load. A healthy /health response is not a latency benchmark.

The weekly local monitor runs this synthetic gate and reports only meaningful regressions, failures or missing evidence. It does not collect customer messages or reset quotas. It complements the release gate; it cannot run when its execution environment is unavailable.

## Rules for future detection changes

Measure against the pinned engine and previous release. Prefer one normalization pass, bounded matching, inexpensive prechecks and grouping compatible patterns; prove equivalence before replacing the matcher. Add hostile near-misses and maximum-size probes for new grammar. Profile before changing algorithms. Avoid an extra whole-input sweep for each new attack category and avoid regex constructions with uncontrolled backtracking. Never skip inspection silently to meet a time budget: incomplete work yields no usable verdict.

Do not optimize by introducing a broad quoted-text or “benign preface” bypass. Run the detection regressions and benign neighbors together with timing checks. A future multi-pattern matcher or staged semantic detector needs its own coverage, CPU/memory and failure-mode evaluation; neither is claimed to exist today.

## Documentation-only release evidence reuse

Preflight optionally accepts --performance-report PATH. The guard revalidates a complete passing report from the last 24 hours only if candidate and baseline bytes, every workload, policy, Python/platform and sample count match. It recalculates budgets rather than trusting the pass flag. A detector change invalidates reuse; this supplies prior evidence, never a fresh timing result. The weekly monitor always measures again. Preserve later failures alongside reused evidence.

During this release, the first ten-workload run passed. Later workstation-load runs produced a tail regression (the separated-letter sample peaked at 1,675.8 ms) and a timeout after five of ten workloads. Normalized detector source was identical throughout. A concurrent unrelated application was consuming substantial CPU; host scheduling/load is a plausible contributor, not a proven diagnosis of every spike. Both failed reports are retained as performance-noise-2026-09-11.json and performance-controlled-2026-09-11.json. The documentation-only release uses the earlier exact-source report; no budget or baseline was loosened. Use a quiet, dedicated runner for more stable release measurements as the project grows.