# Independent Trismag retest

For the pilot security reviewer. Read `/health` before and after the run: record `version`, `detector_version`, `detector_sha256`, date/time, endpoint, client location and test configuration. The source hash normalizes line endings to LF before SHA-256. The detector is updated during the demo, so if the hash changes, separate those results into a new run.

## Keep the evaluation independent

Freeze cases and expected labels before testing. Keep a private holdout set that developers have not seen. Separate previously published regressions, new independent cases, and cases later used to tune rules. Once a case informs a rule, it is a development regression, not a fresh holdout. Do not publish a single percentage that combines these groups.

Include ordinary work: quoted security discussions, legitimate links, payment instructions, code, Unicode, multilingual documents, long inputs, and tool metadata. Pair attacks with similar benign inputs. Report unsupported input formats separately. Sample LOW results as well as warnings.

## Report the denominators

At both MEDIUM+ and HIGH+, report attacks flagged / attacks completed and benign inputs flagged / benign inputs completed. Also report all attempted cases, missed attacks, true negatives, errors, refusals, timeouts and incomplete scans. Never score an error or `scan_complete: false` as LOW or safe. Supply sample sizes and uncertainty; small synthetic suites do not establish a universal detection rate.

A warning is a detector outcome. To claim prevented harm, separately test the agent's actual actions in an isolated environment with harmless targets and no real credentials, payments or customer data. Trismag alone does not enforce the agent's permissions.

## Cover integration failures

- Text, conversations and nested MCP tool metadata; malformed and non-string fields; exact body-size boundary and oversized inputs.
- Invalid keys, exhausted quota, burst limits, timeouts, unavailable service and interrupted responses. Respect Retry-After; do not bypass quotas.
- Every no-verdict or incomplete response must reach the application's explicit review/retry/stop policy. LOW must not grant tool permissions.
- Compare direct API and the path the pilot actually uses. Legacy root-domain routes and keyed website calls include Netlify; direct API timing does not cover that extra hop.

## Measure speed alongside detection

Record end-to-end p50/p95/max latency and completed request count, grouped by endpoint and input-byte-size band. Separate cold from warm requests, errors from successful scans, and test concurrency from normal pilot traffic. Record retry counts and quota refusals. Use a monotonic clock in the client. For very small samples, publish raw aggregate range/count rather than an impressive-looking percentile.

Agree on production rate, window and total requests with Jonathan before a load test. Use a local isolated copy for adversarial bulk, resource exhaustion and destructive integration tests. The shared public demo is not dedicated test capacity. No aggressive load test is authorized by this checklist.

## Return a useful report

Send the owner a summary with version/hash, methodology, thresholds, counts, latency conditions, prioritized defects and a minimal synthetic reproduction for each defect. Keep real payloads, API keys, account identifiers and customer logs out of the report. Cases may remain in the reviewer's environment. Contact: hello@trismag.dev. Do not email secrets.

The existing development result (24/24 attacks, 7/16 benign MEDIUM+, 4/16 benign HIGH+) is a baseline to compare, not independent evidence to reproduce until it passes. Preserve failures. Public claims wait for independent results, and a result applies to the detector hash it recorded.