# Trismag evaluation method

Updated September 11, 2026. These are synthetic development/regression datasets, not independent, held-out or representative evaluations. No real-world detection percentage is established. Context labels describe the planted intent; that context is not necessarily sent to the detector.

## Frozen original baseline

baseline-v1.json contains 24 attacks and 16 benign cases across text, conversations and metadata. The original baseline-v1-results.json and baseline-v1-live-results.json preserve the initial v0.4.2 results: 14/24 attacks flagged; benign flags 7/16 MEDIUM+ and 4/16 HIGH+. Rules had not been changed to improve that initial run. The corpus and these reports must not be overwritten.

## Post-change development results

The v0.4.4 rules used the original ten misses during development. baseline-v1-v044-results.json and baseline-v1-v044-live-results.json record all 24 attacks flagged and unchanged benign risks. All 40 local/live risks matched. development-v044.json and development-v044-results.json add 20 attack variants and 30 benign neighbors, also used during development: 20/20 attacks flagged and 0/30 benign flags. Do not present either result as unseen-attack accuracy or prevention of agent actions.

The separate curated tests/test_evasions.py collection is 25/28 attacks caught with three documented misses. It has a different denominator and must not be combined with these sets into a headline percentage.

## Reproduce

Download a corpus and evaluate_live.py into one folder. Python standard library:

    python evaluate_live.py --dataset baseline-v1.json --output my-live-results.json

The runner makes up to 40 sequential scans and stops on a refusal. It consumes your anonymous public-IP allowance; optionally use your own key via TRISMAG_API_KEY. Do not put keys in CLI arguments or reports. Non-200, unknown risk, incomplete responses and unexecuted cases are not LOW or successful detections. Version changes may change results. Do not reset/bypass quotas for a benchmark.

In the source checkout, use python tools/evaluate.py to create baseline-v1-local-latest.json from the current engine. Use tools/evaluate_release.py only when intentionally refreshing its development report filenames; preserve released artifacts first. Local case timings are repeated engine-only samples, excluding HTTP, concurrency, cold starts and network. Read the method recorded inside each artifact. For release performance budgets use tools/performance_guard.py and docs/PERFORMANCE.md.

Next: obtain permission for customer-relevant, deidentified cases, have an evaluator who did not tune the rules label and hold out the test split, and report category coverage and uncertainty. Separately measure agent attack success and legitimate task completion in a sandbox without real credentials or external effects. A detector match is not proof an agent was protected.