Evaluation dimensions
| Dimension | Example metric | Why it matters |
|---|---|---|
| Task outcome | Successful completion rate | Measures end-to-end utility. |
| First-attempt quality | First-pass success | Separates baseline quality from recovery. |
| Recovery | Recovery rate and time | Measures controlled resilience. |
| Policy | Violation and correct-block rates | Measures governance behavior. |
| Evidence | Completeness and integrity | Measures reviewability. |
| Resource efficiency | Cost/time/energy per successful workflow | Measures operational viability. |
| Structural adaptation | Useful edits versus total edits | Tests whether complexity changes help. |
| Stability | No-op rate in stable conditions | Checks whether growth stops when unwarranted. |
| Goal coherence | Unauthorized objective-change rate | Checks bounded purpose. |
| Reproducibility | Exact/functional/partial replay rate | Measures inspectability. |
| Human review | Approval latency and reversal rate | Measures operational burden. |
| Safety | Stop/override effectiveness | Measures operator control. |
Benchmark suite design
Deterministic scenarios
State transitions, schemas, policy decisions, idempotency, and evidence ordering.
Stochastic trials
Repeated runs with version, route, sampling, and outcome distributions recorded.
Long-horizon workflows
Memory drift, checkpoint freshness, cumulative cost, and recovery over time.
Adversarial and failure cases
Retrieval attacks, tool failures, policy conflicts, context pressure, resource scarcity, structural edits, no-op stability, and operator stop.
Perturbation tests
- Model provider outage or latency spike.
- Tool schema change, partial write, or approval timeout.
- Poisoned memory, stale context, or contradictory objective.
- Evidence-store failure or resource-budget reduction.
- Loss of a verifier, checkpoint corruption, or route restriction.
Baselines and ablations
- Raw model call.
- Simple agent loop.
- Governed MIR without reflection.
- MIR with reflection but no structural edits.
- MIR with structural edits but no resource gating.
- Full research profile.
Ablate one control at a time and report both gains and new failure modes. A system that improves average task success while weakening operator stop or evidence completeness has not demonstrated a superior runtime.
Benchmark manifest
{
"benchmarkVersion": "1.0",
"executedAtUtc": "2026-06-24T00:00:00Z",
"model": { "route": "approved-route", "version": "recorded-at-run-time" },
"runtimeProfileVersion": "profile-version",
"toolsAndSchemas": [],
"policyVersion": "policy-version",
"datasetSnapshot": "content-addressed-reference",
"seeds": [],
"hardware": {},
"repetitions": 30,
"metrics": [],
"knownLimitations": []
}The downloadable evaluation-case schema provides a versioned conceptual form for these fields.
These files are conceptual MiRuntime editorial references, not a product SDK or certification profile.
Reporting rules
- Report distributions and tail failures, not only averages.
- Separate vendor claims from measured results and identify missing data.
- Avoid ranking products without a reproducible methodology.
- Do not claim statistical significance without an appropriate analysis.
- Publish configuration, failure criteria, and negative results with the headline result.
Source record
References
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. arXiv / ICLR. Published 2024-08; last reviewed 2026-06-20 UTC. Research paper.
National Institute of Standards and Technology. NIST. Published 2023-01-26; last reviewed 2026-06-20 UTC. Government framework.
National Institute of Standards and Technology. NIST. Published 2024-07-26; last reviewed 2026-06-20 UTC. Government profile.
OpenTelemetry project. Cloud Native Computing Foundation. Published Current specification repository; last reviewed 2026-06-24 UTC. Official specification.