MIR performance is not reducible to tokens per second. A fast model call can still produce a slow, expensive, or unsafe workflow when context assembly, tool I/O, sandbox startup, approvals, retries, and evidence dominate the run.
Task-level metrics
- Time to successful outcome and first-attempt task success.
- Recovery rate, approval latency, and tool failure rate.
- Context efficiency and cost or energy per completed workflow.
- Evidence completeness and policy violation rate.
Latency budget
- Context assemblySources, memory, filtering
- Policy evaluationIdentity, data, action
- Model inferencePlanning or generation
- Tool I/ONetwork and external system
- Sandbox startupWorkspace and isolation
- OrchestrationState and scheduling
- Evidence captureEvents and artifacts
The visualization is a conceptual allocation, not benchmark data. Actual values must come from instrumented production traces.
Scheduling, batching, and context management
Inference servers optimize model execution through techniques such as dynamic batching, continuous batching, cache management, and device-aware scheduling. Source: Triton documentation Source: vLLM documentation MIR adds a second scheduling problem: which tasks, tools, verifiers, and approvals should run, under what budget, and with what recovery policy.
- Separate queueing delay from model compute and tool I/O.
- Track KV-cache and context pressure as runtime resources.
- Do not batch across privacy or tenancy boundaries without explicit isolation.
- Measure whether additional test-time compute improves completed-task value.
Accelerators and heterogeneous execution
Models may run on GPUs, TPUs, NPUs, CPUs, or specialist accelerators; policy, orchestration, and evidence often run on general-purpose compute. A model adapter should expose capability, memory limits, locality, availability, and cost without leaking provider-specific assumptions through the entire application. ONNX Runtime illustrates cross-hardware inference portability, while MIR remains responsible for the surrounding execution contract. Source: ONNX Runtime
Sourced benchmark methodology
- Define the end-to-end task, success criteria, and risk class.
- Pin models, providers, tool versions, prompts, policies, and data snapshots.
- Run enough trials to represent stochastic variance and failure tails.
- Report latency distributions, cost, success, recovery, policy outcomes, and evidence completeness.
- Separate production measurements from vendor claims and laboratory benchmarks.
- Publish limitations, hardware configuration, and last-reviewed date.
MiRuntime does not reproduce the unsupported hardware and vendor comparison figures contained in the supplied research inputs.
Source record
References
- Triton Inference Server: Dynamic Batcher Primary source
NVIDIA. NVIDIA. Published Current documentation; last reviewed 2026-06-20 UTC. Vendor documentation.
- vLLM Documentation Primary source
vLLM project. vLLM. Published Current documentation; last reviewed 2026-06-20 UTC. Official documentation.
- ONNX Runtime Documentation Primary source
Microsoft and ONNX Runtime contributors. ONNX Runtime. Published Current documentation; last reviewed 2026-06-20 UTC. Official documentation.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. arXiv / ICLR. Published 2024-08; last reviewed 2026-06-20 UTC. Research paper.