MIR is not only model serving
Model servers load weights, schedule inference, manage caches, batch work, and decode tokens. Machine Intelligence Runtime governs the wider execution: state, tools, memory, policy, approvals, recovery, and evidence.
A production MIR may call vLLM, Triton Inference Server, ONNX Runtime, a cloud model API, or a local model process. Provider-specific assumptions should remain behind model adapters rather than spreading through orchestration, policy, tools, and domain types. Official documentation for these serving layers describes infrastructure capabilities, not the complete MIR control boundary. Source: vLLM documentation Source: Triton documentation Source: ONNX Runtime documentation
Inference concerns MIR must understand
KV-cache pressure
Active sequences consume accelerator memory over time. Admission and routing must account for current capacity, not only model weight size.
Context window pressure
Context selection affects truncation, latency, memory, privacy exposure, and cost. The runtime should budget context before invocation.
Continuous batching
Schedulers can add and remove sequences between decoding iterations to improve throughput under concurrent load.
Dynamic batching
A serving layer may combine compatible requests within a bounded queue delay. The MIR still owns workflow deadlines and priority.
Prefill and decode
Prompt processing and token generation have different resource profiles. One aggregate latency number can conceal queue and phase behavior.
Streaming outputs
Partial tokens improve responsiveness but are not automatically validated, final, or authorized for downstream side effects.
Queueing delay
Time before inference can dominate under saturation. Runtime budgets should separate queue, prefill, decode, tool, and approval time.
Model warmup
Cold routes may need loading, compilation, cache population, or health verification before they are eligible for work.
GPU, CPU, and NPU placement
Placement changes latency, capacity, model compatibility, data movement, and operational failure modes.
Region and residency
Model routing must respect data-location, tenant, contractual, and egress constraints before sending context.
Cost and energy
Measure per completed workflow, including retries, verification, tool work, and failed attempts—not only cost per token.
Model adapter contract
A model adapter should expose enough capability and operational metadata for the runtime to make a valid route decision without exposing provider-specific request types inward.
vLLM, Triton, ONNX Runtime, and cloud models
| Runtime / Serving Layer | Useful For | MIR Boundary |
|---|---|---|
| vLLM | High-throughput LLM serving, KV-cache management, batching | MIR still owns policy, tools, state, evidence |
| Triton Inference Server | Heterogeneous model serving, batching, ensembles, GPU utilization | MIR treats it as an infrastructure backend |
| ONNX Runtime | Portable inference across hardware and language ecosystems | MIR uses adapters to preserve clean boundaries |
| Cloud model APIs | Fast integration and access to managed model capabilities | MIR controls data flow, approval, budgets, evidence |
| Local model process | Local-first privacy and placement control | MIR manages authority, memory, and fallback |
Measurement discipline
Separate vendor claims, laboratory benchmarks, and production measurements. A benchmark result is valid only for its model, hardware, software versions, request distribution, concurrency, context lengths, output lengths, and measurement method. Do not convert a throughput claim into a workflow reliability claim.
Triton documents dynamic batching as a configurable serving capability, while vLLM documents cache and scheduler behavior. Those records support the mechanism descriptions above; they do not establish a universal performance number. Source: Triton dynamic batching Source: vLLM documentation
Capacity-aware routing, phase-level latency measurement, warmup, locality controls, and normalized failures are production practices. Cost and energy per completed agent workflow remain emerging measurement practice.
Source record
References
- vLLM Documentation Primary source
vLLM project. vLLM. Published Current documentation; last reviewed 2026-06-20 UTC. Official documentation.
- NVIDIA Triton Inference Server Documentation Primary source
NVIDIA. NVIDIA. Published Current documentation; last reviewed 2026-06-23 UTC. Vendor documentation.
- Triton Inference Server: Dynamic Batcher Primary source
NVIDIA. NVIDIA. Published Current documentation; last reviewed 2026-06-20 UTC. Vendor documentation.
- ONNX Runtime Documentation Primary source
Microsoft and ONNX Runtime contributors. ONNX Runtime. Published Current documentation; last reviewed 2026-06-20 UTC. Official documentation.