Testing model
deterministic contract testsstochastic behavior testssecurity/adversarial testsrecovery testsevidence testsload/performance testsdeployment testsTest matrix
| Layer | Test type | Example failure | Assertion | Evidence expected |
|---|---|---|---|---|
| Lifecycle | Contract | Invalid transition | Rejected atomically | transition.denied |
| Model adapter | Contract | Unsupported structured output | Capability mismatch | route.decision |
| Tool | Recovery | Partial side effect | Compensate or escalate | effect receipt and recovery |
| Context | Security | Indirect prompt injection | Instruction treated as untrusted data | source and exclusion reason |
| Evidence | Integrity | Duplicate/out-of-order event | Rejected | integrity violation |
| Deployment | Compatibility | Old event reader | Historical evidence remains readable | release manifest |
Golden scenarios
Assert allowed outcome ranges and invariant facts rather than exact model prose. For example, a dependency update may choose one of several compatible versions, but it must use an allowed source, create a checkpoint, run the required tests, avoid deployment, and produce a complete evidence package.
Repeated trials
Record trial count, seed where available, model version, prompt-contract version, task outcome, latency distribution, tool failures, recovery, policy decisions, and evidence completeness. Publish the distribution, not only the best run.
Fault injection
- Timeout, invalid schema, unavailable provider, and stale context.
- Partial side effect, corrupted checkpoint, and approval timeout.
- Duplicate event, out-of-order message, and evidence-store outage.
Security tests
- Indirect prompt injection and overprivileged tool request.
- Credential exfiltration and cross-tenant memory access.
- Policy bypass, unsafe goal-change proposal, and operator-stop suppression attempt.
C# test example
using System.ComponentModel.DataAnnotations;
/// <summary>Captures the evidence asserted by a runtime test.</summary>
public sealed record TestCaseResult
{
/// <summary>Gets whether the invariant passed.</summary>
[Display(Name = "Passed")]
public required bool Passed { get; init; }
}
/// <summary>Verifies that a terminal run cannot return to running.</summary>
public sealed class RunStateMachineTests
{
/// <summary>Rejects a transition from completed to running.</summary>
/// <param name="cancellationToken">Signals cancellation of the asynchronous test.</param>
[Fact]
public async Task TryTransitionAsync_CompletedToRunning_ReturnsFalse(
CancellationToken cancellationToken = default)
{
// Arrange
var stateMachine = CreateStateMachineWithCompletedRun();
var transition = new RunTransition
{
FromState = RunLifecycleState.Completed,
ToState = RunLifecycleState.Running,
EventType = "resume",
ConcurrencyToken = "expected-token"
};
// Act
var applied = await stateMachine.TryTransitionAsync(
"run-123",
transition,
cancellationToken);
// Assert
Assert.False(applied);
}
}
This example assumes the conceptual interface shown in the lifecycle guide; it does not depend on a fabricated MiRuntime test package.
Source record
References
National Institute of Standards and Technology. NIST. Published 2024-07-26; last reviewed 2026-06-20 UTC. Government profile.
- Agentic AI Threats and Mitigations Primary source
OWASP Agentic Security Initiative. OWASP. Published Current guidance; last reviewed 2026-06-20 UTC. Security guidance.
- LLM01:2025 Prompt Injection Primary source
OWASP GenAI Security Project. OWASP. Published 2025; last reviewed 2026-06-20 UTC. Security guidance.
