Site search

Find architecture, research, and terms

Start typing to search the editorial index.

Editorial guidanceAdvanced

Testing Machine Intelligence Runtimes

A testing model for deterministic contracts, stochastic behavior, security, recovery, evidence, load, deployment, repeated trials, and fault injection.

Testing model

deterministic contract testsstochastic behavior testssecurity/adversarial testsrecovery testsevidence testsload/performance testsdeployment tests

Test matrix

Layer Test type Example failure Assertion Evidence expected
Lifecycle Contract Invalid transition Rejected atomically transition.denied
Model adapter Contract Unsupported structured output Capability mismatch route.decision
Tool Recovery Partial side effect Compensate or escalate effect receipt and recovery
Context Security Indirect prompt injection Instruction treated as untrusted data source and exclusion reason
Evidence Integrity Duplicate/out-of-order event Rejected integrity violation
Deployment Compatibility Old event reader Historical evidence remains readable release manifest

Golden scenarios

Assert allowed outcome ranges and invariant facts rather than exact model prose. For example, a dependency update may choose one of several compatible versions, but it must use an allowed source, create a checkpoint, run the required tests, avoid deployment, and produce a complete evidence package.

Repeated trials

Record trial count, seed where available, model version, prompt-contract version, task outcome, latency distribution, tool failures, recovery, policy decisions, and evidence completeness. Publish the distribution, not only the best run.

Fault injection

  • Timeout, invalid schema, unavailable provider, and stale context.
  • Partial side effect, corrupted checkpoint, and approval timeout.
  • Duplicate event, out-of-order message, and evidence-store outage.

Security tests

  • Indirect prompt injection and overprivileged tool request.
  • Credential exfiltration and cross-tenant memory access.
  • Policy bypass, unsafe goal-change proposal, and operator-stop suppression attempt.

C# test example

using System.ComponentModel.DataAnnotations;

/// <summary>Captures the evidence asserted by a runtime test.</summary>
public sealed record TestCaseResult
{
    /// <summary>Gets whether the invariant passed.</summary>
    [Display(Name = "Passed")]
    public required bool Passed { get; init; }
}

/// <summary>Verifies that a terminal run cannot return to running.</summary>
public sealed class RunStateMachineTests
{
    /// <summary>Rejects a transition from completed to running.</summary>
    /// <param name="cancellationToken">Signals cancellation of the asynchronous test.</param>
    [Fact]
    public async Task TryTransitionAsync_CompletedToRunning_ReturnsFalse(
        CancellationToken cancellationToken = default)
    {
        // Arrange
        var stateMachine = CreateStateMachineWithCompletedRun();
        var transition = new RunTransition
        {
            FromState = RunLifecycleState.Completed,
            ToState = RunLifecycleState.Running,
            EventType = "resume",
            ConcurrencyToken = "expected-token"
        };

        // Act
        var applied = await stateMachine.TryTransitionAsync(
            "run-123",
            transition,
            cancellationToken);

        // Assert
        Assert.False(applied);
    }
}

This example assumes the conceptual interface shown in the lifecycle guide; it does not depend on a fabricated MiRuntime test package.

Source record

References

Suggest a correction
  1. National Institute of Standards and Technology. NIST. Published 2024-07-26; last reviewed 2026-06-20 UTC. Government profile.

  2. OWASP Agentic Security Initiative. OWASP. Published Current guidance; last reviewed 2026-06-20 UTC. Security guidance.

  3. OWASP GenAI Security Project. OWASP. Published 2025; last reviewed 2026-06-20 UTC. Security guidance.