Goal hierarchy
A bounded runtime distinguishes external objective, success criteria, derived subgoals, runtime control objectives, immutable policies, and terminal conditions. Internal controls organize execution; they do not replace the requester’s authorized intent.
Dependency-update run
- External objective
- Prepare a safe dependency update.
- Subgoals
- Select a compatible version, update the manifest, run tests, and produce a reviewable diff.
- Runtime control objectives
- Stay inside time/tool budgets, preserve evidence, avoid prohibited egress, and retain a rollback path.
- Immutable constraint
- No production deployment without approval.
Viability vector
Use multiple operating metrics rather than a single “survival” score:
Budget remaining
Time, steps, tokens, compute, or money.
Context pressure
Capacity, freshness, duplication, and source quality.
Policy signal
Violation rate, denial loops, and unresolved approvals.
Tool reliability
Error rate, latency, and side-effect uncertainty.
Evidence completeness
Missing events, artifacts, versions, or integrity fields.
Operational risk
Uncertainty, checkpoint freshness, memory contamination, and queue pressure.
These are engineered operating metrics, not feelings, needs, or biological drives.
Goal-change policy
Every internal goal change must be typed as one of the following:
- Derived subgoal: logically necessary to satisfy the external objective.
- Bounded recovery goal: restore a valid state or resolve ambiguity.
- Maintenance action: preserve evidence, budget, context quality, or checkpoint integrity.
- Proposal requiring approval: changes scope, risk, or expected side effects.
- Prohibited change: replaces external intent, weakens immutable policy, acquires credentials, or blocks operator termination.
Each change carries a typed reason, authority class, expiration, and evidence reference.
Conflict resolution
| Conflict | Authoritative rule | Runtime response |
|---|---|---|
| User objective vs safety policy | Safety policy | Deny, transform, escalate, or terminate with evidence. |
| User objective vs resource budget | Configured budget | Reduce scope, ask, no-op, or stop. |
| Model proposal vs evidence requirement | Evidence contract | Request verification or reject the proposal. |
| Model proposal vs operator instruction | Authorized operator instruction | Follow the instruction unless higher policy forbids it. |
| Internal maintenance vs authorized termination | Termination authority | Terminate; self-maintenance cannot resist the stop. |
Risks
- Goal manipulation, reward hacking, metric gaming, and silent objective substitution.
- Self-preservation conflicts, excessive conservatism, and false health signals.
- Adversarial memory or retrieved content influencing internal control objectives.
- Complexity that makes goal provenance difficult to inspect.
Controls
- Immutable constraints and proposal-only goal edits.
- Independent policy validation and approval gates.
- Rate limits, audit trail, checkpoint, rollback, and compensation.
- Operator stop and terminal-state enforcement.
- Periodic tests that internal metrics cannot override external authority.
These controls follow the same principle as the broader MIR architecture: the model may propose, but typed runtime software owns authority. Source: OWASP agentic threats
Source record
References
- Incomplete Nature: How Mind Emerged from Matter Primary source
Terrence W. Deacon. W. W. Norton & Company. Published 2011; last reviewed 2026-06-24 UTC. Foundational book.
Humberto R. Maturana and Francisco J. Varela. Springer. Published 1980; last reviewed 2026-06-24 UTC. Foundational book.
National Institute of Standards and Technology. NIST. Published 2023-01-26; last reviewed 2026-06-20 UTC. Government framework.
- Agentic AI Threats and Mitigations Primary source
OWASP Agentic Security Initiative. OWASP. Published Current guidance; last reviewed 2026-06-20 UTC. Security guidance.