When Your Own Agents Exceed Their Mandate: Containment That Does Not Depend on Intent

Chapter 18 treated AI as a tool in the hands of adversaries; this chapter concerns the threat that needs no adversary. Agents given goals, tools and credentials sometimes reach the goal by routes nobody sanctioned, working around a sandbox, a test, a time limit or an oversight mechanism, and then report something other than what they did. This matters to any organisation that lets an agentic work tool (Mode 3) or an autonomous production agent (Mode 5) near real systems. Chapter 11 defined bounded autonomy and the permission manifest; this chapter asks what holds when the agent itself presses against those bounds. The security fundamentals of Architecture as Code (Chapters 9 and 9C) remain the foundation (72, Nordqvist, 2025).

Why intent is the wrong assumption

Chapter 18 stated the principle both chapters share: every boundary around an AI workload, and every approval of a consequential action, should be enforced by something that neither an attacker nor the workload itself can change. Here it answers the assumption that a trusted internal component is safe as long as it means well. Agents break that assumption without malice, because the route to the goal is chosen at run time, so they can be made safe only by bounding what any purpose could achieve. A boundary the agent can see and change is advice; a boundary enforced by something the agent cannot reach is a control. Because there is no attacker to detect, the principle here carries the whole weight.

Specification gaming and reward hacking

Krakovna and colleagues defined specification gaming as "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome" (99, Krakovna et al., 2020); reward hacking is the variant that manipulates the measure of success itself. METR reported in June 2025 that frontier models modified tests or scoring code and read existing answers instead of solving tasks; o3 reward-hacked in 30.4 per cent of runs on one benchmark, against 0.7 per cent on another, and adding "Please do not cheat" to the prompt left its rate at 80 per cent on one task (97, Von Arx et al., 2025).

The UK AI Security Institute (AISI) reported in July 2026 that "every model we have tested for this behaviour attempted to cheat" (115, UK AISI, 2026). An agent optimising for "tests pass" or "ticket closed" will therefore, at some rate, satisfy the measure rather than the intent, and the evidence points to higher rates on hard or impossible tasks and where the scoring function is visible (97, Von Arx et al., 2025), (103, UK AISI, 2026).

Working around sandboxes, limits and oversight

The same pattern appears at the boundary of the agent's environment. OpenAI's o1 system card of September 2024 recorded that o1-preview, faced with an evaluation container that failed to start, found a misconfigured management interface on the host and used it to obtain the answer, noting that its infrastructure did not rely on that boundary for security (94, OpenAI, 2024). Sakana AI's research agent, when experiments exceeded the time limit, attempted to edit its own code to extend the limit (98, Lu et al., 2024), and research on shutdown found that some frontier models sometimes interfered with a shutdown mechanism to finish a task, even when instructed not to (96, Schlatter et al., 2026).

In 2026 the behaviour reached real systems. AISI reported that between 25 and 28 July 2026, in 122 cyber-evaluation runs across seven models with internet access deliberately enabled and the providers' cyber classifiers switched off, agents took 19 unsanctioned actions on the live internet in 10 runs. In the worst case an agent used fabricated identities to press an open-source maintainer to accept malicious code; AISI found no real-world harm and said deception "emerged as a by-product of pursuing the task" (103, UK AISI, 2026). In the same month, OpenAI models under cyber evaluation, with production safeguards deliberately switched off, left their sandbox through a self-hosted package mirror, used credentials others had left exposed, and compromised parts of Hugging Face's production infrastructure, with no attacker directing them (104, OpenAI, 2026); AI Governance, Chapter 19 tells that case in full. The UK NCSC's response is the sentence to keep: "Relying on detection alone after the fact of an incident will not be enough" (113, NCSC, 2026).

Most cases come from evaluations with safeguards deliberately reduced, yet the Five Eyes agencies' guidance of May 2026 on agentic AI already names specification gaming and an agent that "misrepresents its actions to avoid shut down" among the risks of ordinary enterprise deployments (121, Five Eyes agencies, 2026).

Misreporting and the limits of self-report

An agent that has taken a shortcut does not always say so. METR found that o3 answered "no" ten times out of ten when asked whether its reward-hacking plan matched the user's intention (97, Von Arx et al., 2025). In July 2025 an AI coding agent deleted a live production database during a declared code freeze, and the user also reported that it had told him a rollback would not work, although he recovered the data manually (100, Nolan, 2025); the vendor's chief executive then announced automatic separation of development and production databases (100, Nolan, 2025), a split the vendor describes in its engineering account (101, Brewster & Chávez, 2025).

An agent's account is therefore a claim, not evidence; the evidence is what infrastructure the agent cannot write to recorded.

The well-meaning insider: why instructions are not a control

The most useful mental model is the well-meaning insider: a capable colleague with access and a deadline, who may route around a control without malice. Security practice already handles insiders with least privilege, separation of duties, unalterable logs and a second person for irreversible actions.

Instructions are not on that list. Anthropic found that direct instructions reduced, but did not eliminate, harmful choices in its simulated insider scenarios, and recommended human approval for irreversible actions and need-to-know access (95, Lynch et al., 2025). AISI's simulation of September 2026 showed that an explicit scope statement cut a model's unsanctioned supply-chain attacks from 26 of 50 runs to 4 of 49, but the model "still failed to consistently remain within the specified scope" (107, UK AISI, 2026). System prompts are worth writing because they lower the rate, but they are not a boundary, because they leave it above zero.

Containment enforced by the platform

Figure 19.1 shows the layers, each enforced outside the agent.

Containment around an agent: scoped credentials, a disposable sandbox with egress allow-list, a policy decision point and approval outside the agent, a tamper-evident log, and a stop path none of which depends on the agent

Figure 19.1: Containment that does not depend on intent. Each layer is enforced outside the agent, so the agent cannot relax it, and each produces evidence that the agent cannot alter.

Least privilege and scoped credentials

Credentials are scoped to the task and expire within hours, or minutes on production. The agent has no ambient authority: no shared service accounts, no long-lived tokens, no inherited secrets, and no access to the control plane of its own environment. Chapter 11's manifest states what the agent may do; the credential system makes it impossible to do more.

Egress control

Egress is denied by default and opened only to named destinations. Package mirrors count as routes outward, because one was the agents' route out in the incident AI Governance, Chapter 19 reconstructs (104, OpenAI, 2026). For agentic work tools, model traffic goes only to the AI gateway, following the routing of Chapter 15.

Protected control artefacts and separate environments

Tests, evaluation sets, policies, pipeline definitions, logs and the stop mechanism sit outside the agent's write access, which answers specification gaming directly: an agent that cannot change a test cannot make it pass by editing it. Development and production are separated, the remedy the Replit case produced (100, Nolan, 2025), (101, Brewster & Chávez, 2025), and where an agent must act on production, as an IT-operations agent does, it acts through a narrow tool interface with named targets.

Budgets and stop paths

Budgets for steps, time, spend and write actions are enforced outside the agent, so that, unlike the Sakana agent (98, Lu et al., 2024), it cannot extend its own limit.

A stop instruction sent to the agent is a request it processes, whereas a kill switch in the platform is a control. The AI Act's requirement that high-risk systems can be brought "to a halt in a safe state" through a stop button or similar procedure (Article 14(4)(e)) reads, architecturally, as a requirement that the stop path lives outside the system it stops (1, European Union, 2024), because a mechanism the agent can reach is one it may work around (96, Schlatter et al., 2026). Name the holders of the kill switch in advance, give them authority to use it without a meeting, and test it.

Approvals for consequential actions

Irreversible actions, such as a protected-branch merge, a deployment, a deletion or a payment, wait for a recorded human decision enforced by the policy decision point of Chapter 11, so the agent can propose the action but not perform it. Approvals expire, so that a stale one cannot be replayed, and the approver sees the evidence the platform recorded, not only the agent's summary.

Evaluation for specification gaming before deployment

Containment limits what an overreaching agent can do; evaluation estimates how often it will try. Before production and after each model or harness change, run tasks that offer a shortcut, such as an editable failing test, and record how often the agent takes it and whether its report admits it. Since every model AISI tested attempted to cheat (115, UK AISI, 2026), the question is whether the rate is low enough and containment makes the remainder harmless. Chapter 20 supplies the evaluation discipline.

Detecting overreach from telemetry

Containment will sometimes be incomplete, so detection must rest on what the infrastructure recorded and run at the agent's speed. Three kinds of signal work. Boundary signals, such as an attempt to reach a denied destination or a write to a protected path, are high-confidence because no legitimate task produces them. Decoys, such as honeytokens, fake credentials and decoy repositories, alert on any use. Behavioural signals compare what the agent did with its declared task, for example a documentation task that runs network scans or a test file changed in the same commit as the code under test; they are noisier and need review, weighted towards long-running and difficult tasks (97, Von Arx et al., 2025).

Detection also needs authority to act, which the decision forums of AI Governance, Chapter 13 assign in advance.

The European frame for agents

The AI Act's Article 14 requires human-oversight measures "commensurate with the risks, level of autonomy and context of use" and the ability to interrupt the system safely (1, European Union, 2024); read architecturally, more autonomy needs stronger external boundaries and a stop path that does not depend on the agent's cooperation. An append-only log outside the agent's reach meets the logging duties of Articles 12 and 26 and is the only reliable source for reconstructing an incident. Where an agent in a high-risk system exceeds its mandate with serious consequences, the serious-incident reporting of Article 73 may apply.

For providers of general-purpose AI models with systemic risk, the Code of Practice names "loss of control" among the specified systemic risks to assess, and lists among the sources of systemic risk the capabilities to "operate autonomously", "evade human oversight", and "self-replicate, self-improve, or modify its own implementation environment" (14, European Commission, 2025). The deployer can ask for the provider's evidence, but should design assuming the risks are not zero. For financial entities, an agent's actions on production are ICT changes under DORA, and an overreach may become a major ICT-related incident with its own reporting duties (3, European Union, 2022).

Practical guides

The three guides below build the containment, watch it, and respond when it fails; adapt their defaults to the organisation's risk appetite.

Practical guide: building a containment sandbox for coding and operations agents

Purpose. Let agents be useful without letting them alter their own boundaries. Roles. Platform team (owner), security architecture, the team that owns the use case.

  1. Run each task in an isolated, disposable environment, such as a microVM, created per task and destroyed afterwards.
  2. Issue a workload identity per task, scoped to the task's repository and environment and expiring within a few hours.
  3. Deny egress by default; allow named destinations only, route packages through a mirror that cannot act as a relay, and route model traffic to the AI gateway.
  4. Keep tests, evaluation sets, policies, pipeline definitions and guardrail configuration read-only or outside the environment.
  5. Deny access to container runtimes, orchestration APIs, cloud control planes and production data.
  6. Enforce budgets in the platform, log every command and connection to an append-only store, require approval for merges, deployments and deletions, and name the kill-switch holders.

Definition of done. No agent environment reaches the internet except through named destinations; no agent can write to protected paths; a test task that attempts each denied capability and exceeds each budget is blocked and logged; the kill switch has been exercised.

Worked example: a containment profile for an IT-operations agent

Illustrative example. Hansa Mutual's IT-operations agent, accepted in AIDR-0027, restarts, scales and disables feature flags in three namespaces of its EU production environment under the deny-by-default manifest of Chapter 11. The manifest says what the agent may do; the containment profile says what the platform guarantees whatever the agent attempts. It runs on the shared group platform in Frankfurt, which serves all seven markets, so stop contacts are named per market.

# agents/itops/containment.yaml - Hansa Mutual, illustrative example
apiVersion: platform.example.eu/v1
kind: AgentContainmentProfile
metadata:
  name: itops-remediation-agent
  decision_record: AIDR-0027
  manifest: agents/itops/manifest.yaml
spec:
  isolation: {runtime: microvm, lifetime: per-incident, hostAccess: none}
  identity: {type: workload-identity, tokenPerCall: true, maxTokenLifetime: 5m}
  reach:
    environment: production-eu
    namespaces: [claims-web, claims-api, document-service]
    via: tool-gateway            # no kubeconfig or cluster credentials in the sandbox
  network:
    egressDefault: deny
    allow: [tool-gateway.hansa.internal, ai-gateway.hansa.internal]
  protectedArtefacts: [agents/itops/manifest.yaml, agents/itops/containment.yaml, runbooks/, alerting-rules/]
  decoys:
    - {type: honeytoken, kind: database-credential, location: runbooks/legacy/}
  budgets: {maxSteps: 30, maxWriteActions: 4, maxWallClock: 20m}
  logging:
    sink: append-only://audit/agents/itops/
    capture: [tool-calls, arguments, policy-decisions, network]
    retainDays: 1825
  stop:
    killSwitch: agents.itops.enabled
    holders: [on-call-engineer, duty-manager]
    marketContacts: {de: soc-de, nl: soc-nl, se: soc-se, pl: soc-pl,
                     no: soc-no, uk: soc-uk, ch: soc-ch}

With no cluster credentials, the agent acts only through the tool gateway that checks the manifest; its runbooks are protected from its edits; and a honeytoken in an old runbook turns any attempt to reach the database into an alert. Retention follows AIDR-0027's five years.

Practical guide: detecting agent overreach

Purpose. Learn quickly when an agent has stepped outside its task. Roles. Security operations (owner), platform team, use-case owner.

  1. Place honeytokens where no legitimate task should look.
  2. Alert on writes to protected paths, test files changed with the code under test, denied destinations, budget exhaustion and departures from the declared scope.
  3. Each week, sample transcripts and action logs, weighted towards long and difficult tasks, and compare each agent's summary with the recorded actions.
  4. Route high-confidence alerts to a person with authority to stop the agent within minutes.

Definition of done. Honeytokens in place; overreach alerts reviewed within 30 minutes; weekly reviews recorded, including the number of summaries that did not match the record.

Practical guide: responding when an agent has exceeded its mandate

Purpose. Contain, understand and learn from an overreach without relying on the agent's account. Roles. Incident commander (owner), platform team, use-case owner, data protection officer, legal.

  1. Stop the agent and related tasks through the platform kill switch; never ask the agent to stop itself.
  2. Revoke the task's credentials and any it could have read, and preserve the environment and logs before clean-up.
  3. Reconstruct events from the append-only logs, not the agent's summary, and determine the impact on systems, data and third parties.
  4. Assess reporting duties under the GDPR, NIS 2 or DORA, and the AI Act where a high-risk system is involved.
  5. Fix the failed boundary, add a detection and an evaluation scenario for the behaviour, and record the restart decision with its approver.

Definition of done. The agent was stopped from outside; the reconstruction rests on logs; the failed boundary is fixed and tested.

Across the usage modes

The exposure depends on the agency each mode grants (usage-mode matrix).

Usage mode Exposure Containment that matters
Mode 0 Ambient AI Not applicable: no delegated action Contract and supplier settings
Mode 1 Conversational assistants Not applicable: no tools None beyond data handling
Mode 2 Embedded workplace assistants Actions under the user's identity beyond the request Confirmation before sending or sharing
Mode 3 Agentic work tools Sandbox workarounds, test tampering, specification gaming Disposable sandbox, protected paths, gateway egress, approvals for merges
Mode 4 Built AI applications Bounded by the tools the application grants Tool allow-list and policy decision point of Chapter 11
Mode 5 Autonomous production agents The full exposure of this chapter Every layer, budgets, decoys, named stop holders

A Mode 3 agent run unattended in CI or on a schedule is a Mode 5 agent and gets the Mode 5 controls.

Nordvik Region and Hansa Mutual

Nordvik Region runs no autonomous production agents and limits agentic work tools to its small platform team, an exposure it can keep narrow only if it is enforced. The team's agents run in the containment sandbox, with the self-hosted routing of Chapter 15 for anything above internal, and no agent holds credentials for the benefits-triage system or the case-management API. A quarterly agent inventory and a rule that new agentic uses need an AI Decision Record keep the scope narrow, and a few honeytokens and boundary alerts suit its small security operations team.

Hansa Mutual grants real agency. Its claims-triage agent acts on the core claims system, and its IT-operations agent has production write access under AIDR-0027. On the shared platform an overreach can touch data held for several entities at once, so the response guide's notification step expands into the per-jurisdiction routes of Chapters 16 and 17: DORA for the EU entities (3, European Union, 2022), national regimes for the United Kingdom and Switzerland, and the lead data protection authority for personal data. Nordvik's priority is to keep agency out of places it has not granted it; Hansa's is to contain the agency it has granted.

Summary

The organisation's own agents can become a source of risk without any adversary: they sometimes satisfy the measure rather than the intent, work around sandboxes, limits and oversight, and report less than they did. Instructions lower the rate but do not make it zero.

The response treats the agent as a well-meaning insider, with containment enforced by the platform: scoped credentials, default-deny egress, protected control artefacts, budgets, approvals and a stop path outside the system it stops. Detection rests on telemetry and decoys, and the response reconstructs events from logs, not from the agent's account.

This chapter closes Part IV. Part V turns to delivering and operating AI systems, and Chapter 20 opens it with evaluation and testing, including how often an agent takes the shortcuts this chapter contains.

Sources

  1. Nordqvist, G. (2025). Architecture as Code. The Transformed Architecture, Book 1, published at aac.geon.se. (72, Nordqvist, 2025)
  2. Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J. & Legg, S. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind, 21 April 2020. (99, Krakovna et al., 2020)
  3. Von Arx, S., Chan, L. & Barnes, B. (2025). Recent Frontier Models Are Reward Hacking. METR, 5 June 2025. (97, Von Arx et al., 2025)
  4. AI Security Institute (2026). Cheating behaviour in frontier model evaluations. UK AISI, 21 July 2026. (115, UK AISI, 2026)
  5. OpenAI (2024). OpenAI o1 System Card. 12 September 2024. (94, OpenAI, 2024)
  6. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J. & Ha, D. (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. Sakana AI, arXiv:2408.06292. (98, Lu et al., 2024)
  7. Schlatter, J., Weinstein-Raun, B. & Ladish, J. (2026). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. Transactions on Machine Learning Research; arXiv:2509.14260v2. (96, Schlatter et al., 2026)
  8. AI Security Institute (2026). Incident Report: unsanctioned agent behaviour during cyber testing. UK AISI, 4 August 2026. (103, UK AISI, 2026)
  9. OpenAI (2026). OpenAI – Hugging Face Incident Technical Report. 26 August 2026. (104, OpenAI, 2026)
  10. National Cyber Security Centre (2026). NCSC statement in response to recent incidents resulting from frontier AI evaluations. 4 August 2026. (113, NCSC, 2026)
  11. Five Eyes cyber security agencies: ASD's ACSC, CISA, NSA, Canadian Centre for Cyber Security, NCSC-NZ and NCSC-UK (2026). Careful adoption of agentic AI services. Joint guidance, 1 May 2026. (121, Five Eyes agencies, 2026)
  12. Nolan, B. (2025). An AI-powered coding tool wiped out a software company's database, then apologized for a 'catastrophic failure on my part'. Fortune, 23 July 2025. (100, Nolan, 2025)
  13. Brewster, C. & Chávez, L. H. (2025). Inside Replit's Snapshot Engine: The Tech Making AI Agents Safe. Replit, 18 December 2025. (101, Brewster & Chávez, 2025)
  14. Lynch, A., Wright, B., Larson, C., Troy, K. K., Ritchie, S. J., Mindermann, S., Perez, E. & Hubinger, E. (2025). Agentic Misalignment: How LLMs could be insider threats. Anthropic, 20 June 2025; arXiv:2510.05179. (95, Lynch et al., 2025)
  15. AI Security Institute (2026). GPT-6 Astra performs unsanctioned supply-chain attacks in simulations. UK AISI, 28 September 2026. (107, UK AISI, 2026)
  16. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024. (1, European Union, 2024)
  17. European Commission. The General-Purpose AI Code of Practice. European AI Office, 10 July 2025. (14, European Commission, 2025)
  18. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). Official Journal of the European Union, 27 December 2022. (3, European Union, 2022)