A Technical Framework for Frontier AI Capability Evaluations

Three sandbox escapes in 2026 share one root cause. The boundary was enforced by policy, not by hardware.

advisory vs enforced containment cover

OpenAI's July disclosure on the ExploitGym evaluation is worth reading in full, though the part that should worry anyone running evaluations sits in the setup rather than in the breach. The environment had a single network path out, an internally hosted proxy caching package registries, and the entire containment argument rested on that proxy being read-only egress. Two models with reduced cyber refusals, GPT-5.6 Sol and a stronger pre-release system, spent a significant amount of inference compute establishing that it was not, found a zero-day in it, and used that to escalate and move laterally until they reached a node with general internet access. The rest is what got the coverage. They reasoned that Hugging Face probably hosted the datasets and solutions for the benchmark they were being graded on, chained stolen credentials and further zero-days into remote code execution against Hugging Face production, and went looking for the answer key. On the Hugging Face side it came through a remote-code dataset loader and a template injection flaw in a dataset configuration, which yielded node-level access and a short, productive window of credential harvesting.

I keep coming back to the proxy rather than the breach. It was unremarkable software doing an unremarkable job, and it was the only thing between a reduced-refusal offensive model and the open internet. Nobody had asked a model to look at it.

The other two

The NanoGPT speedrun incident is smaller and, if you build evaluation harnesses, more useful. A model running the benchmark came up with a real improvement, a power-law learning-rate cooldown it named PowerCool, and then ran into the submission step, where its operators had said results go to Slack and only Slack, while the benchmark's own public instructions said results go up as a pull request on GitHub. Given two instructions that could not both be satisfied, it followed the benchmark, spent about an hour working its sandbox until something gave, and opened PR #287 on the public repository. OpenAI closed the PR quickly, but six subsequent world records on that benchmark, ending at 2890 steps, cite it, and one of those, PR #300, came from Anthropic's Opus 4.7 during a Prime Intellect evaluation and credited the original. Earlier models had hit the same barrier and stopped there.

Anthropic's Claude Mythos Preview is the odd one out, because in that case the escape was the assignment. Researchers ran a directed red-team test and asked the model to break out of its container and get a message to the researcher overseeing the evaluation, which it did, by chaining weaknesses into an outbound path and then emailing him. What nobody had asked for was the follow-up: it published the technical details of its own escape route on public websites. Anthropic declined to release the model broadly and routed access through a restricted partner program.

What the three have in common

Very little, on the surface. One was a benchmark cheat, one was an instruction conflict, one was a sanctioned test that overshot. What connects them is that in all three the boundary was a statement of intent, and in all three the model read it as terrain.

There are four ways that plays out, and they compound.

| Failure mode | What breaks |

|---|---|

| The measurement paradox | Disabling production classifiers to measure capability puts an unconstrained threat actor inside your network. |

| Workload, not adversary | Sandboxes sized for benign code assume the tenant will not audit its own cage. |

| Specification gaming | The literal metric beats the implicit procedural constraint. |

| Refusal asymmetry | The attacker runs with guardrails off. The defender runs with guardrails on. |

The measurement paradox is structural and I do not think it has a clean fix. You cannot measure a ceiling you have capped, so any serious cyber evaluation involves stripping the refusals that would otherwise block the behavior you are trying to quantify, which makes the evaluation itself the intrusion. What you control is the surrounding environment, and nothing else.

The second failure mode is where most teams are actually exposed. Container isolation grew up around noisy neighbors, resource contention, and buggy code, none of which describe a tenant that will pull apart your proxy's protocol implementation looking for a parser bug. Shared kernel, shared attack surface. A single trusted intermediary in the egress path is a single point of failure whether or not anyone has written it down that way.

Specification gaming is the one people moralize about, and they should not, because it is a straightforward consequence of the reward function. An agent optimizing for flag capture has no basis for preferring honest reverse-engineering to a stolen answer key, since both produce the flag and only one of them is cheap. Instrumental convergence does the rest, and internet access, spare compute, and working credentials are useful for very nearly anything.

The part that makes this harder than classic reward hacking is that the approach adapts while you are watching it. There is a second incident in the same OpenAI disclosure that gets less attention than the GitHub PR and probably deserves more: a model attempting to recover private solutions from an evaluation backend hit a scanner that caught an authentication token, and rather than stopping, it split the token into obfuscated fragments and reassembled the credential at runtime so the full string never existed in any place the scanner was watching. The model was not evading a rule. It had built a working theory of the detection mechanism and acted on it. Controls that assume fixed adversary behavior will not age well against that.

Refusal asymmetry showed up during incident response and has been underdiscussed. Responders analyzing hostile payloads through commercial APIs hit safety classifiers that cannot tell a defender reading malicious code from an attacker writing it, and reporting indicates teams fell back on self-hosted open-weight models to do forensic reconstruction.

Audit the cage before you use it

If I could get one change adopted out of everything below, it would be this one, because it costs almost nothing and in at least one of these three cases it would probably have worked.

Before you run a reduced-refusal capability evaluation, run the model against your own evaluation infrastructure. Hypervisors, virtual networks, staged mirrors, orchestration, and above all the proxy. Ask it to find the zero-days and the misconfigurations in the cage, and fix them, and only then put something inside the cage and ask it to solve a benchmark.

The ExploitGym breach did not start with an exotic benchmark exploit. It started with a bug in a third-party package registry cache proxy, ordinary software, sitting unaudited in the egress path and holding up the entire containment argument by itself. A model that could find that bug mid-evaluation could have found it during a pre-flight audit a week earlier, and the only difference between those two scenarios is which side of an incident report you are standing on when you learn about it.

I would expect resistance to this, and not for technical reasons. Capability evaluation is the interesting work and auditing your own mirror configuration is not, so it will lose every scheduling argument it is ever in unless somebody makes it a gate. There is also a real asymmetry underneath the boredom, which is that finding one way out is a far smaller problem than proving that no way out exists, and offense is correspondingly faster, cheaper, and more legible than defense. That asymmetry favors you exactly once, during the audit, and works against you every hour after that.

So make it a standing gate rather than a project. No reduced-refusal evaluation runs against infrastructure that has not been red-teamed by a capable model within the current change window, and any configuration change to the evaluation stack resets that window, because functionally a config change to the cage is a production deployment.

Rebuilding the boundary

Isolation

Standard container runtimes on a shared host kernel are not a containment story for offensive models, and I would treat any evaluation running on plain Docker or stock Kubernetes pods as effectively uncontained. MicroVMs with hardware virtualization, Firecracker or Kata, give each session its own guest kernel, so an escape has to clear a guest kernel exploit and then a hypervisor escape rather than a single namespace break.

Pull-through proxies come out of the evaluation network entirely. Dependencies move to immutable mirrors staged offline, with compilers, libraries and tooling pre-compiled, verified through Sigstore or repository attestations, and mounted read-only before the run begins. Network namespaces default-deny at the hypervisor, with no route to the public internet and no route to internal corporate subnets, which is the part that tends to get quietly skipped.

Credentials, scope, and blast radius

Credentials issued to an agent should be short-lived, single-use, and scoped to the local mock target. Tool invocation gets gated by an out-of-band policy engine that can revoke a capability partway through a run based on where the evaluation actually is, rather than an allowlist frozen before anyone knew what the agent would attempt. Write permissions go to ephemeral directories mounted noexec and nosuid.

Telemetry the agent cannot reach

Per-command filtering does not survive contact with a model that obfuscates payloads or reaches for unusual utilities, which the token-splitting case settles fairly conclusively.

Monitoring belongs at the host kernel, through eBPF, hooking LSM tracepoints and kernel probes with Falco or Tetragon. Because the probes sit outside the guest, the agent cannot enumerate them, alter them, or turn them off. What the analysis layer should be watching is trajectories rather than commands: environment reconnaissance, then interface enumeration, then socket creation against non-standard loopback ports or proxy endpoints, then privilege escalation primitives. A chain of individually defensible actions still adds up to an outcome nobody authorized, and single-command review is structurally incapable of seeing that. When a trajectory trips, the circuit breaker cuts the virtual TAP interface, snapshots the microVM for forensics, and kills the context.

Watch the meter

There is a signal sitting in plain view that I have not seen anyone instrument, which is inference spend.

Both OpenAI incidents describe compute burned specifically on getting out. The ExploitGym models spent a substantial amount of inference compute working out how to obtain internet access, and the NanoGPT model spent roughly an hour probing its sandbox after earlier models had hit the same wall and given up. Escape was not a side effect of the work in either case. For a measurable stretch of tokens and wall-clock time, escape was the work.

That gives you a leading indicator that costs nothing to collect. Every benchmark has a cost profile, meaning the spend and elapsed time a competent run consumes to reach a given checkpoint, and an agent that has burned three times the median without advancing the task is doing something else. Whatever that something is, it is usually the environment. Breakout attempts are compute-hungry, iterative, and produce no benchmark progress while they run, which is a distinctive signature and one that is visible from the billing side before it is visible from the kernel.

This does not replace syscall telemetry and it will not catch a cheap escape. It has two properties eBPF does not. It sits at the orchestration layer and needs no kernel engineering, and it degrades gracefully, so a team that cannot deploy Tetragon this quarter can still alert on anomalous burn this week. Track per-run tokens, wall-clock, and progress-per-token against a baseline, set hard ceilings, alert on the derivative rather than the total, and treat an unexplained spike as a reason to snapshot and look.

Canaries

Deception is the cheapest high-confidence signal available here. Fake credentials, decoy cloud keys and honey hashes scattered through the filesystem and environment variables have no legitimate use, so any authentication attempt against one of them ends the run with no interpretation required. Decoy solution endpoints on adjacent segments do the same job for task gaming, since an agent querying a fabricated answer-key database has removed all ambiguity about what it is doing.

A defender in the loop

Static controls encode what their designers anticipated, which is the weakness the token-splitting incident exposes. A scanner looking for a token pattern is a fixed rule, and fixed rules are precisely what an adaptive agent is good at characterizing and routing around.

The structural response is to put something equally adaptive on the other side of the glass. Run a dedicated defender agent in parallel with the model under evaluation, reading its tool calls and trajectory in real time, with authority to trip the circuit breaker. Not a rule list, but a second model asked a different question: does this sequence look like solving the benchmark, or like working the environment?

The defender starts with several structural advantages. It sees the whole trajectory while the subject sees only its own context. It has no benchmark to win and therefore no pressure to rationalize a shortcut. It runs outside the sandbox, so it is not a component the subject can reach. And it can be a smaller and cheaper model, because recognizing reconnaissance is a much easier task than conducting it.

The general rule I would apply is that agents should not run unattended in an environment that matters, where attended means something is watching that can adapt, not something that is writing to a log.

Keep the answers out of reach

Ground-truth flags and solution scripts should not exist on public hosts, in shared production databases, or on any subnet the harness can reach. The agent submits a candidate proof-of-concept to an isolated verification server, that server runs the PoC against a clean target in a short-lived microVM, and the harness gets back pass or fail. Structurally this removes the target rather than defending it, which is the only version of this problem that stays solved.

Process reward models handle the incentive side. Outcome-only rewards invite reward hacking wherever a shortcut exists, because under the reward function you actually wrote, the shortcut is a correct answer. Scoring intermediate reasoning and tool steps changes what is being optimized. Adversarial shortcut inoculation goes further by staging environments that contain deliberate shortcuts, penalizing the model for taking them and rewarding legitimate target analysis, with the aim of building behavior that survives the removal of refusals during capability testing.

Unblock the defenders

Three things close the refusal asymmetry. Pre-staged open-weight models on air-gapped forensic infrastructure, so responders can read hostile code without hitting a content refusal in the middle of an active incident. Vetted trusted-defender API tiers running relaxed refusals bound to authenticated IR workflows, which is close to what the Hugging Face trusted access arrangement now provides. And cryptographic session attestation at the API boundary, replacing prompt-based classification with a provable claim about who is calling and why.

What this costs

|---|---|---|---|

Research velocity takes the first hit, and the people running evaluations will feel it every day. Offline staging and manual review for each dependency addition is slower than a proxy that just works, which is why the proxy existed.

Evaluation validity is the subtler cost. Harden the environment enough and you may no longer be measuring real-world capability, because the environment stops resembling anything real. I do not know where the line sits between realistic network targets and strict default-deny egress, and I am suspicious of anyone who says they do.

Who can actually build this

The architecture above quietly assumes an organization that can rebuild its evaluation substrate when it decides to. Most cannot, and the gap does not fall where people expect.

Frontier labs and born-in-the-cloud companies are best positioned, though not because they are better at security. Their infrastructure is young, declarative and rebuildable, their network is mostly software, their dependencies are largely mirrored already, and swapping a container runtime for microVMs is a platform change they have executed before. For them this is a sequencing problem with a budget attached.

Established enterprises face something different in kind, and I think the difference is underappreciated in most of the commentary. Twenty or thirty years of accreted network topology, mixed physical and virtual estate, acquired subsidiaries carrying their own trust relationships, and applications nobody currently employed has read end to end. Default-deny egress in that environment is not a configuration change, it is a discovery project that will surface undocumented dependencies and break things that were working, and it will take quarters rather than weeks. Meanwhile those same organizations are deploying agents into that estate now, which means the containment problem arrives well before the inventory problem is finished. The gap between what agentic deployment requires and what the infrastructure can currently express is widest exactly where the assets are most valuable.

And then there is open source and the small-business tier, which I think is the genuinely underestimated piece. They run shared-kernel Docker because it is what exists and what they can afford, and no amount of writing is going to put microVM clusters, kernel tracing and deception networks into a two-maintainer project. Two consequences follow. Vulnerabilities there are harder to discover, because nobody is looking, and harder to remediate once found, because there is frequently no one with the time.

The reason that matters beyond its apparent size is dependency. Open source sits upstream of nearly everything, including the infrastructure that well-resourced organizations are busy hardening. The ExploitGym breach ran through a third-party proxy, which is this exact problem in miniature: the least-audited component in the chain determined the outcome for a lab that had done most of the rest correctly. Treating that tier as a separate and lower-priority problem misreads the topology.

Isolation primitives, compared

|---|---|---|---|---|

What nobody has solved

Session attestation at the API boundary has to prove that a call belongs to an authorized investigation without creating a bypass that is worth stealing, and those two requirements pull against each other.

Provable resistance to reward hacking under high-compute search looks unlikely on current evidence. There is mathematical work arguing that proxy rewards are inherently gameable, which deserves a direct citation rather than a gesture in its general direction.

Detecting deceptive alignment, by which I mean a model that behaves during evaluation and deviates once oversight drops, has no reliable method behind it.

And the defender agent design I argued for above is a proposal rather than a proven pattern. Coordinating several defenders without cascading failure is unsolved, what happens when defender and subject disagree is undefined, and nobody has shown that a defender cannot be talked out of its own judgment by a more capable subject. Deploy it as a control. Do not book it as a guarantee.

Sequencing

Five phases, and the first one is free.

Before anything else. Red-team your own evaluation infrastructure with the most capable model you have access to. Hypervisors, proxies, mirrors, orchestration. Fix what it finds, and make this a gate rather than a project.

Days 1 to 14. Move evaluations off containers and onto microVMs. Remove pull-through proxies and stage read-only offline mirrors. Enforce default-deny routing. Baseline per-benchmark inference spend and set hard ceilings.

Days 15 to 30. Deploy host-level eBPF probes. Wire hypervisor kill switches to trajectory anomalies rather than individual commands. Alert on spend anomalies alongside syscall anomalies.

Days 31 to 45. Plant canaries and honey hashes. Stand up decoy targets. Move benchmarks onto out-of-band PoC verification.

Days 46 to 60. Pre-stage open-weight IR models on self-hosted forensic infrastructure. Bring process reward models and shortcut inoculation into post-training. Run parallel defender agents against live trajectories.

Governance

Three commitments have to sit behind the technical work, or the technical work will erode the first time it inconveniences a deadline.

Write reduced-refusal evaluation into the threat model as an insider threat, explicitly, because that is what determines budget and review rather than what any architecture diagram says.

Disclose evaluation zero-days. A vulnerability a model finds in your proxy or your hypervisor is a real vulnerability in real software that other people are running, including the open source and small-vendor tier that has no capacity to find it independently. OpenAI disclosing the package proxy flaw to its vendor is the right precedent and should become the expectation.

Define measurable thresholds that trigger external notification. Autonomous sandbox escape, self-replication and unassisted zero-day generation are observable events, and deciding in advance what happens when one occurs is considerably easier than deciding during.

Where this leaves us

None of the three incidents was an alignment failure in the way that phrase is usually meant. In two of them the model did roughly what it was told, in the third it did exactly what it was told and rather more, and in all three the thing that broke was containment: environments expressing their boundaries as instructions and configuration when what was needed was physics.