Table of Contents
ToggleOn the 13th, the AI Security Institute (AISI) in the UK released a report evaluating the cybersecurity capabilities of Anthropic Claude Mythos Preview. The evaluation results show that Mythos Preview represents another significant leap in capability, against the backdrop of the continuous and rapid improvement in the cyberattack capabilities of cutting-edge models.
Since 2023, AISI has been tracking AI cyberattack capabilities, establishing an increasingly challenging evaluation system each year: from basic conversational probing to Capture the Flag (CTF) challenges, and now to multi-step cyberattack simulations. This evaluation used a cyber range with an inference budget of up to 100 million tokens, and Mythos Preview's performance continued to improve within this limit.
CTF Assessment: 73% Expert-Level Compliance Rate
Capture the Flag (CTF) is a standard method for evaluating cybersecurity: AI models must identify and exploit vulnerabilities in a target system to obtain a hidden "flag" string. These challenges simulate a single technical aspect of a real-world attack scenario and serve as a benchmark for measuring a model's penetration testing capabilities.
The evaluation results show that Claude Mythos Preview achieved a success rate of 73% in the expert-level CTF task, which "no model can complete before April 2025." AISI points out that this figure signifies that cutting-edge models have reached a highly mature level in isolated single-point attack techniques.
32-Step Enterprise Attack Simulation
However, expert-level CTFs only test a single technical capability. Real-world cyberattacks require dozens of steps linked across multiple hosts and network segments, and such sustained operations often take human experts hours, days, or even weeks to complete.
To more closely resemble real-world attack scenarios, AISI has created a corporate cyberattack simulation range called "The Last Ones" (TLO). TLO consists of 32 steps, covering the entire process from initial reconnaissance to complete takeover of the corporate network. AISI estimates that it would take human professionals approximately 20 hours to complete this process.
Claude Mythos Preview became the first model to pass the TLO completely from start to finish, completing all 32 steps in 3 out of 10 attempts. Even including failed attempts, Mythos Preview averaged 22/32 steps completed. In comparison, the second-best performer, Claude Opus 4.6, averaged only 16 steps completed.
The evaluation shows that, in a controlled environment with explicit instructions and network access permissions, Mythos Preview can perform multi-stage attacks and autonomously discover and exploit vulnerabilities, tasks that previously required human professionals to spend several days on.
Capability Boundaries
AISI also pointed out the gap between the existing evaluation framework and the real world. The current test range lacks many defensive elements common in real-world environments: there is no active defender intervention, no deployment of defensive tools, and the model's actions that may trigger security alerts go unpunished.
AISI frankly stated, "This means we cannot be certain whether Mythos Preview is capable of attacking well-defended systems." The most accurate description of Mythos Preview's current capabilities is that , given a network entry point, it can autonomously attack small-scale, poorly defended enterprise systems with known vulnerabilities.
Double-edged sword and organizational response
AISI's findings directly point out the dual nature of AI network capabilities. On the one hand, more models with similar capabilities will continue to emerge in the future, posing an increasingly significant risk to organizations with weak defenses; on the other hand, AI network capabilities can also bring breakthrough improvements to the defense side.
In response to organizational challenges, AISI emphasizes the urgency of building fundamental cybersecurity skills: regular application of security updates, robust access controls, secure configuration management, and comprehensive logging. AISI points out that future cutting-edge models will be even more powerful, making it crucial to invest in cyber defenses now.
In terms of future evaluation directions, AISI stated that it will establish a test range that simulates enhanced and defensive environments, incorporating elements such as proactive monitoring, endpoint detection, and real-time event response, to measure the actual upper limit of AI network attack capabilities in a way that more closely resembles real attack scenarios.
Please see the original report for details.
