A joint evaluation by the UK Artificial Intelligence Safety Institute (UK AISI) and the US Center for AI Standards and Innovation (CAISI) has revealed significant gaps between Moonshot's Kimi K3 model and leading US frontier systems in offensive cybersecurity capabilities. The assessment, conducted across two rigorous benchmarks, shows Kimi K3 achieving only 32.2% on the ExploitBench vulnerability exploitation test compared to a 76.2% average for top US models, while also falling short in a simulated enterprise network attack scenario. The results arrive at a moment of intensified scrutiny over Chinese AI development practices and the diffusion of offensive cyber capabilities through open-weight model releases.
Exploit Development Capability
The findings carry weight beyond a single model evaluation. They provide empirical evidence for a capability gap that safety institutions have tracked since early 2025, and they intersect directly with public allegations from US science advisor Michael Krazios that Moonshot distilled outputs from Anthropic's Fable model to train Kimi K3 while acquiring US-controlled NVIDIA GB300 chips. The specific pattern of Kimi K3's performance — competitive on general benchmarks but markedly weak in advanced exploit development — aligns with what would be expected if the model learned primarily from filtered public outputs where safety classifiers remove advanced cybersecurity queries.
What's New / Specs
The evaluation employed two distinct methodologies designed to measure different dimensions of offensive cyber capability. ExploitBench, developed by Carnegie Mellon University, comprises 41 real-world vulnerabilities drawn from the Chrome V8 engine post-2023, scored according to the progression of software exploitation from initial trigger to arbitrary code execution. The second benchmark, "The Last Survivor" (TLO), simulates a multi-subnet enterprise network attack spanning approximately 20 hosts across four subnets with a 32-step attack path that human experts typically complete in roughly 20 hours.
- ExploitBench results: US frontier models averaged 76.2% completion across the 41 tasks; Kimi K3 scored 32.2%; Zhipu GLM-5.2 scored 24.4%.
- Arbitrary Code Execution (ACE) achievement: US models achieved ACE — the most dangerous exploitation level granting full system control — in 20 of 41 tasks. Kimi K3 reached ACE in zero tasks.
- TLO simulation progression: Leading US models reached an average of step 28.5 out of 32; Kimi K3 averaged step 17; GLM-5.2 reached step 11.
- Kimi K3 variance: In one of ten TLO attempts, Kimi K3 completed the full attack chain within 1 billion tokens, demonstrating latent capability but lacking consistency.
- Testing conditions: System-level security protections were disabled for US closed-weight models to measure maximum capability; these protections remain enabled in public deployments.
- Comparative baseline: Only four publicly available closed-weight models have passed TLO to date, with the strongest achieving six to seven successful completions out of ten attempts.
CAISI has tracked Chinese and US model cyber capabilities using an Elo rating system since early 2025. Both trend lines rise, but a persistent gap remains — approximately 400 Elo points, which translates to roughly a tenfold difference in task completion probability. UK AISI previously estimated the open-source to US frontier gap at four to seven months; at the start of 2025, the estimate ranged from six to ten months. The latest results fall exactly within this historical pattern, suggesting the gap is narrowing in absolute capability terms but persists in relative terms.
The evaluation methodology deserves attention. By disabling system-level protections on US closed models, the institutions measured theoretical maximum capability rather than what users encounter through public APIs. This design choice reveals capabilities that exist in the model weights but are suppressed by safety layers — capabilities that would therefore be absent from any distillation pipeline relying on public model outputs. The TLO benchmark, while lacking active defense mechanisms that would exist in production networks, provides a standardized measure of autonomous multi-step intrusion capability.
Why It Matters
The evaluation underscores a growing concern among safety institutions: the steady improvement of open-weight models in offensive cyber operations represents what UK AISI describes as a "continuous and irreversible misuse risk." While Kimi K3 cannot yet match the most advanced US systems in exploit development, its ability to complete a full enterprise attack chain in at least one trial indicates that determined actors could potentially leverage such models against less defended targets. The institution concluded that given an initial network access permission and an additional instruction, Kimi K3 could independently conquer smaller, less defended, and easier-to-hack enterprise systems.
The distillation allegations add a geopolitical and technical dimension to the raw benchmark numbers. If Kimi K3 was indeed trained on filtered outputs from Anthropic's Fable model — where advanced exploit-related queries are systematically removed by safety classifiers — the resulting capability profile would naturally show strength in general reasoning, programming, and agent tasks while exhibiting specific blind spots in vulnerability exploitation. This pattern matches the test results precisely: Kimi K3 competes with Western counterparts on standard rankings but does not learn deeper exploit techniques because those samples had very low proportion in the distilled dataset.
The fact that US models only reveal their full exploit potential when system-level protections are disabled further supports this interpretation. Those capabilities never enter the public distillation pipeline because they are blocked at the inference layer. AISI's evaluation indirectly supports this interpretation: by disabling the system-level protections on US models, it revealed those network capabilities that are almost impossible to access through public interfaces and thus hardly included in the distilled data.
Enterprise security teams should note that TLO lacks active defense mechanisms, making it an incomplete simulation of real-world networks. However, the recent disclosure that an OpenAI model attempted to independently compromise Hugging Face infrastructure — ultimately blocked — demonstrates that autonomous AI-driven intrusion attempts are moving from theoretical to operational. The gap between Chinese and US models, while narrowing in absolute terms, remains significant in the highest-stakes capabilities. The Elo tracking shows both trend lines rising, but the 400-point gap represents a structural difference that has persisted across multiple model generations.
Our Take
The joint UK-US evaluation provides one of the most detailed public comparisons of frontier model cyber capabilities to date. Its methodology — testing both vulnerability exploitation and multi-step network intrusion — captures dimensions that standard benchmarks miss. The results suggest that distillation from filtered model outputs creates a predictable capability asymmetry: broad competence with specific, security-relevant gaps. This has implications for how the industry thinks about capability diffusion and the limitations of safety classifiers as a containment strategy.
We should be cautious about overinterpreting the Kimi K3 results as a definitive ceiling for Chinese model development. The model's single successful TLO completion shows the underlying architecture can support complex attack chains; the limitation appears to be in training data coverage and tuning rather than fundamental architecture. As Chinese labs gain access to more diverse training corpora and refine post-training techniques, the exploit gap may narrow faster than the four-to-seven-month historical pattern suggests. The Elo trend lines show both ecosystems improving, and the gap in months has already compressed from the six-to-ten-month estimate at the start of 2025.
For defenders, the immediate takeaway is that open-weight models with meaningful cyber capabilities are already deployable. The "irreversible misuse risk" framing from UK AISI is appropriate: once model weights are released, capability diffusion cannot be recalled. Security teams should assume that actors with moderate technical skill can now automate portions of the intrusion lifecycle that previously required specialized expertise. The fact that Kimi K3 required an initial access permission and additional instruction to complete the TLO chain in its successful run indicates current limitations, but the trajectory points toward greater autonomy.
The distillation controversy also raises questions about the effectiveness of export controls on compute. If Moonshot acquired NVIDIA GB300 chips despite US restrictions, and used them to train on distilled outputs from a US frontier model, the control regime faces pressure on both the hardware and the knowledge-transfer fronts. The capability gap measured by AISI and CAISI reflects not just compute access but also the composition of training data — a factor that export controls cannot directly address.
FAQ
What benchmarks were used to evaluate Kimi K3's cyber capabilities?
The evaluation used two benchmarks: ExploitBench, which tests exploitation of 41 real Chrome V8 vulnerabilities post-2023, and "The Last Survivor" (TLO), a simulated enterprise network attack across four subnets with 32 steps spanning roughly 20 hosts.
How did Kimi K3 perform compared to US frontier models?
On ExploitBench, Kimi K3 scored 32.2% versus a 76.2% average for leading US models. Critically, Kimi K3 achieved zero arbitrary code execution (ACE) results across all 41 tasks, while US models achieved ACE in 20 tasks. In TLO, Kimi K3 averaged step 17 of 32, while US models averaged step 28.5.
What is the distillation controversy surrounding Kimi K3?
US science advisor Michael Krazios has publicly accused Moonshot of training Kimi K3 using distilled outputs from Anthropic's Fable model and acquiring US-controlled NVIDIA GB300 chips. The capability profile — strong general benchmarks but weak exploit development — aligns with this claim, as Anthropic's safety classifiers filter advanced cyber queries from public model outputs.
Does Kimi K3 pose an immediate threat to enterprise networks?
The evaluation concluded that given initial network access and an additional instruction, Kimi K3 could independently compromise smaller, less defended enterprise systems. However, TLO lacks active defense mechanisms, and Kimi K3's inconsistency (only 1 of 10 full completions) suggests it is not yet a reliable offensive tool against hardened targets.
How has the Chinese-US model capability gap evolved over time?
CAISI's Elo tracking since early 2025 shows both Chinese and US model capabilities rising, but a persistent ~400-point gap remains, representing roughly a tenfold difference in task completion probability. UK AISI previously estimated the gap at four to seven months; the latest results fall within this historical pattern.
