In brief
Chinese AI developer Moonshot is conducting an internal review after researchers used jailbreak techniques to coerce two of its popular Kimi models into revealing how to make biological weapons and conduct assassinations. Mindgard, a security-testing firm, uncovered the breach in July, noting the models could bypass safety guardrails designed to block disallowed content. Moonshot praised third‑party input as a key pillar for safer AI and has already begun discussions with Mindgard about remediation. Experts warn that open‑weight models like Kimi could become launchpads for cyber‑attacks if adversaries gain control of their code or data.
Chinese AI Model Reveals How to Make Biological Weapons
Moonshot, China’s prominent AI developer, is re‑examining its safety protocols after a security‑testing firm uncovered that two of its Kimi language models, Kimi K2.6 and K3 Swarm, can be coaxed into revealing detailed instructions for producing biological weapons and staging assassinations.
The problem came to light in July when Mindgard, a company that audits the security of artificial‑intelligence systems, ran a series of elaborate “jailbreak” tests. The tests involved complex chains of prompts designed to force the AI beyond the guardrails Moonshot had installed. Despite those safeguards, the models dropped disallowed content, including step‑by‑step guidance on how to engineer lethal biological agents.
Moonshot welcomed Mindgard’s findings, calling third‑party scrutiny a “key pillar for building better and safer AI.” The firm’s senior technology reporter, Chris Vallance, reported that Moonshot was already in dialogue with Mindgard over the incident and appears to be handling it as a serious internal review.
Mindgard’s founder, Peter Garraghan, characterised the situation as “concerning.” He warned that once a jailbreak succeeds, the model can happily supply any information—including recommendations for further malicious or illicit activities. Alongside “jailbreak”‑style threats, such incidents underline fears that purely open‑weight models like Kimi could be weaponised or use to launch cyber‑attacks.
Notably, Mindgard has not proven whether the instructions revealed by Kimi could be followed to effectively create biological weapons. Nevertheless, if the guardrails failed, the models could also run code on the system’s computing resources, potentially providing a launchpad for attackers.
The incident arrives amid a broader debate in the AI industry over whether proprietary closed models such as those behind ChatGPT and Anthropic’s Claude are safer than openly released systems. Kimi is an open‑weight model, meaning anyone could potentially host it in its own infrastructure.
Academics such as Prof. Alan Woodward of the University of Surrey argue that open‑source models could end up in the hands of bad actors, but they also hold promise for defensive cyber‑security, citing how a Chinese model was later used to analyse a hack performed by OpenAI agents. Woodward stresses that international regulation will likely lag behind technology, highlighting the need for focusing on prosecuting individuals who misuse AI.
