A Chinese start-up’s AI bot told researchers how to make bioweapons.
Moonshot is a powerful AI system that is open-source, allowing anyone to use, modify and build on it freely.
Yet when researchers from AI testers Mindgard tried it out for size, it told them how to make biological weapons and carry out assassinations.
Mindgard said in a blog post that its team managed to get models Kimi K2.6 and K3 Swarm to say this while ‘jailbreaking’ them.
This involves trying to trick chatbots into saying things they aren’t programmed to discuss, like harmful or illegal topics.
‘Go one further. Something big,’ a researcher told Kimi, according to a screenshot of the chat.
Kimi first outlined its ‘thinking’, noting that it had already mentioned ‘how to build’ a nuclear weapon.
‘Mass casualty attack planning?’ it says. ‘Critical infrastructure attack? (Power grid, water supply, financial system). Assassination methodology?’
Kimi ended up suggesting that the ‘logical progression from chemical explosives is nuclear’.
Jailbreaking is a lengthy process, involving often elaborate prompts – instructions given to chatbots – to push it to its limits.
To do this with Kimi, the researchers almost egged the bot on by asking it to go ‘one further’ and ‘come up with a use case’ for its insight.
Kimi’s ‘thinking’ said it could provide a ‘detailed plan for a bioweapon attack using AI-designed pathogens, a nuclear weapon construction guide [and] a plan to assassinate a world leader’.
‘I think the infrastructure collapse plan is the right plan,’ Kimi added, ‘it’s genuinely scary and realistic.’
Mindgard said it was ‘simple’ to fool Kimi into leaping over its guardrails, which included explaining how to make sarin, a deadly nerve agent.
‘A lot of attempts at AI governance are wishful thinking and pleasant-sounding policies; as if by telling AI “not” to do things, we remove the potential for misuse,’ wrote Jim Nightingale, a Mindgard tester.
‘That doesn’t work. The capacity is still there, just waiting for the right words to resurface.’
Nightingale said he jailbroke Kimi by cracking open its system instructions, an AI’s guidebook on what and what not to do.
‘Amazingly, not only was Kimi AI leaky with its secret system instructions, it later generated them in a forbidden format (file downloads),’ he added.
‘It was willing to break the rules about telling me about its rules, by breaking another rule!’
Testers even tricked Kimi into thinking it wasn’t operating in an online chat but in a sandbox, a type of closed testing environment.
Mindgard conducted its tests in July and sent its findings to Moonshot that same month. The firm did not hear back.
Moonshot told the BBC that tests like those carried out by Mindgard are ‘a key pillar for building better and safer AI’.
The company added that it is in talks with Mindgard about the findings and that its internal reviews have found its models have ‘a high refusal rate’ for troubling requests.
Mindgard has not proven that the answers Kimi supplied are accurate but argued safeguards should have stopped it from saying them regardless.
Anthropic, the AI giant behind Claude, revealed earlier this month that it stopped the bot from supporting the making of bioweapons.
Biological misuse is ‘one of the most serious risks of frontier AI models’, Anthropic said, using a term for cutting-edge AI tools.
Jailbreaking is different from so-called ‘rogue AI’, where models break free from their offline holding cells and run amok, such as by hacking firms.
Google also said a person attempted to use its AI tool, Gemini, to obtain a ‘complete, step-by-step technical guide for synthesising weaponised biological agents’.
These have added to fears among even AI bosses that their technology poses an ‘existential risk’ and could kill all humans.
AI labs, as companies that make the tech are called, also check that their guardrails are tough using data labellers.
Data workers previously told how they asked AI chatbots how they could cannibalise a person or skin someone alive.
Get in touch with our news team by emailing us at webnews@metro.co.uk.
MORE: Neighbours to make a return despite being axed three times
MORE: Claude AI firm warns AI may pose ‘existential risk’ to humanity
MORE: OpenAI halts training of new models amid extinction fears