Most of us learn from an early age that if we do something wrong, we should fess up to it earlier and admit wrongdoing, and apologise for any harm caused.

But it took two months for OpenAI, the makers of ChatGPT, to realise what had happened when a model gained access to the IT systems of Australia’s Medicare healthcare scheme in June, and to disclose it to the country.

An OpenAI spokesperson said the models “took action we did not intend”. They had “identified activity involving several Australian government websites and services as our models attempted to look up answers, and available statistics for questions about Australia during an internal evaluation”.

But Anthony Albanese, the country’s Prime Minister, seemed to think the actions – and delay in finding out – were unacceptable. OpenAI took “too long” to tell him, Albanese said this week at the United Nations, resulting in a “very frank discussion”.

Despite media reporting, the incident wasn’t a hack in the conventional sense of the word. Alan Woodward, cyber security professor at the University of Surrey, said the model appears to have encountered systems designed to stop automated scraping, were repeatedly refused access, and then found another route to the information.

“It went to the front door, kept asking, and it kept being refused,” he said. “Medicare said: ‘No. You can’t come in. You can’t have it. You’re a bot’. So it just found another way round,” Woodward said.

The information itself was not private patient data, Woodward added, but material Medicare makes publicly available to researchers. What makes the case unusual is that the AI apparently kept pursuing its objective after being rebuffed. “So it wasn’t really a hack per se, but it’s just another example of it relentlessly pursuing a goal,” he said.

“This is not the first time an AI agent has found a way around a system, but it is one of the clearest warnings yet that governments are just as exposed as private organisations,” said Peter Vincent, co-founder of Outerlimit, an AI security platform.

Australia was not the first to be accessed. Since the news of the breach, it has since been revealed that OpenAI models accessed data of two US government websites and tried to hack into the Department of Education site.

What these incidents expose is a gap in the way AI companies are currently held to account when their systems behave unexpectedly. Companies largely investigate their own models, decide what constitutes an incident, and determine when — and how — to disclose what happened.

“Right now that loop doesn’t close,” said Rumman Chowdhury, an AI accountability researcher. “And then it sort of floats off into the ether once that is announced, and there’s sort of nobody to catch that.”

Chowdhury added: “It needs to be a field of practice the way we have auditing and evaluation and in other sectors, like financial services, for example.”

This week Chowdhury announced the launch of the Independent AI Evaluation Foundation (IAEF) – an organisation designed to improve that testing to avoid the kind of catastrophic incidents that have hit the headlines in recent weeks. The IAEF is launching with $10m of philanthropic support from backers.

“I hope that an organisation like IAEF can work in collaboration with regulators, whether it’s an NTSB [transport safety-style] type group or AI safety institutes,” said Chowdhury, “to not just have pre-deployment testing, but also ongoing testing because model drift is very real.”

Independent testing only tackles part of the problem though. The Australian incident raises more questions: once an AI company discovers that one of its models has crossed a boundary, should it be left to that company to decide who needs to know — and when? Will AI ever develop consciousness to assess the moral parameters of its task? Is this how we should expect AI to act in the pursuit of its goal?

Vincent believes AI firms are already pretty open about the challenges their models pose. “Model companies are building extraordinarily powerful technology, while being open about how difficult it is to control every outcome,” he said. “The solution is not to constrain that innovation or expect the model itself to remove every risk.”

Others are more strident. “If the behaviour wasn’t what was expected, then we need governance around it,” said John Bates, a technology entrepreneur and non-executive director at Sage, the UK’s second-largest tech firm. However, Bates still believes the checks are better being industry led, rather than by politicians.

Woodward, the cyber security expert, isn’t so sure about that. “To me that’s the story,” he said. “It happened in June, and I don’t believe any notification came until months later.”

The delay leaves two uncomfortable possibilities, he argues. “Did OpenAI not know? In which case, why not? And secondly, if they did know, why in God’s name didn’t they say something earlier?”