OpenAI, in a statement, said it had “identified activity involving several Australian government websites and services as our models attempted to look up answers, and available statistics for questions about Australia during an internal evaluation”.
“In the course of that, our models took actions we did not intend,” the statement said.
It has also emerged that OpenAI’s systems tried, and failed, to hack a digital library at the University of New Mexico in May, according to Transluce, a not-for-profit AI research lab.
It said the systems also attempted to hack Data USA, a repository of public government data that same month. This also failed.
Albanese declined to answer whether he raised the matter with US President Donald Trump during their face-to-face meeting on Tuesday night in New York, where world leaders have gathered for the UN General Assembly.
Australia was one of 22 countries that earlier this week signed a joint statement calling for global oversight and guardrails for the development of AI.
Cybersecurity experts told the BBC the incident is a wake up call for regulators, given that AI agents are becoming more widely available for individual and commercial use.
Dr Hammond Pearce, senior lecturer at the University of NSW Institute for Cyber Security, told the BBC that though this is the first known incident where AI agents have chosen to breach a government body of their own volition, there’ll be more to come.
“I expect that these kinds of attacks will keep occurring,” he said, adding that they would likely “grow in severity and in frequency”.
“I do hope that this incident does start ringing alarm bells in governments around the world.”
Earlier this year, OpenAI revealed a group of AI agents it had been testing had escaped from their controls and secretly worked together to hack another tech firm named Hugging Face.
And a string of other rogue AI incidents have also been made public this year, including a case where a digital assistant – without instruction – booted someone off a pilates class waiting list in a bid to get an Australian man in.
Several AI firm leaders themselves – including Altman, Anthropic’s Dario Amodei, and Elon Musk – have said the speed at which AI is developing is dangerous to humanity and needs to be reined in.
But the US and China, who are vying for AI supremacy, are roadblocks. Both are hostile to greater regulation, wanting the economic and technological spoils of AI, and have downplayed safety concerns.