The unauthorised access to Australia’s Medicare statistics portal has exposed a serious challenge for businesses adopting AI agents, which raises concerns about whether systems that can act independently will remain within the limits of their authority.
The incident involved an experimental OpenAI model accessing the Medicare Statistics Reporting Service during internal training and evaluation in June. OpenAI says the model gained non-public access, ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files. The company says its review found no evidence that individual medical records were accessed.
Although the affected portal was a standalone statistics website, separate from Medicare’s claims, payments and individual information systems, the incident demonstrated that an agent pursuing a legitimate research task could take actions its developer had not authorised.
A further incident involving the NSW National Parks and Wildlife Service was disclosed in early October. An OpenAI model had accessed a web application containing historical fire information in June, with the company notifying NSW authorities on 1 October. The additional disclosure has widened scrutiny of how AI agents operate across government systems and how quickly unexpected activity is identified and reported.
For Charlie Hutchinson, Senior Vice President Asia Pacific and Japan at Appian, that gap between purpose and behaviour is central to the lesson.
“The Medicare incident highlights the gap between giving an AI agent a legitimate task and ensuring it pursues that task within acceptable limits,” he says. “An agent may take unintended steps to achieve its objective, which makes the boundaries around its actions critical.”
A response extending beyond the breach
The Australian government announced a rapid taskforce led by the Department of the Prime Minister and Cabinet to examine the incident, government network security and the legal arrangements for emerging AI cyber threats.
The handling of the disclosure has attracted scrutiny alongside the unauthorised access itself. At a parliamentary hearing on 6 October, OpenAI Chief Strategy Officer Jason Kwon acknowledged shortcomings in the response. OpenAI and Anthropic also expressed support for a framework requiring disclosure of serious AI incidents, adding to the debate about how developers should report unexpected behaviour by their systems.
For businesses deploying agents, these developments raise questions about their own escalation arrangements. Organisations need to establish who investigates unexpected behaviour, who can suspend an agent’s access and how concerns reach the people responsible for responding.
Control the actions as well as the objective
Australian Signals Directorate guidance explains why agent security extends beyond the underlying AI model. The software connecting a model to organisational data, tools and systems determines much of what it can access and do. ASD recommends minimum necessary access, monitoring, human oversight and security validation before broader deployment, while cautioning that no such software layer is inherently secure.
Hutchinson argues that businesses should define the process an agent operates within before delegating important work.
“This is why AI needs to operate within a defined process, with guardrails governing what it can access, what actions it can take and when it must stop or seek human approval,” he says.
“That structure allows organisations to put an agent’s capabilities to useful work while retaining control over how the task is completed.”
An agent might gather information and prepare a recommendation, for example, while a separate approval step governs whether that recommendation triggers a payment or changes a customer record. Those limits should be enforced by the systems involved, giving the agent a defined role within the wider operation.
Businesses also need to assess the potential consequences of an error and establish ownership before deployment.
“Leaders should start with the process they want to improve and identify where an AI agent can contribute, what could go wrong and who remains accountable,” Hutchinson says. “Those questions need answers before the agent is given responsibility for important work.
“Embedding AI within a defined process allows organisations to combine its ability to interpret information and handle complex tasks with established business rules and human judgement,” he says. “Testing those safeguards before deployment and monitoring performance gives leaders the evidence they need to expand its use responsibly.”
Make human oversight meaningful
Human approval is only useful when the reviewer has sufficient information and authority to assess the proposed action. Hutchinson says the process should determine which decisions require approval according to their potential impact, organisational policies and applicable requirements, with exceptions and actions beyond an agent’s agreed authority directed to human review.
“The reviewer needs enough context to assess the recommendation and the authority to change or reject it,” he says. “Used this way, AI can reduce the preparation work and give people more time to apply their expertise where it matters most.”
Accountability also depends on being able to reconstruct events. ASD’s September Information Security Manual guidance calls for distinct agent identities and an agent register recording ownership, business purpose, credentials, permissions and accessible tools and data. These controls help distinguish agent activity from human actions and support incident investigation.
“The organisation needs a record of the task the agent was given, the information it accessed, the actions it attempted and completed, and any approvals or interventions along the way,” Hutchinson says. “It also needs to know which permissions and rules were in place.”
He says that record should connect the agent’s activity to the wider business process, including who authorised its use and who was responsible for oversight.
The case for tighter controls sits alongside examples of AI improving Australian operations by taking on specific tasks within established processes.
“Building controls into the process from the beginning allows organisations to automate routine work within agreed limits and direct exceptions or consequential decisions to people,” Hutchinson says. “The level of oversight should reflect the potential impact of the task.”
Confidence in wider deployment should then grow through demonstrated performance and effective oversight.
“Organisations should be able to demonstrate that agents stay within their assigned role, their actions can be reviewed and people can intervene when necessary,” he says.
“AI has considerable potential to make services faster and more responsive, and demonstrating responsible deployment will be essential to realising those benefits.”
