AI ownership is not security

I spent years as an investigator with the Toronto Police and watched organizations pour money into firewalls, monitoring systems and security audits to protect against cybercriminals. Breaches still happened, but we generally understood the problem. We knew what we were defending, how attacks worked and how to reduce risk.

Today, most discussion about AI security focuses on how powerful models could help attackers find software vulnerabilities or automate cyberattacks. Those concerns are real, and they deserve attention, but are only half of the story.

Far less attention is paid to the question of how we secure the AI systems themselves.

If you have used a chatbot to draft a work email, asked an AI tool to summarize a report or let your phone suggest a reply to a text message, you have relied on the judgment of a machine. Most of us have done this without thinking twice.

But think about the impact of this behaviour across society — like every hospital scheduling surgeries or police services analyzing evidence.

Federal and provincial governments are now spending billions to ensure Canada controls its own AI infrastructure. The federal government’s recently announced AI strategy doubles down on that ambition.

It is a sensible and important goal for Canada not to be dependent on foreign platforms for technologies that will become the essential fabric of our daily lives, from our national economy and security to our workdays and accessing of public services.

But sovereignty and security are not the same thing.

Building sovereign AI is about ownership and control. Securing it is about trust, about understanding whether the systems we rely on are behaving as intended.

AI systems are fundamentally different from the software we have spent decades learning to protect. Traditional systems are deterministic, meaning the same input produces the same output. You can test them, patch them and draw reasonably clear boundaries around what they do.

AI, however, is probabilistic, meaning the same question can produce different answers at different times. Its behaviour is shaped by training data, by ongoing interactions and by guardrails that are ultimately instructions rather than hard walls.

An attacker does not need to break into an AI system. They just need to persuade it.

We have seen tiny glimpses of this problem in the real world — it’s likely that only a very few have been identified and documented. For example, in 2023 a car dealership chatbot was convinced to agree to sell a $70,000 vehicle for $1. In neither case was the system hacked in the conventional way.

This example is not of a life-threatening nature and is seemingly isolated to a single transaction. But it’s not hard to conjure up how this style of boring persuasion could have a truly harmful impact.

Imagine an AI system used to help allocate health-care resources across a large public system. Over time, manipulated inputs or poisoned data influence how the system interprets patient needs and risk factors. No single recommendation would appear obviously wrong and no alarm would be triggered to catch the problem. Yet thousands of small decisions would gradually shift resources away from the people who need them most.

The problem is not obvious corruption, the way traditional attacks have played out, but a slow drift. A slow movement away from intended outcomes that is difficult to detect and even harder to explain.

A carefully worded prompt can convince an AI system to disclose sensitive information it was instructed to protect. Poisoned data can corrupt future decisions without anyone noticing. Researchers have repeatedly demonstrated these kinds of attacks known as “prompt injections” or “poisonings” and they continue to evolve.

The challenge is that the security tools most organizations rely on today were not designed to detect them. Firewalls, antivirus software and monitoring systems watch networks, devices and users. They do not watch how an AI reasons or responds.

The federal government’s new AI strategy is an important step toward building domestic capability and reducing dependence on foreign technology providers. But if AI is becoming part of the country’s critical infrastructure, then we must start treating AI security as a distinct discipline, not merely an extension of cybersecurity.

True sovereignty means control and control requires the capacity to notice when it has been lost.

Devi Narayan is a former investigator with the Toronto Police Service and is now CEO of Autnhive, a Canadian cybersecurity firm specializing in AI system security.